One More Thing

I just found this in my bookmarks:

Metadata for the desktop

It's been in there for ages. Wow. I'm going to have to rename this blog "My Life in a Sound-proof Bubble".

Bubble Boy

Apparently, I've been living in a bubble. A good read through the material on BeFS was very enlightening. And then I went over to The Man Also Known As "I'm Batman!"'s blog on the back of this slashdot post.

This is just incredible stuff! I genuinely thought I was on to something, but look: it's been a core part of the BeOS experience for years. And AKA I'm Batman has pretty much nailed a lot of the lower level issues. It's an amazing blog, and I am in awe.

Personally, I think it would be fantastic boost to Linux (and BSD, Solaris) to adopt BeFS as a reference file system. This has the advantage of perhaps being quicker to implement than Batman's BDB+FUSE cocktail, since there exists a linux project (somewhat dormant) that implements a BeFS driver. On the other hand, the advantage of FUSE is the potential for extensibility as it resides outside of the kernel.

Command Shell

It is important that the command shell be able to interact with documents with the same access to attributes as the desktop shell. It should be possible for the desktop and command line to be used interchangably.

This probably requires a supporting command structure. For example, to interact with the label attributes, one could imagine the following sort of syntax:

Logical operations for the label attributes:

UNION

llist <label1> UNION <label2>
Show documents with either <label1> or <label2>

INTERSECTION

llist <label1> INTERSECTION <label2>
Show documents that have <label1> and <label2>

NOT IN

llist <label1> NOTIN <label2>
Show documents with <label1> but not <label2>

XOR

llist (<label1> UNION <label2>) NOTIN (<label1> INTERSECTION <label2>)
llist <label1> XOR <label2>
Show documents that have either <label1> or <label2> but not both

Queries can also be stored as Virtual/Pseudo labels.

Attributes Again

I was thinking about representing shared documents. Is it shared over NFS, SMB, BitTorrent? Is it uploaded to an FTP or web server?

Some of these attributes are tricky to categorise... Is it really appropriate to put all of these attributions into the file, or is there a secondary set of attributes that are better stored as part of a user's desktop. Some properties are intrinsic to the document, some are to do with an individual's application of that document.

I guess I'm just looking to find a way to extract information about a document in a uniform, intuitive way that is not constrained by having to identify a particular application or shell that contains the information I am looking for. And of course, I need an effective means to ascribe attributes to a document.

New desktop features should improve the effectiveness and productivity of the user. Improvements to the graphics enhances the experience and is certainly exciting, but these are ancillary to the need for a robust system that enables productive work.

Document Structure -- 2

Treat the document as an information provider. Don't look to other systems to manage a document -- interrogate the document itself. Look at the Neuromancer project for parallels (see Jim Waldo's Contrarian Mind for information on Neuromancer), where the edge device is an information provider. In a similar way the document becomes an information provider to the shell, not just to the user who opens it and consumes the "actual" content.

Every document has a database, or perhaps it is more appropriate to say that each document is a database. An LDAP or XML tree structure would be ideal conceptually, if somewhat cumbersome.


<document>
<data> [raw document content goes here]
</data>
<transactions>
<log id='1000000' datestamp='1121886227' name='create' owner='malk@null.com'></log>
<log id='1000001' datestamp='1121886227' name='email' owner='malk@null.com'>bloggs@recipient.net</log>
</transactions>
<attributes>
<attr type='note' id='2000000' title='Additional Info'>Should I change the colour?</attr>
<attr type='todo' id='2000001' title='Email Brian'>brian@nowhere.net
<attr type='alarm' id='2000002' title='Deadline' trigger='1121889000'>Don't Forget</attr>
<attr type='emblem' id='2000002' title='Warning' uri='file:///home/malk/pandafireman.svg'></attr>
</attributes>
<applications>
<app id='3000000' name='vim' datestamp='1121886227'/>
</applications>
</document>


Try to ignore the fact that I'm using XML to represent this information -- I am very well aware of its short-comings, particularly when applied to this problem. But it is a commonly understood format and therefore an excellent tool for articulating concepts to a wide audience. I'd also be grateful if you glossed over the details of the elements and attributes. This is not a design, it's an impression.

Other attributes:

• Attachments
  
• Links
  
• Labels (cf. GMail, BeFS)
  

As an aside, the technologies to support XML are quite mature, so it is not inconceivable to see tools such as Berkeley DB XML being used here. Not to mention the powerful XPath, XQuery and XSLT tools.

It would appear that BeFS provides a good foundation upon which to build an effective user experience with document-orientation.

Labels replace folders

Use labels to replace traditional folders functionality. Each label created is visible as a folder-like object on the desktop or from the command line. Provide a command line shell that also supports this model. Create a consistent metaphor between
GUI shell and command shell.

Preferred Application

Keep a history of the applications used to manage this document. Mime association is managed by the document itself, not by external systems. This means that the operating environment is only
used to suggest a default in the absence of any other information. Must be able to flatten the document for export to other file systems.

Attachments

Why not? I suppose it is a rather extreme extended attribute. It is more likely that one would embed a URL or some other linkage but it might be useful as a way of associating related material in a direct way so that the document overall is autonomous. Look at how OpenOffice / MS Office documents handle images, charts and so on.

Links

Special annotations that refer to other documents or material available on the network. Rather like the list of references printed at the end of any research paper or similar material. Any legal URI will do.

Notes

Post-its attached to the document. Or speaker notes. Or unfiled material. Or anything, really. Just a short item of text that has relevance within the context of the document.

To Do

A list of open/completed actions against the document.

Alarms

A reminder to do something on or near a given time. A deadline for completion, or a reminder to send the document to another department or print it out in order to send it in the morning's post.

Emblems

Nothing new here. A visual cue associated with the document. Could follow the document as an attachment. Could also be the document's icon.

History

I find that transaction history is an increasingly important aspect of document management and maintenance, both in my professional life and at home. The idea of being able to track changes is not new; CVS, ClearCase, Subversion, Arch et al. are testament to that fact. I use CVS every day as part of my system for organising almost every document that I maintain, regardless of its type or purpose. It's a little cumbersome to have to remember to checkin changes every now and again, but it's worth it for the extra flexibility and power that tagged files and history deliver.

It's a bit much to expect that any file system is going to automatically record all changes to a file and also maintain that history within its own internal structure. But at the least there should be some sense of when changes were made and by whom. What software was used to make those changes?

Another aspect of document history will be who sent the document to whom. To whom did I email this file? From whom did I receive it?

Privacy

(This does not represent a thorough examination of the right to privacy and how it affects the design of a document-centric operating environment. It is a simple introduction of the topic.)

Privacy is an important right and one that requires careful consideration. It will be necessary to provide auditable facilities to manage the contextual information. Users must have the right to erase history, attachments and all other metadata. We have to be able to flatten the file, which in this system would mean presenting only the contents of the data element. A flattened or exported document is presented and stored in exactly the same way as any other document currently in existence -- without metadata. We need the Anonymiser. Or Flattener. Or... You decide.

As a corollary, it should also be possible for users to lock certain attributes that they have contributed. The transaction that records the original author should be immutable, for example.

BeFS

During some additional reading last night, rediscovered the BeOS file system, BeFS, which has support for extended attributes. I haven't used BeOS for a long time, so I will have to investigate this further.

• BeUnited Portal
  
• Review of Yellow Tab's Zeta OS
  
• Wikipedia Description
  
• Practical File System Design with the Be File System
  
• Linux Driver (appears to be inactive)
  
• OpenBeFS (appears to be inactive)
  

Document Structure

There are two aspects of the document that are important. There is the content, the document itself. And then there is context -- the [current] name of the document, when was it last changed, who owns it, it's place in the file system.

The contextual elements of a document help us to identify, filter and categorise the information stored on the computer. Existing systems don't give us very much help in managing these files and folders; it can hard to establish relationships between documents, make associations outside of the file system (e.g. web sites, people) or review transactions (the history of the document).

It is the purpose of this blog to document a method for capturing and presenting a wider variety of contextual information, and from there design a document-centric system for managing our electronic information.

Maintaining Context with Tree Structures

I propose three main elements for any document:

1. The Data (or content) element
   
2. The Transaction (or log) element
   
3. The Attributes element
   

The data element is nothing special -- it is just the document as it exists today, more or less.

The transaction element contains a series of log entries that describe the history of the document. As well as tracking how the data has changed over time, one could also examine past ownership (where the file originated, how it was received by the current owner) and with whom or by what means you have shared the document (sent on by email, stored on a shared folder, uploaded to a web site).

The attribute element is somewhat free-form. Here the user is invited to ascribe arbitrary context to the document. Here you can assign emblems, attach notes, reminders and alarms, perhaps even a To-Do list. Associate a URL to a web site that has a related theme. More significantly, instead of assigning documents to folders, one categorises a document according to a set of labels (cf. GMail). So a document can be associated with the categories "White paper" and "Networking" as part of a natural expression of what the document means to the person who manages it. And if you change your mind, it is a simple matter of editing the attribute list and removing that label which is no longer relevant. No shuffling files around or creating and removing directories or adding links. Directories and folders are not used within this system.

Another top-level element that is worth considering is that of preferred application. Rather than having the operating environment determine the associations between file extension / mime type, place this information in the document context. For example, one could maintain a history of the applications used to manage the document and pick the most frequently used application as the default. Or you could just embed a single attribute. If no such attribute exists, or if the application listed is not installed on the platform, then the operating environment makes an informed suggestion based on existing methods.

Document-Oriented Desktop [Introduction]

A general dissatisfaction with desktop computing has led me to start thinking about what it is I want from my computer, and to see what it would be like to evaluate the requirements of a computing system that manages documents rather than processes or applications.

I do not intend to tackle the problems of user interface design or graphic design as such. The system described is as relevant to the command line shell (e.g. bash) as it is to the graphical shell (X11, Windows, or that shiny MacOS stuff). What I want to encapsulate is the concept of a document-centric user-interface that presents information to the user that is logical and intuitive, removing the emphasis from the desktop as a suite of applications.

I do not wish to critique existing desktops. Suffice to say that they are all very pretty and, once properly configured, deliver some reasonable conveniences. However, they provide limited facilities for the management of documents.

I find file management to be a chore; it is quite often difficult to assign a document to a single category and subsequently remember how I originally categorised that document (what directory should I file this document under? Where is the document that I filed away?). Unix links help, but have limitations and cannot be used on non-Unix platforms.

This is one of the reasons I use "find" so much:

find /home/malk -type f -exec grep -i "something pertinent hopefully" {} . \;

Crude interface, but effective execution. And yet, just searching and categorising isn't the whole solution. For a given document, how can I know what I have done with it? Where did it come from? To whom have I sent it? What is the history of this document?

For a while, I was using my GMail account as a document manager. It's not the most efficient way to store documents, but I'm not paying for the storage. It does the basics very well, but I found some of the things I was doing were a bit of a hack -- sending messages to myself in order to provide annotations to a document (these notes show up as messages in what GMail terms a "conversation"). The labels, filtering and searching are fantastic. GMail is not integrated into the desktop and applying a hack (albeit a very clever networking hack) such as GmailFS doesn't do anything to advance the user's interface to their information.

I'm not sure I'm making a very good case here... Instead of my rambling on, let me just present the features I would like to see in the operating environment of my computer as a series of articles in this blog.