By day I write documentation for systems administrators, and as a result I spend a lot (perhaps too much?) time thinking about how we organize computer systems so that they can be both useful and easy to manage in the long run. “Right, so…” you say? Well indeed. Recently it’s become clear to me that there are some generalizable lessons to be learned from sys-admining that might be helpful to those of us who are less organized than they’d like to be.

Which is pretty much everyone, right?

Right. In brief:

• Automate everything that can be automated.
• Closely followed by don’t automate something that doesn’t need automation.
• Prefer simplicity over complexity, and prefer systems that require you to remember fewer things.
• Design systems to make it possible for others to easily understand what you’ve done.

To elaborate:

Automation

Computers are really good at doing what you tell them to do, and although we often like to finddle with them to make them work better, ideally the more we let systems take care of themselves. Also tasks that are automated, if the automation is designed and tested properly don’t make silly mistakes. If you’ve written systems to automate your tasks, you can understand and predict how your system is going to handle the kind of data that you throw at it.

The admonishment to “not automate” until you need something, is basically a variant on age old recommendation to “avoid premature optimization.” While automation is a good thing indeed, and if the thing you’re automating is really something that can be delegated to the machine without intervention on your part, then that may be worth your while to automate that task. By the same token, it’s easy to think “we’re going to need to do this thing a lot, I might as well automate it before hand.” Which is a reasonable thought to hand, but this puts the cart before the horse, and leads to two undesirable and possible outcomes: first the task doesn’t need to be automated because it isn’t needed very often; you misunderstand what needs to be done and automate the wrong part of the task, or your automation doesn’t cover the edge cases and will need to be rewritten later.

Conventionally, automation tends to cover “coding” or scripting of some sort of task. Outside of programming and systems development, “automating” a task could be as simple as creating some sort of editor macro, or developing some new structure in your data store (database, files, etc.) to hold or manage a particular kind of data.

Simplicity and Complexity

The basic reasoning here is that while complex solutions are often elegant and attractive, and make a lot of sense when you’re setting something up, they always make you scratch your head six months or a year later when you need to go back and find something that you did back then or make a change to the system. Be wary of solutions to any problem that require too much consistency on the part of the user. If a system only works if you must remember to follow more than a few steps in a precise order, chances are things are too complex, and you’ll end up screwing yourself over later.

Ergo: Err on the side of simplicity, you’ll thank yourself later.

The more components and connections there are in a website application or deployment server the more potential for breakage is. The more complexity there is the better chance that FurtureYou or someone working in your footsteps will be totally confused by what you have set up. The same thing holds for whatever your trying to organize and manage.

Generalizable Organizational Methods

Chances are you’re the only one who will be taking notes/organizing your work/storing information in your system. Nevertheless, I think it always helps to assume that other people are going to need to be able to make sense of your system. Be it your notes, and research or in your web-servers. Other people are sometimes our future selves.

I tend to use the word system, in a way that most people would use the word “method.” I hope that’s not too confusing or distracting. I think I’ll probably elaborate on these topics a bit more before in a later post. In a lot of ways this is part of the core of Cyborg Institute, and if you feel interested or inspired by this kind of stuff, I’d love to hear more from you. Be in touch!

Before I got started down my current career path--that would be the information management/work flow/web strategy/technology and cultural analyst path--I worked in a library.

I suppose I should clarify somewhat as the image you have in your mind is almost certainly not accurate, both of what my library was like and of the kind of work I did.

I worked in a research library at the big local (private) university, and I worked not in the part of library where students went to get their books, but in the “overflow area” where the special collections, book preservation unit, and the catalogers all worked. What’s more, the unit I worked with had an archival collection of film/media resources from a few documentary film makers/companies, so we didn’t really have books either.

Nevertheless it was probably one of the most instructive experiences I’ve had. There are things about the way Archives work, particularly archives with difficult collections, that no one teaches you in those “how to use the library” and “welcome to library of congress/dewy decimal classification systems” lessons you get in grade school/college. The highlights?

• Physical and Intellectual Organization While Archives keep track of, and organize all sorts of information about their collections, the organization of this material “on the shelf” doesn’t always reflect this.

  Space is a huge issue in archives, and as long as you have a record or “where” things are, there’s a lot of incentive to store things in the way that will take up the least amount of space physically. Store photographs, separately from oversized maps, separately from file boxes, separately from video cassettes, separately from CDs (and so forth.)

• “Series” and intellectual cataloging - This took me a long time to get my head around, but Archivists have a really great way of taking a step back and looking at the largest possible whole, and then creating an ad-hoc organization and categorization of this whole, so as to describe in maximum detail, and make finding particular things easier. Letters from a specific time period. Pictures from another era.

• An acceptance that perfection can’t be had. Perhaps this is a symptom of working with a collection that had only been archived for several years, or working with a collection that had been established with one large gift, rather than as a depository for a working collection. In any case, our goal--it seemed--was to take what we had and make it better: more accessible, more clearly described, easier to process later, rather than to make the whole thing absolutely perfect. It’s a good way to think about organizational project.

In fact, a lot of what I did was to take files that the film producers had on their computers and make them useful. I copied disks off of old media, I took copies of files and (in many cases, manually) converted them to use-able file formats, I created index of digital holdings. Stuff like that. No books were harmed or affected in these projects, and yet, I think I was able to make a productive contribution to the project as a whole.

The interesting thing, I think, is that when I’m looking through my own files, and helping other people figure out how to manage all the information--data, really--they have, I find that it all boils down to the same sorts of problems that I worked with in the library: How to balance “work-spaces” with storage spaces. How to separate intellectual and physical organizations. How to create usable catalogs and indices’s of a collection. How to lay everything down so that you can, without “hunting around” for a piece of paper lay your hands on everything in your collection in a few moments, and ultimately how to do this without spending very much energy on “upkeep.”

Does it make me a dork that I find this all incredibly interesting and exciting?

I wrote a series of articles nearly two years ago to rethink the GTD system, which I think is worth revisiting again. Not the essays, which were from when I called the website “TealArt” (don’t ask) and were before I really discovered free software and open source in a big way; but rather, I think two years out from my original article and even further out from the heart of the GTD fad, I think that it’s worthwhile to explore GTD again.

For those of you playing along at home, GTD (Getting Things Done) is really a “personal productive methodology” designed by David Allen that swept the geek community a few years ago. It’s good stuff, and while it’s certianly not a one-size-fits-all miricle cure for umproductive and overwhelmed folk; it promotes (to my mind) a number of goals that I think are quite admirable:

• Have a single system, that integrates across all aspects of your life. One place where systems can fail is if you’re using different “databases” (in the non-technical sense) to store information and tasks, and you have a piece of information that might fit in either system: when you go to look for it later (or need to be notified by it later) the chance of you missing the task because it’s on the wrong list is much higher if you have more than one place where lists might be.
• Think about tasks and projects being broken into “actionable items,” and have actions be the unit of currency in your system. As you assimilate information be sure to record anything that needs doing and keep it in your system
• Attach two pieces of metadata to your action: project (what larger goal does the action help you acomplish; you’ll likely have a list of these projects), and “contexts” (where do you have to be in order to do the action, things like “phone” “office” “erands”) are helpful for focusing and making it easier to move your projects forward.
• Do regular reviews of the information on your todolists, and spend (an hour?) once a week making sure you’re not foregetting things and that you’ve checked off all the actions that you’ve actually done and so forth.

There are other details, precise methods which GTD people focus on, talk about, and provide in their software applications. Frankly I’ve not read the book and I’m by no means an expert on the subject. I continue to have objections to the system: it assumes large tasks and quickly and easily be broken down into smaller tasks (which isn’t always true), and that projects follow linear and predicatable sequences, which I find to be almost universally false. While the reviews help counteract these sorts of assumptions about projects, I have always tended to find GTD a poor solution to the productivity problem:¹ both for myself and in my observations of how other people work.

At the same time, I think the notion of a single system that comes to the mainstream via GTD and of weekly/regular reviews, another artifact of GTD, are both really helpful and powerful concepts for organizing ourselves. The other aforementioned “features” are helpful for many, but I feel that very often organinzing the “GTD list,” and our lives to fit ino a GTD list is often too much of a burden and gets in the way of doing things.

I’m interesting in finding out how people these days are talking and thinking about GTD these days. I think the fad has died down, and I’m interested in seeing what we’ve as a geeky community have learned from the experience.

Interestingly, I’m probably doing something much closer to what GTD recomended these days than I ever have before. org-mode is (among many other things) a capiable GTD tool. I think it’s successful not simply because it supports GTD, and the task-management features seem to have grown out of an emacs/writing writing platform rather than a calendar platform. The end result is that I’ve found the GTD way to be quite effective, though its largely unintentional.

I’m interested in hearing where your own systems are, and how you feel about GTD these days:

• Do you use GTD or GTD based methodologies for your personal organization?
• If you only use some which, and why?
• If you don’t use GTD, what system if any do you use?
• If you once used GTD but stopped, or have considered using GTD and then didn’t, I’m particularly interested to learn why you came to these conclusions?
• What current factors influence the way that you organize your work?

I hope that covers everyone. I’m particularly interested by how creative folks work, but i think in the right light that covers most of us. I look forward to hearing from you?

Cheers, sam

I wrote in my post on the one true system about the informal systems that we use to interface the way we interact with knowledge and information in the “real world” with the way we represent that information on our computers. Exploring these systems lay at the core of the cyborg question, but today’s essay¹ is more about how our logic systems adapt as we use computers and as the kinds of information we need to store change and grow.

As near as I can tell there are a few kinds of “systems review” that we tend to do. Theoretically if you develop a system that’s flexible and that accounts for all of your future information needs, then you shouldn’t have to modify your system very much. Theoretically this is a good thing: better to spend time “doing things,” rather than thinking about “how you’re going to do things.”

The sad truth is that this doesn’t work out very well pragmatically: we change our work habits, and our information changes, and our projects change, and our informal logic for interacting with our computers fails to address the problems, and eventually everything spirals out of control. This is pretty abstract, but every time you see someone with hundreds of icons stacked on your (or someone else’s) desktop, or you find yourself with hundreds of unsorted (and unread!) email messages, or you have to hunt through half a dozen places for a PDF you are witnessing the symptoms of a flawed system.

The only way to address this is to review your “systems,” and make sure that you capture any problem before information spirals out of control:

1. Have an overflow bin, but only one overflow bin. This is important, but counter intuitive. By overflow bin, I mean something where unfile-able items are placed. This hopefully alleviates the tension to file away information that hasn’t been fully processed, or that doesn’t fit into your system, or might be ambiguously filed².

2. Do major reviews of your system only infrequently. By major review, I mean, think about how you use your information, what has worked, and what hasn’t, and then use this as a model for revising your system. Don’t do it regularly, even if you know that something isn’t working. Think of this as something that you do only about twice as often as you get a new computer. As part of this major review process:

   Keep a regular journal when you aren’t in the process of updating procedures. Track about what works and what doesn’t. Often I’ve found that I have ideas about how things should change, but the changes aren’t the kind of thing that I could reasonably change during normal work. These insights/problems are useful, eventually even if they aren’t always immediately relevant. So record them for later.

3. Do Minor reviews regularly. Look in the “overflow bin” from item one and see what’s falling through the cracks, file things that do need to be filed. The GTD folks call this a “weekly review,” and while GTD-task processing is only part of “the system."³ It depends on what kind of information you’re managing, but staying on top of and in touch with your “stuff” is important.

4. Be sure to “touch” your information regularly. While I’m in favor of keeping information even when it’s not apparently useful (you never know), I also agree with the idea that information is only really useful if you use it. I’ve often found myself falling into the trap where I’ll stockpile stuff “to read later,” which of course rarely happens. Avoid this and browse from time to time.

I mean in the end, I’m just a guy who writes more than he should, and has a pile of digital information that’s probably a bit too big, but this is how I do things, and I think the lessons I’ve learned (and continue to learn) may be helpful to some of you. Reviewing and thinking about systems before hand is, if nothing else, instructive.

Onward and Upward!

Email is a beast. While I would say that I don’t--at the moment--get a huge amount of email, I get enough that if I didn’t have a system in place for dealing with the email, it would be completely useless. I’ve not written about it here, but I have spent some time over the past few weeks working out how to replace a good but faltering system with a much more robust set up. Here’s the story:

(Warning, this is really nerdy)

First off this kind of really robust email solution isn’t for everyone, and there are a couple of unique factors in my setup that require the extra effort of this system. First of all, I need something that works because I hate the phone. If someone wants to get a hold of me, I’d much rather they write an email than call. If I don’t respond to email, people might call, or if I’m feeling overwhelmed by email, I might tell them to call. Both should be avoided. There are also a score of other reasons: I moderate a pretty high volume listserv, I need to send email from several different addresses and names/identities, and I have some pretty specific filtering needs, not to mention the fact that I have a number of pretty old email addresses that require a really powerful spam filter. I’m going to cover both what I used to do, and what I do now.

I *used to*: Collect a number of different email addresses into one gmail account and then check that email with either POP3 or IMAP. Gmail with IMAP was and is a great thing. With it, I could do a lot of in gmail-filtering and then have all of that just show up in my mail program. The problem is that there aren’t really good offline imap clients. Things don’t sync right. Mail.app can’t efficiently cope with new mail that doesn’t arrive in the inbox. You have to screw around with mail.app to get the multiple email identities to work. Mail.app wasn’t incredibly stable (though it has, to be fair gotten more stable.) Also mail.app’s filtering doesn’t work splendidly with IMAP.

Having an email program that works consistently and effectively is the key to keeping it under control.

A lot of my problems with this set up could surely be solved by using POP rather than IMAP, but after a year or so with IMAP (and gmail) I feel like the combination of the back up and having this account be useable and web-accessible is really ideal. I promised a post here on backup, and while I also can’t get into this here, I’m through with having my own machine be the only copy of important emails.

So what to do?

• Filter the hell out of my gmail account so that everything lands pretty much where it would need to so I can find it several months from now. Somewhere on this page is a list of all the funky boolean operators that gmail allows.

• Forward email out in chunks (so the lists get forwarded to one place, all of my frequent blog-related correspondents to another, moderation) to different addresses on my web-server that include 15 character random strings).

• This is actually a really sneaky way of passing gmail’s filtering downstream, and is otherwise a red hearing. I think however, that I could have probably eliminated the number of email addresses at play here by using “plus addresses” and eliminate the next, but it’s not a huge deal.

• Funnel all these things the forwarded email to a holding email address that automatically deletes everything after a week. This is a short term backup, if I accidentally delete something or whatever.

• Rather than use gmail’s built in forwarding, I made a filter that searches for another, longer random string in all the email. If it doesn’t find this (which I suspect it never will,) it forwards to my “home” email address. Again this isn’t a public email address.

  Time out, so what we have here. is gmail sorting forwarding two copies of each email to two different addresses, at once. All the email is sent to one address, and the second address depends on how gmail is filtering the email.

• Set up procmail locally to filter based on the random character string from the second step.

• Do some additional procmail filtering (which I think, as I figure it out, I’ll start to do more here, with and use geektool

• Read messages with mutt, because it sucks less than anything else

• Write messages with textmate

• Send mail using msmtp

And that’s about it. The getting it setup was the really hard part now all I have to do is use it, and everything lands where it should. I think functionally this is pretty damn good. It might be preferable to get something that isn’t in situations when I don’t have my laptop with me oraccesable. I read something about using something like rsync to handle mail box delivery. Might git work as well? I’m not sure. But that’s another battle for another day.

Onward and Upward!