“blah blah tycho says he’s not a programmer, blah blah.” I say stuff like this (more or less) quite a bit, right? But I know a lot about how computers work, a lot about how programs are constructed (at least in the abstract,) and while I’ve not (really) written code of any note I do read a bunch of code, and can almost always figure out how things work. So I suppose I should confess that I’m… working on learning how to program (in my copious free time).

I, of course, will keep you posted about this as it progresses, but for now I’ll settle to just explain the origin of this development: I started using emacs a few months ago, and I found the lisp-dialect that serves as emacs' extension language to be quite understandable. Maybe it’s something about how emacs programs are written to be hacked on, and even poorly documented emacs-lisp is rather well documented. I read the beginning of an introduction to haskell, and I think it has something to do with the functional aspects of the language. In any case, something clicked, and so here I am.

I mentioned the other day that I was looking to build a fact file, but that I wanted to follow up on this in a bit more length. As you know and/or have probably guessed, I’m a writer, and I consume a lot of information in the pursuit of this practice. That’s par for the course. The problem is figuring out a way to store collected information so that it’s useful later. Here’s a story:

I have a story that imagines a future where there are colonies and outposts throughout our solar system (among other things). I found myself a few days ago plotting out some details and I realized that I might be imagining a future with not only far flug outposts but also a substantially different system of orbital mechanics. I know about Hohmann transfers and enough about gravity assists, that getting between planets in the solar system is complex. My hope is that by knowing a bit about these things, I can avoid rank absurdity. In any case, I found myself looking up the length orbits of Jupiter and Saturn around the sun.

Now mostly this was just to get a sense of the distance, because unless I also posit free, lightweight, super-powerful propulsion systems, a trip between Jupiter and Saturn is going to take a number of years (say, 5-15; and even with posted amazing-drive, we’re probably still talking several years,) and no matter how fast/what kind of propulsion system you use, there are going to be a very limited “windows” for transport. If you miss the buss from Jupiter to Saturn, it could be 10 or 20 years before you could get off world. Not to mention the huge impact this would have on the course of cultural development on these colonies.

But having said that, this detail about the orbital lengths (though burned into my memory at this point), isn’t the kind of thing that I track very well, and what about the next time I have a question like this? Or what about a news story that I come across, or the abstract of a scientific paper that catches my interest? This detail about orbits of the outer planets was really just the straw that broke the camels back.

It was clear that I needed a system for storing information, facts, notes for later retrieval. I wasn’t sure what that system would look like, or what I would need from it, but I was sure that something was better than the “read and hope” method I’ve been using.

I did some brainstorming and came up with some basic requirements. I needed something pretty unstructured so that “records,” which were just links and a few words would be just as complete as records that had lengthy notes. I needed built in meta-data functionality to store categories, tags, and citation information (links, date, identifiers). Easy capture and editing is a must, and while I was, and am, willing to consider functionality outside of emacs, but emacs is preferred, and it would take a lot for me to want something that couldn’t be stored in plain text files.

When other people (individuals) come to me with similar problems I almost always recommend private instances of Wordpress where people can post notes. This often is just the right thing. Conceptually it works like a notebook or journal in the physical world, but it has good meta-data support (categories, tags, dates), you can use it from anywhere (the web, the API), and it’s a mature system. This works great for a lot of people, but I’m not incredibly happy with the web-interface, and I’d need to rely on search more than I think I’d like.

I put out a call on identi.ca for help on this problem. There were suggestions of PlannerMode and various systems based on org-mode (which is what I’m inclined to use at this point.) I then had this “moment” issue about not wanting my “fact file” to be built on some sort of to-do list. Todo lists are great, I love todo-lists (perhaps too much?) Having my information management software be built around “tasks” and “projects” provides too much of the wrong kind of structure.

I was frustrated, as you might imagine. After all I just wanted some sort of index-able note-card system, that I could use to store some basic information without fuss. I’ve settled down a bit and I’m using a format using a skeleton/org-remember template to store my fact file in an org-mode text file (data.org). The entries look something like this:

* The title                    :a:list:of:tags:
  :PROPERTIES:
  :date: <2009-03-14 Sat>
  :cite-key: a-u/uid
  :link: http://tychoish.com
  :END:

And then finally some notes text

And the input is just an prompt-tab interface. I can also add more details to the properties, section. If that makes sense at a later date. So the creation of these records is pretty quick. I’ll get come code up when I have something a bit more clear. In the mean time, I’d like to continue to have a bit of a discussion about this information management problem. It’s a domain that is clearly very suited to technological enhancement, but at the same time there aren’t a lot of solutions on the ground. I suppose that’s where I come in, but feedback is most appreciated on interface questions, on alternate use cases and applications. I look forward to hearing from you…

One of the things that I find a lot when I’m seraching the internet for emacs things (or, in the case of my google alert, when emacs stuff on the web finds me.) Are people writing blog posts that are along the lines of, “so I was playing around with emacs weblogger mode…” Which is pretty much what this is.

I’ve been toying with the idea of switching to a git-based blogging platform/site generator, that would be much more slim than my current tool, which (though I love it, and recomend it to other people regularly) doesn’t seem to fit my workflow particularly well. Since leaving the (wonderful) TextMate blogging bundle behind, I’ve been in search of a blogging tool… and, well, I’m still looking.

When I find it, I assure you that you’ll be the first to know.

I’m working on a longer post on the topic, but the past few days have seen me spend a bit of extra attention trying to develop some way of doing a “fact file” inside of emacs. Basically what I need is some way of storing information (links, quotes, text) along with bibliogrpahic information (where I got it, when I accessed it) and some sort of meta-data (tags, categories, search index) so that I can find things again. Largely, this data isn’t for any specific project, but it is a way to record general notes on the information that I consume in an effort to make that data more useable in the future.

As I firm up the solution, I’ll write more about it, but I’ve discovered something about the software targeted at this domain (generally, “personal information management,") that’s worth sharing: there are a lot of notebooks and databases around, but they’re all very much alike: anything that’s structured as a database is generally desgined to store addresses, contact information and other “CRM” data, which is nice, but not what I’m going for at all. There are also a lot of notebook applications that are modeled on wikis (too unstructured) or outliners (ok, but hierarchical). Interestingly, it seems every kind of personal information management package that isn’t an address book (for emacs, but I think this is to some degree generalizable) has grown to a point where it can manage and extrapolate todo-lists from your notes.

Once upon a time, emacs itself was critiqued (and still is) because it has the ability to send and recieve email, and that this feature (email sending) was a sign of “feature bloat,” becasue after all, sending email with a text editor. That’s a bit a field. I’ll let you decide how history has dealt with the email issue. I in the mean time am going to write, and figure out some way of doing a better job with this information.

So I get a Google alert in my RSS reader for “emacs,” this way when anyone mentions that operating system text editor on the Internet, I know.

I did it, initially, as a gimmick. The interesting thing, is that it’s been really interesting. In the grand scheme of things, I’m pretty new to emacs so I’m like a sponge for new information. I’ve learned, chiefly, about emacs-fu, which is a delightful little blog.

Perhaps more interesting is the fact that fully half of the sites that pop up on the emacs alert are sites that I’m already familiar with, or worse (better?) things that I’ve written. This leads me to think that the active discussion/community regarding emacs is actually pretty small, in a day to day sort of way.

It’s possible that most people who use emacs don’t feel the need to really talk about it, but given how vehemently many people feel about emacs that doesn’t seem terribly likely. It’s also possible that emacs' niche is shrinking in the face of competition from the Java IDEs, TextMate, and Firefox.

Another theory is that most of the conversation regarding emacs happens on the emacswiki, and that the sort of “here’s a protip, about emacs,” is too much of a niche, and not the kind of thing that’s really appreciated, so the blogging happens here and there. Or… no one’s doing search engine gaming/hacking (SEG? SEH?) on emacs, so the alert isn’t clogged up with crap from dweebs and industry.

In whatever case, it’s interesting. Or interesting enough.

Onward and Upward!

(ps. sorry for the short post and the weird schedule this week. Next week should be better. --ty)

I promised that I’d post some of the stuff from my .emacs file that makes my org-mode system work. Here we are.

There are some basic settings that I use on all major modes that I use in emacs. Basically, I want to attach the spell checker (the minor modes, flyspell-mode and auto-fill-mode). These lines do this:

(add-hook 'org-mode-hook 'turn-on-auto-fill)
(add-hook 'org-mode-hook 'flyspell-mode)

I also, attached “.org” as the file extension to org-mode. This setting is good for this kind of thing:

(setq auto-mode-alist
(cons '("\\.org" . org-mode)auto-mode-alist))

The following are a list of basic org-mode related settings that I’ve found helpful. In some sequence, I keep org-mode files in the ~/org/ directory, with codex.org being my general catch-all file. I like my agenda views to include todos, even if they’re not date-specific (this is a great boon) I’ve included the diary in the agenda views for grins, though I’m not yet smart enough to really make the most of that one.

The odd-levels-only, and hide-leading-stars are aesthetic settings only, and can be changed/converted from at any point, but I like them

The org-todo-keywords setting allows you to specify alternate todo-statuses. I’ve found that this sorting is a useful and allows me to visually sort out things I need to write, versus chores and other more clerical tasks. The pipe seperates finished statuses from open statuses. I debated for a long time about weather “differed” should be “done” or “not done,” but decided that with “pending,” I was safe to use “differed” tasks for “not my problem any more” items.

(setq org-directory "~/org/")
(setq org-default-notes-file (concat org-directory "/codex.org"))
(setq org-agenda-include-all-todo t)
(setq org-agenda-include-diary t)
(setq org-hide-leading-stars t)
(setq org-odd-levels-only t)
(setq org-todo-keywords
      '((sequence "TODO"
                  "WRITE"
                  "REVIEW"
                  "PENDING" "|"
                  "DIFFERED"
                  "DELEGATED"
                  "DONE")))

The next bit, is something that I got from Jack. It creates an org-mode file with time-stamp headlines which you can use to create a journal file to record daily activities.

The first block sets up which file the journal should be in, and the second sets up entry. My main complaint with this is that I’m not very habitual about using it.

(defvar org-journal-file "~/org/journal.org"
   "Path to OrgMode journal file.")
(defvar org-journal-date-format "%Y-%m-%d"
   "Date format string for journal headings.")

(defun org-journal-entry ()
  "Create a new diary entry for today or append to an existing one."
  (interactive)
  (switch-to-buffer (find-file org-journal-file))
  (widen)
  (let ((today (format-time-string org-journal-date-format)))
    (beginning-of-buffer)
    (unless (org-goto-local-search-headings today nil t)
      ((lambda ()
         (org-insert-heading)
         (insert today)
         (insert "\n\n  \n"))))
    (beginning-of-buffer)
    (org-show-entry)
    (org-narrow-to-subtree)
    (end-of-buffer)
    (backward-char 2)
    (unless (= (current-column) 2)
      (insert "\n\n  "))))

The integration between remember-mode functionality and org-mode is one of those things that just makes org-mode amazing and awe inspiring. The sad part is that it takes some setup to make it work right and therefore doesn’t work straight out of the hook.

I’d explain the template syntax better if I understood it a bit better. I should look into that.

(require 'remember)
(setq remember-annotation-functions '(org-remember-annotation))
(setq remember-handler-functions '(org-remember-handler))
(add-hook 'remember-mode-hook 'org-remember-apply-template)

(setq org-remember-templates
      '(("todo" ?t "* TODO %?\n  %i\n  %a" "~/org/codex.org" "Tasks")
        ("notes" ?n "* %?\n  %i\n  %a" "~/org/codex.org" "Inbox and Notes")
        ("blog" ?b "* %U %?\n\n  %i\n  %a" "~/org/blog.org")
        ("technology" ?s "* %U %?\n\n  %i\n  %a" "~/org/technology.org")
        ("fiction" ?f "* %U %?\n\n  %i\n  %a" "~/org/fiction.org"))

Finally, key bindings that make org-mode functionality accessible whenever I need it in emacs. I should do things to have raise emacsclient windows from other applications, but I’ll deal with that later. There aren’t that many, and I put org-mode stuff under control-c (C-c).

(global-set-key "\C-ca" 'org-agenda)
(global-set-key "\C-cr" 'org-remember)
(global-set-key "\C-cj" 'org-journal-entry)

And that’s it. If you use org-mode, what’s the killer snippet that I’ve forgotten? If you don’t use org-mode but are curious, what should I talk about next. If you’re still not clear what org-mode is, ask, as I should work on getting better at explaining.

Thanks for reading. Cheers!

The last time I talked about org-mode, I covered a number of my “beginner mistakes,", this time I think it’d be a bit more productive to explain how I’m actually using org-mode. For those of you playing at home this is one of those systems I talked about.

I’m going to try and avoid including bits of code/lisp in this post, as I don’t want to confuse “what I’m doing,” with “how I’m doing it.” We’ll get to that point in time, and there’s plenty to talk about.

At the moment the features of org-mode that I use the most are:

• org-agenda-mode
• checkboxes
• org-remember-mode
• and org-mode outlines

I also store org-mode files in a git repository that lives in my home folder of all machines that I use with any regularity. I’m running emacs-23 (ubuntu package: emacs-snapshot), generally in server-mode, with two frames open, one has “the thing I’m writing” and the other has the org-mode-agenda and some org-mode file in the other window.¹

System Layout

At the moment I have 9 org-mode files:

1. A file the novel project, that contains background notes, and outline of the story, as well as project management stuff.

2. A file that contains notes for blog posts. This is mostly organizational, and I just add ideas as I have them and check them off and archive them when I’m done.

3. A general “codex” file, that I dump links. I’ve had a file like this for a number of years, and I generally find that having a catchall like this is productive.

4. A general fiction file, for project management issues related to the fiction projects that aren’t the novel. I’ll probably split this up as needed in the future.

5. I have two files to manage web development/design projects that I’m working on for myself.

6. One file to manage my academic/research project. Includes notes/content as well as todo items.

7. A “technology” file that contains lists of tweaks and other technological projects that I need to undertake (like switching to OpenDNS).

8. A file for work projects and notes
9. A journal file, for tracking activities and accomplishments.

Org-Mode Features

The features that I use:

Outlines and Checkboxes

This is a no brainier, but it makes sense. Org-mode is at it’s core an outlining program, but I think it is important to underscore this. It’s an outlining program. use it to create outlines. All of the talk and videos that I’ve seen explaining and demonstrating org-mode talk about the cool agenda functionality, or the table feature (which are great tools) but this washes over the fact that it’s an outlining program.

Why am I stressing this? Because the agenda commands and all the other functionality actually works better when you have a bunch of files with outlines in them. When you have outlines to deal with, you can tag the items that are truly “actionable” with “TODO” (or other action words) and include them with your agenda items. Or if an item/heading on your outline is time sensitive you can use the org calendar system to toggle deadlines/schedule points. And lists below headings can have checkboxes, which is a great boon, as well. But, if you work backwards and attempt to anticipate what agenda is going to do, then you’re loosing some of it’s functionality.

Brainstorm. Write notes. Add TODO flags to items that require attention. Add deadlines or schedule points if they’re time sensitive. But don’t organize the outlines for anything other than their utility as a reference document. Org-mode does the rest with….

Agenda

The org-agenda mode is the glue that makes the whole system work, as it aggregates items together from all your org-mode files. I’ve set mine up so that it displays a weekly agenda with all open TODO items. This way I can see everything that needs action, without needing to add deadlines to tasks which didn’t have deadlines so much as “don’t forget me” status.”

There’s all sorts of stuff that one can do with Agenda to generate views that filter based on tags or some such, but I’ve not gotten there. It’s really flexible.

Remember

Remember is a feature that originated independently, but has been enhanced to work with org-mode. Basically you trigger it with a key-binding and it pops up a temporary window which allows you to select a template and then add notes and other items, trigger save, and it disappears and writes to the relevant org-mode file.

One of the great pains of digital systems for note taking and organization is that if you have an idea, it is often hard for us to find a way to record these notes quickly before the moment leaves us. Remember changes that, and it’s great.

Next time? Code. Cheers!

It takes me a while to adapt to new things with the computer. While I pick things up pretty quickly, I’m aware that it takes a while for me to adapt to new ways of working with the computer. I like to settle into a system before I start changing the key bindings or writing scripts (which I always think will be more successful than they are). This seems reasonable, and is generally something that I’d recommend to other people looking to learn “cool new stuff.” Go slow.

In light of this realization this post will be a review of some of my more recent adventures with org-mode. For the uninitiated, org-mode is really a subsystem of the emacs text editor.¹ It’s really powerful, and has a lot of disparate features that combine to do something really kind of magical. On one level, org-mode is just an outlining program: it provides some shortcuts and interfaces that make writing structured text really productive and pleasurable (e.g. easy navigation, block folding). On another level, org-mode is a fully featured calendaring and task/project management tool. The brilliance, is that it isn’t just and outliner or a task manager, but it provides accessory tools (like a great calendar, and even a table/spreadsheet functionality) that integrate the environment.

I found about this two months ago, and I’ve slowly started to ease into using it. This past week, however, I think I crossed some sort of boundary, because I started using org-mode for most of my note-taking and project planning/organization work. And now that I’m here, I think a small series of blog posts is in order.

I’ll start, I think with “mistakes I made,” as I couldn’t hope to explain everything the system does, and explaining how I’m working right now isn’t nearly as interesting.

My first approach was to keep an org-mode file in all of my project directories, so that the .org file would be close to the files where the actual work was going on.

Wrong.

Turns out, keeping all your org-mode files together seems to work a lot better. Org’s calendar/project planning features work by generating “agenda views” of todo items and time-specified events. While the system could pull from a half dozen different project directories. Keeping things centralized means that you can add new files willy-nilly, it makes it easy to keep things synchronized between machines.

My second approach was to keep one org-mode file and divide projects using the hierarchical nature of the file to keep things straight.

Wrong.

Turns out, that while we can represent all of our projects hierarchically² beyond a certain level it doesn’t make a lot of sense to implement “categories” or project headings in this way. Org-mode provides a tagging infrastructure and the aforementioned agenda tool to tie together files, so while you can go overboard in creating new org files, you’re probably not incredibly likely to for a while. Use outline hierarchies to represent information, use files and tags to categorize and sort your headlines.

When I discovered the agenda capabilities, I made todo items out everything and started giving all of my tasks dates so that they’d be sorted into the agenda view.

Wrong.

Turns out, you can get agenda to generate a list of undated todo items before the agenda view, and that--at least for the way I work--setting soft deadlines and scheduling tasks just confuses the point rather than facilitating action. Org-mode also has a system for task priorities--which I haven’t felt the need for--but I think the larger lesson here is don’t attach arbitrary information to items in your org-mode files. Let the agenda mode do it’s work, and you do yours, and it’ll work out.

I started my ~/org/ directory³ by creating .org files for all of the major projects in my life. Fiction writing, day-job, the blog, academic work, so that I’d be able to collect similar kinds of notes and todo-lists together in files.

Wrong.

Turns out, that while some of the setting up big sphere-based org-mode files is unavoidable, the truth is that given the power of the agenda filter, putting a lot of information that is only casually connected in the same file doesn’t make much sense when tagging can provide needed meta-data. There’s a happy medium between “dividing things too much to the point where there’s too much ‘system’ to manage,” and “not dividing enough so that you have to build informal systems in the outline which complicates the organization.” In my experience “too much division” is much harder to reach, and “not dividing enough,” is quite easy.

I’ll be blogging more about org-mode in the coming days and weeks. Stay tuned :)

Onward and Upward!