I’m a huge fan of emacs' org-mode on so many levels: as an IDE for knowledge workers, as a task management system, as a note taking system, and as the ideal basic mode for so many tasks. However, I’ve been bucking against org-for a number of tasks recently. The end result is that I’m becoming less org-dependent. This post is a reflection on how I’ve changed the way I work, and how my thinking has changed regarding org-mode.

Fair warning: this is a really geeky post that has a somewhat specialized context. If you’re lost or bored. check back later in the week.

The Perils of Org

The problem I keep running into with org is that I really don’t prefer to work in org-mode.¹ Org is great and very flexible, but I don’t like that it means that all text-based work is dependent on emacs. My brain is already wired for Markdown and reStructured Text from years of blogging and work projects respectively.

And then there’s this organization problem. There are two ways you can organize content in org-mode. The first is to just dump every thing in one org-mode file and use the hierarchical outlining to impose organization to organize everything. The second is to have every project inside of it’s own file and use outlining incidentally as the project needs it. Content aggregation happens in the agenda.

The problem with the “large files” approach is that you end up with a small handful of files with thousands of lines and imposing useful organization is difficult (too many levels and things get buried; not enough and inevitably your headings aren’t descriptive enough and you get confused. Furthermore, I end up living in clone-indirect-buffer-other-window’d and org-narrow-to-subtree’d buffers, which is operationally the same as having multiple files it just takes longer to set up.

The problem with the other approach, having lots of different files, is that I have a hard time remembering what is in each file, or in logically splitting big projects into multiple files. The agenda does help with this, but the truth is that the kinds of org-headings for organization and tasks are not always the same kinds of headings that make sense for the project itself. I often need more tasks than organizational divides in a project. I tried this approach a couple of times, and ended up with useless mush in my files.

Typically, I can never make the “lots of file approach” really work, and the big files problem lead me to general avoidance of everything. Not good. The key to success here is good aggregation tools.

Hodgepodge

In response, I’ve made a couple of tweaks to how I’m doing… pretty much everything. That is:

• I’ve moved most of my open projects into a locally ruining and compiling ikiwiki instance. Both laptops have this setup, and there’s a central remote to keep both (all?) machines in sync.
• I’m using ikiwiki tasklist to basically replicate the functions of org-agenda. Basically this crawls the entire wiki looking for lines that begin with certain keywords and generates a “todo” page based on these notes. Really simple, incredibly useful and it solves much of my aggregation needs.
• I still have some stuff in org-mode: notes for the nearly-finished novel, lots of random old (legacy) data, 12 various open tasks, and org-capture. I’m thinking of pointing various org-capture templates at files in the wiki but haven’t gotten there yet.
• I’ve basically taken the “lots of little files,” approach to my writing and work. I’ve not over-leaded the system yet. Each major project gets a page in the root level of the wiki for overview and planing, and then sub-pages for all related project files (if/as needed)
• It turns out that the markdown-mode for emacs has gotten a few improvements since the last time I downloaded the file, including better support for wiki-links that are mostly compatible with ikiwiki. Also from the same developer deft which implements a pretty nifty incremental search for text files in a given directory. So between these tools, ikiwiki, and the ikiwiki-tasklist there’s support for the most important things.
• In terms of publishing, beyond ikiwiki for tychoish.com and the personal organization instance, I have a couple of other smaller wikis (also ikiwiki powered,) and I’ve been playing with Sphinx as publishing for more structured documents and resources (i.e. documentation, novels, and collections,) particularly those that need multiple formats and presentations.

I’m sure there will be more shifts in the future, I’m sure. I think this is a good start. Thoughts?

This post is adapted from a post I made to the org-mode email list a few weeks ago. I proposed an application to compliment MobileOrg for writing. Where MobileOrg collects the core bits of org-mode’s task planning functionality in a form that makes sense for smart phone users, the parts of org-mode functionality that people use to for writing and organizing the content of larger form projects isn’t particularly accessible.

I spend (or should spend) 70% or more of my time in front of a computer writing or editing something in org-mode. Most of my org files have tens of thousands of words of blog posts, notes, drafts of articles, and so forth. While I can store that data on an android device with only minor problems using a little script that I put together, and I can capture content into my org-files using email and some nifty filters, and there are text editors that can let me edit these files: it could be better.

The proposal is simple. Can we build something like Epistle for org-mode? It might just render org-mode text to HTML, and frankly that would be enough for me. If the editing interface had an org-indent-mode equivalent, org-syntax highlighting, and even collapsing trees or org-narrow-to-subtree, that’d be kind of like heaven.

I’m not a mobile developer, so I can’t promise to start making an app this instant if there’s interest but if anyone’s bored and thinks this might be a good idea (or knows of something that might work better for this.) I’d love to hear about it. If someone wants to start work on this, I’ll do whatever I can to help make this a reality.

Onward and Upward!

Org-mode is great. Org is this super-task management package that merges outlining and structured document editing abilities with task management glue. It sounds like a weird combination at first, when you realize that it means that you can effectively do work and organize your work in the same set of files without needing to “switch modes,” between a planning/task list interface and a “doing things” interface, the effects are amazing.

I’ll leave “the awesomeness org-mode” to other people and other posts, and spend time here focusing on why “org-mode is awesome but won’t do your work for you,” and “if you’re new to org-mode you will probably want to do things in a certain way, but don’t” issues. Org-mode is great and it can be even better if you avoid developing a few bad habits.

Use `org-capture <http://orgmode.org/manual/Capture.html>`_ as much as possible. Org-capture is a quick interface within emacs that lets you open a temporary buffer, take a few notes, and save the notes into a file. It has advanced features for more complex data entry features and data types. The temptation is to use it as a “quick entry” tool for task list items, but don’t just use it to capture new tasks. Capture links and bookmarks, store notes and important information, If org-mode is your outboard emacs-based brain, then org-capture is it’s main input device. Fighting it, means that your org files end up being less useful.

Avoid using org-mode as a simple task list, and particularly avoid constructing the content in your org files to “game” your agenda view. The agenda compiles a working task list for your actionable notes, working backwards from this means your notes are less than useful things can get lost and all of the really cool things that org lets you do aren’t accessible to you.

This may just be the application of a good general information management practice, but: distribute information evenly throughout all levels of the organizational hierarchy of the file. Use the org-narrow-to-subtree function to focus your current work on a specific portion of a file, and don’t bury information or have it sitting around in one big unorganized pile.

Hope this helps!

I’ve written about this problem in org-mode, the emacs outlining and organization tool that I us, before, but I’m readdressing it for my benefit as well as yours.

Org mode is an outlining tool, fundamentally. It provides a nice interface for editing and manipulating information arranged in an outline format. Additionally, and this is the part that everyone is drawn to, it makes it very easy to mark and treat arbitrary items in the outline as “actionable,” or todo items in need of done. The brilliance of org-mode, I think, is the fact that you spend all your time working on building useful outlines and then it has a tool which takes all this information and compiles it into a useful todo list. How awesome is that. For more information on org-mode, including good demonstrations, check out this video.

The problem is a common and recurring one for me. I basically live in the agenda mode--that compiled list of todo items--and I don’t so much use org-mode for making outlines. Truth is, I have a “Tasks” heading in most org files, and I use the automatic capture option (e.g. org-remember) to stuff little notes into the files, and beyond that, I mostly don’t interact with the outlines themselves.

This isn’t a bad thing, I suppose, but it means that org-mode can’t really help you, and you’ve short-circuted the ability of org-mode to improve the organization. Under ideal circumstances, org allows you to embed and extract todo lists from the recorded record of your thought process. If you’re not actively maintaining your thoughts in your org-mode files, it’s just another todo list. That isn’t without merit, but it doesn’t allow the creation of tasks and the flow of a project to spring organically from your thoughts about the project, which is the strength of org mode.

Intermission: I took a break from writing this post to go and reorganize my org files. What follows are a list of “things I’ve been doing wrong” and “things I hope to improve.”

• I don’t think I had enough org-files. There are lots of approaches to organizing information in org: one giant file, lots of small files for individual projects, a few mid to large files for each “sphere” of your life.

  Initially I took the “medium sized files for major ongoing projects.” I had a writing file, and a work file, and a writing file, and files for the fiction projects that I’m working on, and a notes file, and a clippings file, and so forth. Say about 8-10 files. It works, but I think the thing it did was it caused me to use the org-remember functions to just dump things in a “tasks” heading, and then work from the agenda buffer, and not ever really have to touch the files themselves. Org files need to be specific enough that you would want to keep them open in another window while you’re working on a project. I think the point where you know you’ve gone too far is when the first level headings start to replicate organization that might better be handled by the file-system.

• Use the scheduling and deadline functions to filter the todo list into something that is workable. It’s easy to just look at the task list and say “oh no, I don’t want to work on this task right now because it depends on too many things that aren’t done, and there are other things that I could work on.” Scheduling an item, if not setting a deadline, forces me (at least) to think practically about the scope of a given project, what kind of time I’ll have to work on it, and what other tasks depend upon it.

• When you’re using org to manage huge blocks of text--or any system, really--it can be difficult if you have multiple hierarchies and depths of greater than two or three. It just gets hard to manage and keep track of things and figure out where things are, particularly given how useful and prevalent search tools are.

  Having said that, When you’re organizing tasks in org, that limitation, one that I find myself imposing upon myself doesn’t really work terribly well, and leads to files that might actually be more difficult to read and to work out of.

• I started using the “org-archive-subree” function for archiving content when I was through with parts of the outline, This sends the archive to a separate file and while it works, I find it… less than useful. I’ve since discovered “org-archive-to-archive-sibling” which is a great deal of awesome, and I recommend using it exclusively.

• Write content in org mode when possible. Though some people (hi Matt!) are keen on using org as a publication system, I’m not sure if this is the right answer, but I do think that its good during very creative phases of projects to do the work in org, mostly as I think it facilitates focusing on the current problem (through collapsing of the tree to show you just what you’re working on,) and also for working non-linearly as you can leave yourself TODO items for later action.

At the same time, if you tend to maintain org files that contain planning for more than one project, I find it cumbersome to also draft in these files. So I think “keep smaller very focused org files, and maybe do drafting in them if appropriate.”

That’s a start at least. I’ve made these changes--which are really quite subtle--and I like the way it feels, but we’ll have to see how things shake down in a few weeks. As much as I want to avoid tinkering with things--because tinkering isn’t the same as getting things done--I really do find it helpful to review processes from time to time and make sure that I’m really working as effectively as I can.

After a long time of intentionally avoiding tweaking my emacs file, I’ve gotten back into tweaking and hacking on my setup a bit in emacs land. Rather than wax philosophical about emacs and plain text, I thought I’d share a few things with you all in the hopes that this will prove helpful for you.

I’ve given some thought to publishing a git repository with my emacs files, my awesome config, and the useful parts of my bashrc files. My only hesitation is that all of these files aren’t in one repository right now, and I’d need to do some clean up to avoid publishing passwords and the like. Encouragement along this direction might be helpful in inspiring me to be a little more forthcoming in this direction.

Keybinding “Name Spaces”

I’ve begun reorganizing key-bindings in a standard pattern, in order to avoid collision of bindings in certain spaces. The problem with the “C-x C-[a-z]” bindings is that it’s hard to get really good mnemonic bindings for whatever you’re trying to do, and there are few of them. I’ve taken to putting all of my custom bindings (mostly) under “C-c [a-z],” and then grouping them together, based on mode or function.

(global-set-key (kbd "C-c o a") 'org-agenda-list)
(global-set-key (kbd "C-c o t") 'org-todo-list)
(global-set-key (kbd "C-c o p") 'org-insert-property-drawer)
(global-set-key (kbd "C-c o d") 'org-date)
(global-set-key (kbd "C-c o j") 'org-journal-entry)
(global-set-key (kbd "C-c r") 'org-remember)
(global-set-key (kbd "C-c a") 'org-agenda)

(global-set-key (kbd “C-c w s”) ‘w3m-search) (global-set-key (kbd “C-c w t”) ‘w3m-goto-url-new-session) (global-set-key (kbd “C-c w o”) ‘w3m-goto-url) (global-set-key (kbd “C-c w y”) ‘w3m-print-this-url) (global-set-key (kbd “C-c w l”) ‘w3m-print-current-url)

You can see here, org-mode related bindings and w3m related bindings. “C-c o” is wide open, and I haven’t yet found anything in that space that I’ve overwritten. Same with “C-c w”. Even though the command key-chains are a bit longer than they might be if I piled things more sporadically, I can remember them more quickly.

Org-journal is something I got from metajack, and I don’t use it as much as I should. Everything else is standard org or w3m functionality.

I suppose I should make mode-specific key-bindings so that I’m not eating away global name space for mode-specific functionality, but I’m not sure that would make things too much clearer or easier to remember.

Also I really like the (kbd ") syntax for specifying key sequences. Much easier to read and edit.

Custom File settings

A while back I pulled my customize-set variables out of my main init-file, and gave them their own file, which means my init-file isn’t quite so long, and the variables that I’m not setting.

Nevertheless, I like to set as many variables by hand with setq just so that I can be in better touch with what settings I’m changing. This code, moves custom-set variables out of main file:

(setq custom-file "~/path/to/emacs.d/custom.el")
(load custom-file 'noerror)

Window Transparency and Font Settings

At the top of my init file, I have the following four lines to set font and window transparency.

(add-to-list 'default-frame-alist '(font . "Monaco-08"))
(set-default-font "Monaco-08")
(set-frame-parameter (selected-frame) 'alpha '(86 84))
(add-to-list 'default-frame-alist '(alpha 86 84))

Note that this depends on running a composting manager like xcompmngr, and the transparency is quite subtle. With great pleasure, running this code at the begining of the init file means that emacs’ looks and behaves correctly when I start it using a plain,

emacs --daemon

command from a regular bash prompt. I’m running fairly recent (but perhaps not the actual release?) builds of emacs 23. Note that I’d had trouble getting daemonized versions of emacs to start and capture the right information about font and transparency. That seems to be resolved.

Aliases

Here’s the alaises I use to make key-commands less work to type. It’s sort of a space between “creating a key binding” and just using the function from M-x Here’s the current list:

(defalias 'wku 'w3m-print-this-url)
(defalias 'wkl 'w3m-print-current-url)

(defalias ‘afm ‘auto-fill-mode) (defalias ‘mm ‘markdown-mode) (defalias ‘rm ‘rst-mode) (defalias ‘wc ‘word-count) (defalias ‘wcr ‘word-count-region) (defalias ‘qrr ‘query-replace-regexp) (defalias ‘fs ‘flyspell-mode) (defalias ‘oa ‘org-agenda) (defalias ‘uf ‘unfill-region) (defalias ‘ss ‘server-start) (defalias ‘se ‘server-edit) (defalias ‘nf ‘new-frame) (defalias ‘eb ‘eval-buffer) (defalias ‘mbm ‘menu-bar-mode) (defalias ‘hs ‘hs-org/minor-mode)

There are a number of these that I don’t use much any more, but it’s not worth it to edit the list down.

New Modes

A few new modes that I’ve been using

yassnippet

I’ve started using yasnippet more, and I’m quite fond of it for managing and inserting little templates into files as I’m working. There’s not a lot of example code that I can share with you, as it just works, but I do have a couple of notes/complaints:

• I have to use C-i to expand snippets. The “tab” key doesn’t seem to work to expand snippets ever.
• The organization of the snippets directory is absurd. I understand how the structure of the hierarchy mirros the way modes are derived from one another, and having the expansion triggers as file names also makes sense, but it’s really hard to organize things. Do people use modes that aren’t derived from “text-mode”? Are there any? There should be a “global” directory in the snippets folder (next to text-mode) where all of the files in any number of folders beneath “global” are available in all modes.
• It’s amazing useful, and there are some things that I need to create snippets for that I haven’t. This is on my list of things to do.

w3m

w3m is an external text-mode browser that emacs hackers have written a good bridge to emacs for. What this means is you get a text-mode browser that works in emacs, but it’s speedy because page rendering happens outside of emacs.

It works, and it’s immensely use-able, though the key-bindings are a bit hard to remember and there are too many of them to change at once without completely driving yourself crazy.

I read a thread on the emacs-devel list a few months back about embedding something like uzbl inside of emacs (making emacs more like a window-manager) and I think the project presents an interesting possibility, but I think w3m succeeds because it makes the text of a website accessible within emacs.

Embedding a “real” browser in emacs, would just duplicate window manager functionality, and add complication. I think better to make a uzbl config file that was emacs-friendly, and some sort of “create emacs buffer with selected uzbl text” bridge would be nice, but anything more than that seems foolish.

My (few) w3m key-bindings are above.

nxml mode

With all this web-design work I’ve been doing, (eg. cyborg institute) I’ve needed to stray into using HTML and CSS modes. There’s this newer mode called nxml-mode which is delightful because it validates your html/xhtml/xml file on the fly (great!) but I’ve found it less than helpful for situations where I just have a snippet of HTML/XML in a given file, because it gets included later. Nonetheless, powerful stuff.

That’s about it for now. There are few other things, but I don’t feel ready to really explore them at this point, mostly because I haven’t gotten familiar enough to know if my modifications have been useful. Muse-mode, etc.

Any good emacs code that I should be looking at?

With all luck, I’ll have most of a draft of the short story I’ve been working on done by the time this goes live, but if not certainly rather soon there after. This is an exciting announcement in and of itself, but perhaps the more interesting thing is that in the process of doing this I sank into writing this story in org mode.

My general M.O. for writing for the last several years has just been to write and store the files in markdown and use whatever text editor I fancy. I write the blog this way, I write papers this way. Everything seems to work fine, there are converters for LaTeX, HTML, and the plain text format is absolutely and completely readable to people who aren’t as obsessive about text files as I am.

While I’m a huge org-mode proponent, I don’t tend to think that org-mode makes a particularly good writing environment (or haven’t, heretofore) because unless you use org-mode org files are sometimes a bit ugly, and the syntax is enough different from markdown to confuse me, and…

The general consensus, that I’ve seen is that while org-mode is indeed a great boon to the intensive-emacs user, that it’s not an ideal production editing environment. muse-mode, or my favored markdown-mode might be better if you’re actually writing text.

And then, as I got into the writing of this story, I realized that I was flipping rather seriously (and annoyingly) between my notes for the story and the story I was writing. Also, when I’m writing book-length (or conceptually book-length) work, I tend to break up the text into more manageable chapter-length or scene-length files, which is conceptually useful for me.

In a short story, it didn’t seem to make sense to break things up into more than one file, and after I’d written a couple thousand words, I realized that something needed to be done. I created a file, with some header meta-data (using the yaml form that jekyll), an org-mode statement to define custom-status words that seem relevant to the writing/editing process, and then first level headers define key scenes or breaks in the story. I’ve never written (or read, to the best of my memory) a story that required more than one level of organization (but ymmv), and then--and this is the clever part as far as I’m concerned--property drawers for notes about what happens in the scene.

Property drawers stay folded by default, and are intended to store a collection of key-value pairs, but they don’t get exported by default, and so are a good way to keep your notes and your writing together and then export, as needed when drafting is done.

Also, I’ve recently added the following to my key-binding list, which adds a property drawer to the current heading, which is indeed a good thing:

(global-set-key "\M-p" 'org-insert-property-drawer)

I’ve posted a copy of my template file for your review and edification.

Comments?

I have a confession. I last week (briefly) considered getting a Nokia N810 so that I could sync and use org-mode when I was away from my computer/laptop. The N800/810 is a small tablet that runs a Debian based operating system, which means it could run emacs, and I could write little clickable scripts that could do all of the syncing and awesomeness that I’ve grown accustomed to.

Then I realized how absurd this is, and cast it aside. My laptop is really mobile, and if I needed it to be lighter or more mobile, I could buy a new battery for it. And it has a full sized keyboard.

This is a sickness right?

In my post about my fact file I said that I was going to “try things out and see how it goes” before I posted code to see how things work in the system. Well, I think things are pretty stable (I haven’t tweaked much), so I’m going to post my remember template, for the system described in that post.

(setq org-remember-templates
   '(("data" ?d "* %^{Title} %^g \n :PROPERTIES:\n :date:
       %^t\n :cit e-key: %^{cite-key}\n :link: %^{link}\n
       :END:\n\n %x %?"  "~/org/data.org")))`

If you want to tweak further, check out the relevant section of the org manual.

Enjoy and collect facts with abandon!