When I bought what is now my primary laptop, I had intended to use the extra flexibility to learn the prevailing (industrial-grade) virtualization technology. While that project would have been edifying on its own, I also hoped to use the extra flexibility to some more consistent testing and development work.

This project spurned a xen laptop project, but the truth is that Xen is incredibly difficult to get working, and eventually the “new laptop” just became the “every day laptop,” and I let go of the laptop Xen project. In fact, until very recently I’d pretty much given up on doing virtualization things entirely, but for various reasons beyond the scope of this post I’ve been inspired to begin tinkering with virtualization solutions again.

As a matter of course, I found myself trying KVM in a serious way for the first time. This experience both generated a new list of annoyances and reminded me about all the things I didn’t like about Xen. I’ve collected these annoyances and thoughts into the following post. I hope that these thoughts will be helpful for people thinking about virtualization pragmatically, and also help identify some of the larger to pain points with the current solution.

Xen Hardships: It’s all about the Kernel

Xen is, without a doubt, the more elegant solution from a design perspective and it has a history of being the more robust and usable tool. Performance is great, Xen hosts can have up-times in excess of a year or two.

The problem is that dom0 support has, for the past 2-3 years, been in shambles, and the situation isn’t improving very rapidly. For years, the only way to run a Xen box was to use an ancient kernel with a set of patches that was frightening, or a more recent kernel with ancient patches forward ported. Or you could use cutting edge kernel builds, with reasonably unstable Xen support.

A mess in other words.

Now that Debian Squeeze (6.0) has a pv-ops dom0 kernel, things might look up, but other than that kernel (which I’ve not had any success with, but that may be me,) basically the only way to run Xen is to pay Citrix¹ or build your own kernel from scratch, again results will be mixed (particularly given the non-existent documentation,) maintenance costs are high, and a lot of energy will be duplicated.

What to do? Write documentation and work with the distributions so that if someone says “I want to try using Xen,” they’ll be able to get something that works.

KVM Struggles: It’s all about the User Experience

The great thing about KVM is that it just works. “sudo modprobe kvm kvm-intel” is basically the only thing between most people and a KVM host. No reboot required. To be completely frank, the prospect of doing industrial-scale virtualization on-top of nothing but the Linux kernel and with a wild module in it, gives me the willies is inelegant as hell. For now, it’s pretty much the best we have.

The problem is that it really only half works, which is to say that while you can have hypervisor functionality and a booted virtual machine, with a few commands, it’s not incredibly functional in practical systems. There aren’t really good management tools, and getting even basic networking configured off the bat, and qemu as the “front end” for KVM leaves me writhing in anger and frustration.²

Xen is also subject to these concerns, particularly around netowrking. At the same time, Xen’s basic administrative tools make more sense, and domU’s can be configured outside of interminable non-paradigmatic command line switches.

The core of this problem is that KVM isn’t very Unix-like, and it’s a problem that is rooted in it’s core and pervades the entire tool, and it’s probably rooted in the history of its development.

What to do? First, KVM does a wretched job of anticipating actual real-world use cases, and it needs to do better at that. For instances it sets up networking in a way that’s pretty much only good for software testing and GUI interfaces but sticking the Kernel on the inside of the VM makes it horrible for Kernel testing. Sort out the use cases, and there ought to be associated tooling that makes common networking configurations easy.

Second, KVM needs to at least pretend to be Unix-like. I want config files with sane configurations, and I want otherwise mountable disk images that can be easily mounted by the host.

Easy right?

This is the story of how I occasionally realize I exist on the continuum of “programmers,” rather than just being an eccentric sort of writer type.

::: {.contents} :::

Evidence

download-mail

I have this somewhat peculiar method of downloading email that I think works great. A few weeks ago, however, I was trying to compress things in “hot storage,” and realized that I had a problem.

For a year or so, I had been automating commits to the git repository that held all my mail. In order to effectively archive and compress some mail, I needed to do some serious rebasing to not only remove a bunch of messages from the current repository but also pull that content from the history and flatten the history somewhat.

The problem was that I had 50,000 commits and there’s simply no effective way to rebase that many commits in a reasonable amount of time, particularly given I/O limitations. So I gave up, started from (relative) scratch, and rewrote the scripts to be a little bit more smart… You know in an afternoon.

See the revised code here: download mail

ikiwiki-tasklist

I’ve written about this before in my post on my new personal organization stuff, but it’s no great announcement that I’m moving away from working in emacs’ org-mode and doing more work with ikiwiki and some hand-rolled scripts. I think org-mode is great, it just ended up getting in my way a bit and I think I can get more of what I need to get done in other ways.

I have learned a great deal from org-mode. I made the biggest leap away from org-mode when I wrote ikiwiki tasklist, which does all of the things I had been using org-mode’s agenda for. It’s not a complicated at all: look in some files for some lines that begin with specific strings and put them into a page that is the perfect task list.

See the code here: ikiwiki tasklist.

Common Lisp Weenie

“What Window Manager is that,” he asked.

“StumpWM, it’s written in Common Lisp,” I said, launching into a 30 second pitch for Stump.

My pitch about stump is pretty basic: the Common Lisp interface allows you to evaluate code during run-time without restarting the window manager or loosing state; it’s functionally like screen, which is very intuitive for window management; and it has emacs-like key-bindings, which I think work pretty well.

“So you’re a Common Lisp programmer?”

“Well not really, I mean, I know enough to get by.”

“Right.”

“Right.”

Conclusion

In several (technical writing) job interviews recently, people asked me about my programming experience, and my answer varied a lot.

I know how computer programs work, I know how people write computer programs, I understand how software testing and debugging works, I understand the kinds of designs that lead to good programs and the kinds that lead to bad software. I don’t write code--really--but I can sort of hack things together in shell scripts when I need to.

The answer to the question, these days, is “I’m a programmer in the way that most people are writers: most people are comfortable writing a quick email or a short blurb, but get stuck and have trouble really writing longer or more complicated kinds of text. Reasonably capable but not skilled.”

The above code examples work: they do what I need them to do, and particularly in the case of the mail script, they work much better than the previous iteration. I need to do more work, and I feel like I’m reaching the boundaries of what can be comfortably done in shell scripting. My next big programming project is to go through these two scripts and port them to Python and see if I can add just a little bit of additional functionality in the process.

I’m sure I’ll report to you on this as my work progresses.

I added a few lines to a script that I use to build my task list, and for the first time ever, I opened a file with code in it, added a feature, tested it, and it worked. Here’s the code with enough context so it makes sense (explained later if you don’t want to spend the time parsing it:)

ARG=`echo "$@" | sed -r 's/\s*\-[c|p|s]\s*//g'`
WIKI_DIR="`echo $ARG | cut -d " " -f 1`"
if [ "`echo $ARG | cut -d " " -f 2 | grep -c /`" = 1 ]; then
   TODO_PAGE="`echo $ARG | cut -d " " -f 2`"
elif [ "`echo $ARG | cut -d " " -f 2 | grep -c $EXT`" = 1 ]; then
   TODO_PAGE="$WIKI_DIR/`echo $ARG | cut -d " " -f 2`"
else
   TODO_PAGE="$WIKI_DIR/`echo $ARG | cut -d " " -f 2`.$EXT"
fi

This is from the section of the script that processes the arguments and options on the command line. Previously, commands were issued such that:

ikiwiki-tasklist [-c -p -s] [DIR_TO_CRAWL] [OUTPUT TODO FILE NAME]

My goal with the options was to have something that “felt like” a normal command with option switches and had a lot of flexibility. The two fields that followed: however, I didn’t provide as much initial flexibility. The directory to crawl for tasks (i.e. “[DIR_TO_CRAWL]”) was specified the way it is now, but the output file was 1) assumed to have an extension specified in a variable at the top of the script, 2) automatically placed the output file in the top level of the destination directory.

It worked pretty well, but with the advent of a new job I realized that I needed some compartmentalization. I needed to fully use the tasklist system for personal and professional tasks without getting one set of items mixed in with the other. Being able to have better control of the output is key to having full control over this.

The modification detects if the output file looks like a path rather than a file name. If it’s senses a path, it creates the task list in the path specified, with no added extension. If a file name specifies the extension, then you won’t get “.ext.ext” files. And the original behavior is preserved.

I’m a hacker by inclination: I take code that I find and figure out how to use it. Sometimes I end up writing or writing code, but I’m not really a programmer. My own code, at least until recently has tended to be somewhat haphazard and until now (more or less) I’ve not felt like I could write code from scratch that was worth maintaining and enhancing in any meaningful way.

Apparently some of that’s changed.

I’ve made a few additional changes to the scripts, but most of these feel more trivial and can be described as “I learned how to write slightly tighter shell scripts. so if you’re using it you might want to update: the ikiwiki tasklist page is up to date.

I had an item on my list of blog posts to write for a couple of years to write something reflecting on “RaceFail,” and finally a gave up, because I didn’t want to write a book, I didn’t know what to say, and I was more interested in the actual discourse itself than finding the “side of right,” in a conversation that was both way too simple and way too complex all at once.

So rather than reboot the conversation, which has ended in some senses and continues on in others, I want to start writing a bit here about race and representation in fiction, but also discussing the way that conversations transpire online. Here’s part one. I’ll figure out some way to index them all together once they’re posted and assembled.

I wrote this scene a while back where a character who grew up on a small¹ outpost visits a space ship. Given relativistic space travel, from the character’s perspective, the crew of the space ship are 750 years old or so, despite being in their subjective early forties. That means the character’s 31st-great-grandparents (roughly) were cousins of the people he’s looking at.

He notices a few things: the people on the ship are all taller than he is and also taller than everyone from the outpost. He also notices that there’s more more skin tone variation amongst the people on the ship than there is among the people in the outpost.

There are a bunches of problems with this story. Including the fact that its not finished and that there are parts of the execution that I think need a lot work. But this part, I quite like. For this story (and I think in general,) I’ve drawn the following conclusion:

• Race is temporally constrained. We understand racial difference and our own racial experiences in terms of our current reality. This changes.
• The aspects of race which are the result of lineage (skin color, bone structure,) are likely to change over time as lineages continue. We can assume that these kinds of changes will be pronounced in smaller populations over longer periods of time.

To a large extent the tension between the “outpost people” and the “ship people” is the core of the conflict in this story. I’ve been thinking in this story about the impact of colonialism (and race as a result) on societies and political outlook. It’s almost certainly not perfect, but I enjoy the possibilities, the story has its moments, and I’m finding the theory building productive.

I’m circling around a point: in-story diversity, particularly, diversity that reflects late 20th/early 21st century notions of difference alone cannot further thought race and racism. In other words, diversity is not criticism. There are many ways to productively further the discussion of difference in (genre) fiction, lets not stop with representation.

I’ll be writing more about this in the future. Comments are very welcome!

I was having a conversation with a (now former) coworker (a while ago) about the future of shared file systems, unstructured organization and management, and access control. What follows are a collection of notes and thoughts on the subject that have stuck with me.

Let’s start with some general assumptions, premises, ideas:

• File system hierarchies are dead or dying. To have a useful file system hierarchy the following qualities are essential:

• Every piece of data needs to belong in one location and only one location.

• Every container (e.g. directory or folder) needs to hold at least two objects.

• Hierarchy depth ought to be minimized. Every system can use two levels. After the second level, each additional level should only be added a very large number of additional objects are added to the system. If you have 3 functional levels and less than 1000 objects, you might be in trouble.

  As you might imagine, this is very difficult to achieve, and the difficulty is compounded by huge amounts of legacy systems, and the fact that “good enough is good enough,” particularly given that file organization is secondary to most people’s core work.

  While there are right ways to build hierarchical structure for file system data, less structure is better than more structure, and I think that groups will tend toward less over time.

• Access control is a lost cause. Legacy data and legacy practices will keep complex ACL-based systems for access control in place for a long time, but I think it’s pretty clear that for any sort of complex system, access control isn’t an effective paradigm. In some ways, access control is the last really good use of file system hierarchies. Which is to say, by now the main use of strong containers (as opposed to tags) is access control.

  I don’t think that “enterprise content management”-style tools are there, yet. I suspect that the eventual solution to “how do I control access to content” will either: be based on a an cryptography key system which will control access and file integrity, or there will be a class of application, a la ECMS, with some sort of more advanced abstracted file system interface that’s actually use-able.

I’m betting on encryption.

• Tagging and search are the ways forward. In many cases, the location of files in hierarchy help determine the contents of those files. If there are no hierarchies then you need something more useful and more flexible to provide this level of insight.

• Great search is a necessity. Luckily it’s also easy. Apache Solr/Lucene, Xapian, and hell Google Search Appliances make great search really easy.

• Some sort of tagging system. In general, only administrators should be able to create tags, and I think single tag-per object (i.e. categories) versus multiple tags per object should be configurable on a collection-by-collection.

  Tag systems would be great for creating virtualized file system interfaces, obviating the need for user-facing links, and leveraging existing usage patterns and interfaces. It’s theoretically possible to hang access control off of tag systems but that’s significantly more complicated.

  One of the biggest challenges with tag systems is avoiding recapitulating the problems with hierarchical organization.

The most difficult (and most interesting!) problem in this space is probably the access control problems. The organizational practices will vary a lot and there aren’t right and wrong answers. This isn’t true in the access control space.

Using public key infrastructure to encrypt data may be an effective access control method. It’s hard replicate contemporary access control in encryption schemes. Replicating these schemes may not be desirable either. Here are some ideas:

• By default all files will be encrypted such that only the creator can read it. All data can then be “world readable,” as far as the storage medium and underlying file systems are concerned.

• The creator can choose to re-encrypt objects such that other users and groups of users can access the data. For organizations this might mean a tightly controlled central certificate authority-based system. For the public internet, this will either mean a lot of duplicated encrypted data, or a lot of key chains.

• We’ll need to give up on using public keys as a method of identity testing and verification. Key signing is cool, but at present it’s complex, difficult to administer, and presents a significant barrier to entry. Keys need to be revocable, particularly group keys within organizations.

  For the public internet, a some sort of social capital or network analysis based certification system will probably emerge to supplement for strict-web-of-trust based identity testing.

• If all data is sufficiently encrypted, VPNs become obsolete, at least as methods for securing file repositories. Network security is less of a concern when content is actually secure. Encryption overhead, for processing isn’t a serious concern on contemporary hardware.

Thoughts?

I’ve written before about different types of documentation, and the different purposes and goals that each type services. Rather than rehash what documentation is, I’m interested in using this post to think about ways of managing and organizing the documentation process to produce better documentation more easily, with the end goal of being able to increase both maintainability and usability of documentation resources.

Different groups of users--say: administrators, end-users, and developers--interact with technology in overlapping but distinct ways. For some technologies, the differences between the classes of users is not significant and one set of documentation is probably good do every one, plus or minus a very small set. In most other cases, multiple resources are required to be able to address the unique needs of different user groups. Figuring out effective ways to address the different kinds of questions that various groups of users ask, but in a way that makes sense to those users is often the primary challenge in writing documentation.

Having said that, writing different sets of documentation for different users is a lot of work, but given time its not insurmountable. The problem is after six months or more (say,) or a couple of new releases when its time to update the documentation, there are three manuals to update instead of one. This is pretty much horrible. Not only is it more work, but the chances for errors skyrockets, and it’s just a mess.

The solution, to my mind, is to figure out ways to only ever have to write one set of documentation. While it might make theoretical sense to split the material into multiple groups, do everything you can to avoid splitting the documentation. Typically, a well indexed text can be used by multiple audiences if its easy enough for users to skip to read only the material they need.

The second class of solutions revolves around taking a more atomic approach to writing documentation. In my own work this manifests in two ways:

• Setting yourself up for success: understanding how software is going to be updated, or how use is likely to change over time allows you to construct documents that are organized in a way that makes them easy to update. For example: Separate processes from reference material, and split up long processes into logical chunks that you can interlink to remove redundancies.

  Unfortunately, in many cases, it’s necessary to learn enough about a project and the different use patterns before you have the background needed to predict what the best structure of the documentation ought to be.

• Separate structure from content: This is a publishing system requirement, at the core, but using this kind of functionality must be part of the writer’s approach. Writers need to build documentation so that the organization (order, hierarchy, etc.) is not implicit in the text, but can be rearranged and reformed as needed. This means writing documentation “atoms” in a structurally generic way. Typically this also leads to better content. As a matter of implementation, documentation resource would require a layer of “meta files” that would provide organization that would be added at build time.

In effect this approach follows what most systems are doing anyway, but in practice we need another processing layer. Sphinx is pretty close in many ways but most document formats and build systems don’t really have support for this kind of project organization (or they require enough XML tinkering to render them unfeasible.) Once everything’s in place and once all of the atoms exist, producing documents for a different audience is just a matter of compiling a new organization layer and defining an additional output target.

This also produces problems for organization. If content is segregated into hundreds of files for a standard-book-length manual (rather than dozens, say) keeping the files organized is a challenge. If nothing else, build tools will need to do a lot more error checking and hopefully documentation writers will develop standard file organizations and practices that will keep things under control.

Thoughts? Onward and Upward!

It’s relatively straight forward to think about knitting in terms of creating two dimensional shapes. Most of us start by knitting something “easy”¹ like a scarf. From there it’s easy enough to teach knitters to create a never ending variety of polygons. This, however, misses what I think of as the really cool part of knitting. I think the way to understand how knitting works, to be able to knit things that more closely resemble what you want, and to have the most fun knitting is to always think about knitting as three dimensional.

This isn’t an elaborate argument in favor of circular knitting: that argument has been fairly well made and I’ll recount my favorite points on request, but circular knitting is a great technique and knitting in three dimensions is an entire practice.

Knitting Gestalts / Knitting Shapes

I’ve written about this before but one of the best parts about sweater knitting is thinking about how the sweater--the whole object--comes together into a garment. Rather than knitting a collection of flat pieces that can be sewn into a garment (tailoring) knitting lets you build and shape garments with various seamless and nearly-seamless methods.

I sometimes describe this kind of knitting as “architectural,” but the key (for me) is thinking about the entire object as a whole. There’s something that’s nearly magical that happens when you can take a few rows curled up on a circular needle and see in your mind fits into the object that you’re knitting. The process of using knitting stitches, increases and decreases to get from the former to the later is relatively trivial if have can think about the entire object (a “knitting gestalt”) in three dimensions in your mind.

Knitting Mechanics

If “knitting gestalts” provide a top-down perspective on knitting, I think there’s a “bottom up” three dimensional perspective that is important when thinking about how stitches fit together. While a big part of knitting has to do wit the shapes and forms, the textures, drape and “hand” of the fabric all have a lot to do with the final evaluation of the object. To understand drape and texture, it’s important to consider the properties of individual knitting stitches and the effects of yarn weight/texture, needle size, and personality of the knitter. The second part (yarn type, needle size, knitting style) is pretty common, the first (knitting stitch) is less so.

I have a favorite example of this kind of thinking. I’m not sure where I learned this but it’s suck with me:

Knitted fabric typically curls. This happens because the “purl side” of the knitted stitch has a greater surface area than the “knit side,” which causes unaltered stocking stitch to roll up. At the same time, the “knit” side of the stitch is a little bit wider than the “purl” side of the stitch, so the edges will curl in. The way to counteract this, is to mix knit-and-purl stitches on the same row to balance the surface areas out and thus counteract the effects. Think about ribbing and seed stitch… Think about knit and purl patterns and how they change the tendency of the fabric to roll. Think about the path of the yarn through a knitting stitch.

See? Isn’t is cool?

In Conclusion

Whatever kind of knitting you want to do is fine with me: I don’t care to tell anyone that the way they knit is wrong. At the same time, I don’t think there’s any sense in being afraid of your knitting: knitting is great fun and I think once you know the basics most knitters can knit just about everything. So my goal in this post, and in all of my knitting posts, is to share my own process and encourage you (all) to branch out in your own work.

Have fun!

Onward and Upward!

My friend Jo wrote a post a while ago that addressed the subject of building an audience for your scholarly work. You can read the post on her blog, here.

One of the things that I think Jo is really great at is thinking practically about academic careers and trajectories in light of the current academic job market. While people working in traditional academic spaces and on a traditional academic course have a different set of challenges than folks like me, her points still resonate.

How do you build networks and audiences? Two things:

1. You talk to people.

Audiences are built on relationships. While we might like to think that writers and scholars are able to attract audiences purely on the basis of their work, in practice additional work is required.

2. You make sure you have something to show for yourself.

Everyone’s got ideas, and projects that they’d like to work on. People love to talk about their ideas. Success, I think, comes when you have something to show for yourself and your projects, and give people some level of confidence that your can make good on your ideas.

In sort, write more, publish more. While quality matters some, being more than someone to talks well at parties is really important.

I think this approach is useful for people doing any kind of creative or intellectual work that engages an audience, but I’m interested in your thoughts.

A couple of people, cwebber and Rodrigo have (comparatively recently) switched to using StumpWM as their primary window managers. Perhaps there are more outside of the circle of people I watch but it’s happened enough to get me to think about what constitutes software minimalism.

While StumpWM is a minimal program in terms of design and function; however, in terms of ram usage or binary size, it’s not particularly lightweight. Because of the way Common Lisp works, “binaries” and RAM footprint is in the range of 30-40 megs. Not big by contemporary standards, but the really lightweight window managers can get by with far less RAM.

In some senses this is entirely theoretical: even a few years ago, it wasn’t uncommon for desktop systems to have only a gig of ram, so the differences would hardly have been noticeable. Now? Much less so. Until 2006 or so, RAM was the most performance effecting limited resource on desktop system, since then, even laptops have more than enough for all uses. Although Firefox challenges this daily.

Regardless, while there may be some link between binary size and minimalism, I think it’s probably harmful to reduce minimalism and simplicity to what amounts to an implementation detail. Let’s think about minimalism more complexly. For example:

Write a simple (enough) script in Python/Perl and C. It should scan a file system and change the permissions of files such that they match the permissions of the enclosing folder, but not change the permissions of a folder if it’s different from it’s parent. Think of it as “chmod -R” except from the bottom up. This is a conceptually simple task and it wouldn’t be too hard to implement, but I’m not aware of any tool that does this and it’s not exactly trivial (to implement or in terms of its resource requirements.)

While the C program will be much more “lightweight,” and use less RAM during while running, the chances are that the Python/Perl version will be easier to understand and use much more straightforward logic. The Python/Perl version will probably take longer to run and there will be some greater overhead for the Python/Perl runtime. Is the C version more minimal because it uses more RAM? Is the Perl/Python program more minimal because it’s interface and design is more streamlined, simple and easier to use?

I’m not sure what the answer is, but lets add the following factor to our analysis: does the “internal” design and architecture of software affect the minimalism or maximalism of the software?

I think the answer is clearly yes, qualified by “it depends” and “probably not as much as you’d think initially.” As a corollary as computing power increases the importance of minimalist implementations matters less generally, but more in cases of extremely large scale which are always already edge cases.

Returning for a moment to the question of the window manager, in this case I think it’s pretty clear: StumpWM is among the most minimal window managers around, even though it’s RAM footprint is pretty big. But I’d love to hear your thoughts on this specifically, or technological minimalism generally.

I’m a huge fan of emacs’ org-mode on so many levels: as an IDE for knowledge workers, as a task management system, as a note taking system, and as the ideal basic mode for so many tasks. However, I’ve been bucking against org-for a number of tasks recently. The end result is that I’m becoming less org-dependent. This post is a reflection on how I’ve changed the way I work, and how my thinking has changed regarding org-mode.

Fair warning: this is a really geeky post that has a somewhat specialized context. If you’re lost or bored. check back later in the week.

The Perils of Org

The problem I keep running into with org is that I really don’t prefer to work in org-mode.¹ Org is great and very flexible, but I don’t like that it means that all text-based work is dependent on emacs. My brain is already wired for Markdown and reStructured Text from years of blogging and work projects respectively.

And then there’s this organization problem. There are two ways you can organize content in org-mode. The first is to just dump every thing in one org-mode file and use the hierarchical outlining to impose organization to organize everything. The second is to have every project inside of it’s own file and use outlining incidentally as the project needs it. Content aggregation happens in the agenda.

The problem with the “large files” approach is that you end up with a small handful of files with thousands of lines and imposing useful organization is difficult (too many levels and things get buried; not enough and inevitably your headings aren’t descriptive enough and you get confused. Furthermore, I end up living in clone-indirect-buffer-other-window’d and org-narrow-to-subtree’d buffers, which is operationally the same as having multiple files it just takes longer to set up.

The problem with the other approach, having lots of different files, is that I have a hard time remembering what is in each file, or in logically splitting big projects into multiple files. The agenda does help with this, but the truth is that the kinds of org-headings for organization and tasks are not always the same kinds of headings that make sense for the project itself. I often need more tasks than organizational divides in a project. I tried this approach a couple of times, and ended up with useless mush in my files.

Typically, I can never make the “lots of file approach” really work, and the big files problem lead me to general avoidance of everything. Not good. The key to success here is good aggregation tools.

Hodgepodge

In response, I’ve made a couple of tweaks to how I’m doing… pretty much everything. That is:

• I’ve moved most of my open projects into a locally ruining and compiling ikiwiki instance. Both laptops have this setup, and there’s a central remote to keep both (all?) machines in sync.
• I’m using ikiwiki tasklist to basically replicate the functions of org-agenda. Basically this crawls the entire wiki looking for lines that begin with certain keywords and generates a “todo” page based on these notes. Really simple, incredibly useful and it solves much of my aggregation needs.
• I still have some stuff in org-mode: notes for the nearly-finished novel, lots of random old (legacy) data, 12 various open tasks, and org-capture. I’m thinking of pointing various org-capture templates at files in the wiki but haven’t gotten there yet.
• I’ve basically taken the “lots of little files,” approach to my writing and work. I’ve not over-leaded the system yet. Each major project gets a page in the root level of the wiki for overview and planing, and then sub-pages for all related project files (if/as needed)
• It turns out that the markdown-mode for emacs has gotten a few improvements since the last time I downloaded the file, including better support for wiki-links that are mostly compatible with ikiwiki. Also from the same developer deft which implements a pretty nifty incremental search for text files in a given directory. So between these tools, ikiwiki, and the ikiwiki-tasklist there’s support for the most important things.
• In terms of publishing, beyond ikiwiki for tychoish.com and the personal organization instance, I have a couple of other smaller wikis (also ikiwiki powered,) and I’ve been playing with Sphinx as publishing for more structured documents and resources (i.e. documentation, novels, and collections,) particularly those that need multiple formats and presentations.

I’m sure there will be more shifts in the future, I’m sure. I think this is a good start. Thoughts?