I spent a lot of time in the past few months thinking about “automation,” as a project to take things that take a long time and require a lot of human intervention into things that just do themselves, and I think this is the wrong approach.

While total automation is an admirable, it’s difficult, both because it requires more complex software to deal with edge cases, but also because it’s hard to iterate into a fully automated solution.

Let’s back up for a moment and talk about automation in general.

Computers are great at automating things. When you figure out how exactly to accomplish something digitally (i.e. polling an information source for an update, transforming data, testing a system or tool,) writing a program to perform this function is a great idea: not only does it reduce the workload on actual people (i.e. you.) I think the difference between people who are “good with computers,” and people who are “great with computers,” is the ability to spot opportunities for these kinds of automations, and potentially implement them..

To my mind the most important reason to automate tasks is to ensure consistency and to make it more likely that tedious tasks get done.

Having said this, rather than develop complete task automations for common functions, the better solution is probably to approach automation on the bottom up: instead of automating a complete process, automate smaller pieces particularly the most repetitive and invariable parts, and then provide a way for people to trigger the (now simplified) task.

The end result, is a system that’s more flexible easier to write, and less prone to failure under weird edge cases. Perhaps this is a manifestation of “worse is better” also.

Thoughts?

Onward and Upward!

This post provides a few quick overviews of cool bits of shell script that I’ve written or put together recently. Nothing earth shattering, but perhaps interesting nonetheless.

Making a Bunch of Symbolic Links.

I have two collection of Maildir folders to store my mail. One in a ~/work/mail folder for my work mail, and ~/mail for my personal projects. I want the work mailboxes to be symbolically linked to the personal ones (some work email, notably from GitHub comes to a personal address, and I want to be able to refile as necessary.) I moved the work maildirs last week, so I needed to create about 15 symbolic links, and so I wrote the following little loop:

cd ~/mail
for i in `find ~/work/mail -maxdepth 1 -mindepth 1`; do
   ln -s $i work.`echo $i | cut -d / -f 6`
done

Should be useful, no?

Commit all Git Changes

For a long time, I used the following bit of code to provide the inverse operation of “git add .”. Where “git add .” adds all uncommited changes to the staging area for the next commit, the following commit automatically removes all files that are no longer present on the file-system from the staging area for the next commit.

if [ "`git ls-files -d | wc -l`" -gt "0" ]; then
  git rm --quiet `git ls-files -d`
fi

This is great if you forget to use “git mv” or you delete a file using rm, you can run this operation and pretty quickly have git catch up with the state of reality. In retrospect I’m not really sure why I put the error checking if statement in there.

There are two other implementations of this basic idea that I’m aware of:

for i in `git ls-files -d`; do
  git rm --quiet $i
done

Turns out you can do pretty much the same thing with the following statement using the xargs command and you end up with something that’s a bit more succinct:

git ls-files --deleted -z | xargs -0 git rm --quiet

I’m not sure why, I think it’s because I started being a Unix nerd after Linux dropped the argument number limit, and as a result I’ve never really gotten a chance to become familiar with xargs. While I sometimes sense that a problems is xargs shaped, I almost never run into “too many arguments” errors, and always attempt other solutions first.

A Note About xargs

If you’re familiar with xargs skip this section. Otherwise, it’s geeky story time.

While this isn’t currently an issue on Linux, some older UNIX systems (including older versions of Linux,) had this limitation where you could only pass a limited number of arguments to a command. If you had too many, the command would produce an error, and you had to find another way.

I’m not sure what the number was, and the specific number isn’t particularly important to the story. Generally, I understand that this problem would crop up when attempting to take the output of a command like find and piping or passing it to another command like grep or the like. I’m not sure if you can trigger “too many arguments” errors with globbing (i.e. *) but like I said this kind of thing is pretty uncommon these days.

One of the “other ways” was to use the xargs command which basically takes very long list of arguments and passes them one by one (or in batches?) to another command. My gut feeling is that xargs can do some things, like the above a bit more robustly, but that isn’t experimentally grounded. Thoughts?

Onward and Upward!

*tl;Dr>* I got one of those nifty laptop risers that puts your laptop up closer to eye level, and it has pretty much improved all of my interactions with computers a thousand fold and it’s made it possible for me to effectively use two screens. This post explores this.

One of my coworkers had a laptop stand she wasn’t using and I asked to borrow it for an afternoon, and my neck stopped hurting. I never thought my neck hurt before, but apparently it does.

Or did.

But there’s more: for years now I’ve kept an extra monitor around (and had one at work) but the truth is that I have never really felt like I’ve been able to get the most out of an external monitor.

Somehow, putting my laptop 4 inches in the air was the little change that made everything better. The laptop is generally on the left of the external monitor, and I have task lists, notes buffers, the chat window, and my status logging window on the laptop, and then three windows on the external (emacs buffer, terminal, emacs buffer) on the right. My primary focus centers between the monitors, but probably edges slightly toward the external, most of the time.

Also, I discovered that I--apparently--have a slight processing/attention defect whereby I find it painful and difficult to focus on things that are happening on the right side of the screen for any amount of time. Which is weird because my right eye has always been noticeably stronger. I’ll ponder this more later.

My virtual desktops for email and web browsing are a bit less rigid, but the same basic idea. Somehow it seems to work. I’ve done a little bit of work recently to get the layouts right, to minimize the impact of the window management of most context switching (scripting various transitions, saving layouts, etc.) In all things are going great.

It strikes me that I’ve not posted here even a little about my setup in a while. The truth is that it’s not terribly surprising and I’ve not changed very much recently. I’m back to one laptop, and as anxious as having one laptop makes me sometimes (I fear the lack of redundancy,) not having to keep it synced makes life easier. I’ve put some time into doing a little bit of polish on all of little bits of configuration/code that I have that makes my computing world go around, but mostly it’s pretty good.

It’s nice, and I’d write more about it, but I want to get back to getting things done around here. Exporting and exploring some of this stuff in greater depth is definitely on my list, so hang in there, and if there’s something you particularly want to see, be in touch.

I have an Aeron chair at my desk at home. Confession.

I got it in April when I moved to New York City. The only piece of furniture that I had that I couldn’t move in my (now former) car was my desk chair. I found a good deal on an Aeron chair and I rationalized to myself that the cost of the chair was actually about the cost of movers. Savings right?

It also helped, that I was leaving a job where I had an Aeron chair in my office, and I knew that in the short term I would be working from home. While my old desk chair was (and is) quite nice, it’s not quite the same. Sit in an Aeron chair for a couple of two years, and it’s hard to go back. I’ve sat in other chairs since then, and it’s never quite the same.

Having said that, after a cleaning incident today, I would like to collect a few gripes about the Aeron chair for your consideration.

• The assembly right beneath the chair collects dust and dirt in a proportion that doesn’t seem quite possible. It’s clearly an artifact of the mesh, and likely a commentary on the air circulation of my apartment.

  Regardless, dusting nightmare.

• The arms scuff and scratch on desks, if the bottom of the desk isn’t completely smooth. This isn’t an actual problem: the chair still works fine and is as comfortable as ever, but it’s a annoying.

I’ve never looked at the underside of a desk before seriously. With every other chair I’ve either ordered a variant sans arms, or I’ve take then arms off as soon as possible.

The Aeron arms are low enough that they’ve never bothered me, so I thought “might as well.” But it’s still annoying.

That’s all.

When I bought what is now my primary laptop, I had intended to use the extra flexibility to learn the prevailing (industrial-grade) virtualization technology. While that project would have been edifying on its own, I also hoped to use the extra flexibility to some more consistent testing and development work.

This project spurned a xen laptop project, but the truth is that Xen is incredibly difficult to get working, and eventually the “new laptop” just became the “every day laptop,” and I let go of the laptop Xen project. In fact, until very recently I’d pretty much given up on doing virtualization things entirely, but for various reasons beyond the scope of this post I’ve been inspired to begin tinkering with virtualization solutions again.

As a matter of course, I found myself trying KVM in a serious way for the first time. This experience both generated a new list of annoyances and reminded me about all the things I didn’t like about Xen. I’ve collected these annoyances and thoughts into the following post. I hope that these thoughts will be helpful for people thinking about virtualization pragmatically, and also help identify some of the larger to pain points with the current solution.

Xen Hardships: It’s all about the Kernel

Xen is, without a doubt, the more elegant solution from a design perspective and it has a history of being the more robust and usable tool. Performance is great, Xen hosts can have up-times in excess of a year or two.

The problem is that dom0 support has, for the past 2-3 years, been in shambles, and the situation isn’t improving very rapidly. For years, the only way to run a Xen box was to use an ancient kernel with a set of patches that was frightening, or a more recent kernel with ancient patches forward ported. Or you could use cutting edge kernel builds, with reasonably unstable Xen support.

A mess in other words.

Now that Debian Squeeze (6.0) has a pv-ops dom0 kernel, things might look up, but other than that kernel (which I’ve not had any success with, but that may be me,) basically the only way to run Xen is to pay Citrix¹ or build your own kernel from scratch, again results will be mixed (particularly given the non-existent documentation,) maintenance costs are high, and a lot of energy will be duplicated.

What to do? Write documentation and work with the distributions so that if someone says “I want to try using Xen,” they’ll be able to get something that works.

KVM Struggles: It’s all about the User Experience

The great thing about KVM is that it just works. “sudo modprobe kvm kvm-intel” is basically the only thing between most people and a KVM host. No reboot required. To be completely frank, the prospect of doing industrial-scale virtualization on-top of nothing but the Linux kernel and with a wild module in it, gives me the willies is inelegant as hell. For now, it’s pretty much the best we have.

The problem is that it really only half works, which is to say that while you can have hypervisor functionality and a booted virtual machine, with a few commands, it’s not incredibly functional in practical systems. There aren’t really good management tools, and getting even basic networking configured off the bat, and qemu as the “front end” for KVM leaves me writhing in anger and frustration.²

Xen is also subject to these concerns, particularly around netowrking. At the same time, Xen’s basic administrative tools make more sense, and domU’s can be configured outside of interminable non-paradigmatic command line switches.

The core of this problem is that KVM isn’t very Unix-like, and it’s a problem that is rooted in it’s core and pervades the entire tool, and it’s probably rooted in the history of its development.

What to do? First, KVM does a wretched job of anticipating actual real-world use cases, and it needs to do better at that. For instances it sets up networking in a way that’s pretty much only good for software testing and GUI interfaces but sticking the Kernel on the inside of the VM makes it horrible for Kernel testing. Sort out the use cases, and there ought to be associated tooling that makes common networking configurations easy.

Second, KVM needs to at least pretend to be Unix-like. I want config files with sane configurations, and I want otherwise mountable disk images that can be easily mounted by the host.

Easy right?

A couple of people, cwebber and Rodrigo have (comparatively recently) switched to using StumpWM as their primary window managers. Perhaps there are more outside of the circle of people I watch but it’s happened enough to get me to think about what constitutes software minimalism.

While StumpWM is a minimal program in terms of design and function; however, in terms of ram usage or binary size, it’s not particularly lightweight. Because of the way Common Lisp works, “binaries” and RAM footprint is in the range of 30-40 megs. Not big by contemporary standards, but the really lightweight window managers can get by with far less RAM.

In some senses this is entirely theoretical: even a few years ago, it wasn’t uncommon for desktop systems to have only a gig of ram, so the differences would hardly have been noticeable. Now? Much less so. Until 2006 or so, RAM was the most performance effecting limited resource on desktop system, since then, even laptops have more than enough for all uses. Although Firefox challenges this daily.

Regardless, while there may be some link between binary size and minimalism, I think it’s probably harmful to reduce minimalism and simplicity to what amounts to an implementation detail. Let’s think about minimalism more complexly. For example:

Write a simple (enough) script in Python/Perl and C. It should scan a file system and change the permissions of files such that they match the permissions of the enclosing folder, but not change the permissions of a folder if it’s different from it’s parent. Think of it as “chmod -R” except from the bottom up. This is a conceptually simple task and it wouldn’t be too hard to implement, but I’m not aware of any tool that does this and it’s not exactly trivial (to implement or in terms of its resource requirements.)

While the C program will be much more “lightweight,” and use less RAM during while running, the chances are that the Python/Perl version will be easier to understand and use much more straightforward logic. The Python/Perl version will probably take longer to run and there will be some greater overhead for the Python/Perl runtime. Is the C version more minimal because it uses more RAM? Is the Perl/Python program more minimal because it’s interface and design is more streamlined, simple and easier to use?

I’m not sure what the answer is, but lets add the following factor to our analysis: does the “internal” design and architecture of software affect the minimalism or maximalism of the software?

I think the answer is clearly yes, qualified by “it depends” and “probably not as much as you’d think initially.” As a corollary as computing power increases the importance of minimalist implementations matters less generally, but more in cases of extremely large scale which are always already edge cases.

Returning for a moment to the question of the window manager, in this case I think it’s pretty clear: StumpWM is among the most minimal window managers around, even though it’s RAM footprint is pretty big. But I’d love to hear your thoughts on this specifically, or technological minimalism generally.

This post is mostly just an overview of Epistle by Matteo Villa, which is--to my mind--the best Android note taking application ever. By the time you read this I will have an Android Tablet, but it’s still in transit while you read this and that’s a topic that dissevers it’s own post.

Epistle is a simple notes application with two features that sealed the deal:

1. It knows markdown, and by default provides a compiled rich text view of notes before providing a simple notes editing interface. While syntax highlighting would be nice, we’ll take what we can get.

2. It’s a nice, simple application. There’s nothing clever or fancy going on. This simplicity means that the interface is clean and it just edits text.

For those on the other side there’s Paragraft that seems similar. While in my heart of hearts I’m probably still holding out for the tablet equivalent¹ of emacs. In the mean time, I think developing a text editing application that provide a number of paradigmatic text editing features and advances for the touch screen would be an incredibly welcome development.

In the end there’s much work to be done, and the tools are good enough to get started.

I have a question for my system administrator readers. And maybe the rest of you as well.

I run a web server on my laptop that hosts about 8 test sites. Nothing special: mostly test and development sites for various public sites, but from time to time I think, “shit, wouldn’t it be nice if I could just give someone a link to this.” My solution is generally to copy whatever it is that I’m working on up to the server that runs this website, and while that generally work just fine, it could be better.

So here’s what I’m thinking:

I’d like to be able to hook my laptop up to the internet and be able to let people access (some) of the content running on this web server. I don’t want it to be automatic, or open my entire machine to the world (though… I could secure it, I suppose.) The options I’ve considered.

• I set up a VPN that I can connect to the public server (that hosts this website) from the laptop, and I have a virtual host (or set of virtual hosts) that proxy requests to the laptop. Wherever I am, it works. I’m not worried about the bandwidth or the strain on the server given the usage pattern I’m expecting.

  • Pros: Simple, Secure, works even if I’m on a weird local network. Potentially useful for other kinds of nifty hacking including tunneling all traffic through the VPN on insecure connections.
  • Cons: Way complex, and I’m not sure if it will work. I’ll need to set up VPN software. And it’s total overkill.

• Some sort of scripted dynamic DNS solution, probably involving running my own DNS server.

  • Pros: less proxy madness. Pretty simple.
  • Cons: running a DNS server. Won’t work on some (most) local networks.

• There has to be some sort of alternate approach using a minimalist tunneling solution. There are a few of them, I think they’re nifty, and it would probably be perfect. I’m just not sure what it is.

  …and then half the night later, I finished deploying the VPN. I have to say that I’m really pleased with it:

• It can (and has) replaced my ssh tunnels for sending email. That’s pretty great.

• The web server stuff works, though I don’t have anything really up there yet. I feel like I need some sort of access restriction method, but I don’t really like any of the options. HTTP Auth is annoying rather than protective, SSL is terribly imperfect and fussy, host based control isn’t very tight.

• I think I will be able to finally sacrifice a laptop to the “homeserver” because aside from dis/re-enabling the “sleep on laptop close” function. If needed it’ll be dead simple to convert a sever laptop to a mobile laptop.

Thoughts?

I’ve been using FBReaderJ to read .epub files on my tablet recently, and I discovered a nitfty feature: you can adjust the screen’s brightness by dragging your finger up or down the left side of the screen. Immediately this felt like discovering a new keybinding or a new function in emacs that I’d been wishing for a while time. Why, I thought, aren’t there more tricks like this?

The iPhone (and the iPad by extension) as well as Android make two major advances over previous iterations of mobile technology. First, they’re robust enough to run “real” programs written in conventional programming environment. Better development tools make for better applications and more eager developers (which also makes for better applications.) Second, the interfaces are designed to be used with fingers rather than stylus (thanks to capacitive touch screens) and the design aesthetic generally reflects minimalist values and simplicity. The mobile applications of today’s app stores would not work if they were visually complex and had multi-tiered menus, and hard to activate buttons.

The tension between these two features in these platforms makes it difficult to slip nifty features into applications. Furthermore, th economy of application market places does not create incentives for developers to build tools with enduring functionality. The .epub reader I mentioned above is actually free software.¹ I write a couple of posts a while back on innovation (one and two) that address the relationship between free software and technological development but that’s beside the point.

Given this, there are two major directions that I see tablet interfaces moving toward:

1. Tablet interfaces will slowly begin to acquire a more complete gestural shorthand and cross-app vocabulary that will allow us to become more effective users of this technology. Things like Sywpe are part of this, but I think there are more.

2. There will be general purpose systems for tablets that partially or wholly expect a keyboard, and then some sort of key-command system will emerge. This follows from my thoughts in the “Is Android the Future of Linux?” post.

I fully expect that both lines of development can expand in parallel.

This post is a continuation of my human solution to IT and IT policy issues series. This post discusses a couple of ideas about “enterprise” software, and its importance the kind of overall analysis of technology that this posts (and others on this site) engage in. In many ways this is a different angle on some of the same questions addressed in my “Caring about Java” post: boring technologies are important, if not outright interesting.

There are two likely truths about software that make sense upon reflection, but are a bit weird when you think about it:

1. The majority of software is used by a small minority of users. This includes software that’s written for and used by other software developers, infrastructure, and the applications which are written for “internal use.” This includes various database, CRM, administrative tools, and other portals and tools that enterprise uses.
2. Beautiful and intuitive interfaces are only worth constructing if your software has a large prospective userbase or if you’re writing software where a couple of competing products share a set of common features. Otherwise there’s no real point to designing a really swanky user interface.

I’m pretty sure that these theories hold up pretty well, and are reasonably logical. The following conclusions are, I think, particularly interesting:

• People, even non-technical users, adjust to really horrible user interfaces that are non-intuitive all the time.

• We think that graphical user interfaces are required for technological intelligibility, while the people who design software use GUIs as minimally as possible, and for the vast majority of software the user interface is the weakest point.

  The obvious questions then, is: why don’t we trust non-technical users with command lines? Thoughts?