File System Metaphors

The file system is dead. Long live the File system.

We live in an interesting time. There are two technologies that aim to accomplish two very goals. On the one hand we have things like Amazon’s S3, Hadoop, NoSQL, and a host of technologies that destroy the file system metaphor as we know it today. The future, if you believe it, lays in storing all data in some sort of distributed key/value store-based system. And then, on the other hand we have things like “FUSE” that attempt to translate various kinds of interfaces and data systems onto the file system metaphor.

Ok, so the truth is that the opposition between the “lets replace file systems” with non-file based data stores folks and the “lets use the file system as a metaphor for everything,” is totally contrived. How data is stored and how we interact with data are very different (and not always connected) problems.

Let’s lay down some principals:

• There are (probably) more tools to interact with, organize, manage, and manipulate files and file system objects than there are for any other data storage system in contemporary technology.

• Most users of computers have some understanding of file systems and how they work, though clearly there are a great diversity of degrees here.

• In nearly every case, only one system can have access to a given file system at a time. In these days of such massive parallel computing, the size of computer networks, (and the associated latency) this has become a rather substantial limitation.

• From the average end user’s perspective, it’s probably the case that file systems provide too much flexibility, and can easily become disorganized.

• There are all sorts of possible problems regarding consistency, backups, and data corruption that all data storage systems must address, but that present larger problems as file systems need to scale to handle bigger sets of data, more users, and attach to systems that are more geographically disparate.

  Given these presumptions, my personal biases and outlook, and a bit of extrapolation here’s a basic feature set for “information storage system.” These features will transcend the storage engine/interface boundary a bit. You’ve been warned.

• Multiple people and systems need to be able to access and edit the same objects concurrently.

• Existing tools need to be able to work in some capacity. Perhaps using FUSE-like systems. File managers, mv, ls, and cp should just work, etc.

• There ought to be some sort of off-network capability so that a user can loose a network connection without loosing access to his or her data.

• Search indexing and capabilities should be baked into the lowest levels of the system so that people can easily find information.

• There ought to be some sort of user facing meta-data system which can affect not just sort order, but also attach to actions, to create notifications, or manipulate the data for easier use.

These sorts sorts of features are of course not new ideas. My sygn project is one example, as is haven, as is this personal information management proposal.

Now all we need to do is figure some way to build it.

Why Bother With Lisp?

I’m quite fond of saying ”I’m not a programmer or software developer,” on this blog, and while I don’t think that there’s a great chance that I’ll be employed as a developer, it’s becoming more apparent that the real difference between me and a “real developer” is vanishingly small. Stealth Developer, or something. In any case, my ongoing tooling around with common lisp and more recently the tumble manager project have given me opportunities to think about lisp and to think about why I enjoy it.

This post started when a friend asked me “so should I learn common lisp.” And my first response was something to the effect of “no, are you crazy?” or, alternately “well if you really want to.” And then I came to my senses and offered a more reasonable answer that I think some of you might find useful.

Let us start by asking ”Should You Study Common Lisp?”

Yes! There are a number of great reasons to use Common Lisp:

• There are a number of good open source implementations of the common lisp language including a couple of very interesting and viable options. They’re also stable: SBCL which is among the more recent entrants to this field is more than a decade old.

• There are sophisticated development tools, notably SLIME (for emacs) which connects and integrates emacs with the lisp process, as well as advanced REPLs (i.e. interactive mode). So getting started isn’t difficult.

• Common Lisp supports many different approaches to programming. Indeed, contemporary “advanced” languages like Ruby and Python borrow a lot from Lisp. So it’s not an “archaic” language by any means. Dynamic typing, garbage collection, macros, and so forth.

• CL is capable of very high performance, so the chance of saying “damn, I wish I wrote this in a faster language,” down the road isn’t terribly likely. Most implementations run on most platforms of any consequence, which is nice.

• You’re probably tired of hearing that “Learning Lisp will make your a better programmer in any language,” but it’s probably true on some level.

The reasons to not learn Lisp or to avoid using it are also multiple:

• “Compiled” Lisp binaries are large compared to similarly functional programs in other languages. While most CL implementations will compile native binaries, they also have to compile in most of themselves.

• Lisp is totally a small niche language, and we’d be dumb to assume that it’s ever going to take off. It’s “real” by most measurements, but it’s never really going to be popular or widely deployed in the way that other contemporary languages are.

• Other programmers will think you’re weird.

Having said that all of I think we should still start projects in CL, and expand the amount of software that’s written in the language. Here’s why my next programing project is going to be written in lisp:

• I enjoy it. I suspect this project like many projects you may be considering is something of an undertaking. Given that I don’t want to have to work in an environment that I don’t enjoy, simply because it’s popular or ubiquitous.

• Although Lisp isn’t very popular, it’s popular enough that all of the things that you might want to do in your project have library support. So it’s not exactly a wasteland.

• The Common Lisp community is small, but it’s dedicated and fairly close knit. Which means you may be able to get some exposure for your application in the CL community, simply because your app is written in CL. This is a question of scale, but it’s easier to stand out in a smaller niche.

Of course there are some advantages to “sticking with the crowd” and choosing a different platform to develop your application in:

• If you want other people to contribute to your project, it’s probably best to pick a language that the people who might be contributing to your application already know.

• While there are libraries for most common things that you might want to do with Common Lisp, there might not be libraries for very new or very esoteric tasks or interfaces. Which isn’t always a problem, but can be depending on your domain.

• The binary size problem will be an issue if you plan to deploy in limited conditions (we’re talking like a 15 meg base size for SBCL, which is a non issue in most cases, but might become an issue.)

• If you run into a problem, you might have a hard time finding an answer. This is often not the case, but it’s a risk.

Onward and Upward!

Where is Java Today?

A few weeks ago a coworker walked into my office to talk about the architecture of a project, complete with diagrams, numbers I didn’t grasp (nor really need to,) and the examples of potential off the shelf components that would make up the stack of the application at hand. I asked scores of questions and I think it was a productive encounter. Normal day, really. I seem to be the guy developers come to and pitch ideas to for feedback. Not sure why but I thin think that the experience of talking through a programing or design problem tends to be a productive learning experience for everyone. In any case the details aren’t terribly important

What stuck in my head is that an off the self, but non-trivial part of the system was written in Java.

We all inhaled sharply.

I don’t know what it is about Java, and I don’t think it’s just me, but the moment I find out that an application is written in Java, I have a nearly visceral reaction. And I don’t think it’s just me.

Java earned a terrible reputation in the 90s, because although it was trumped as the next big thing every user facing application in Java sucked: first you had to download a lot of software (and hope that you got the right version of the dependency) and then when you ran the app it took a while to start up and looked like crap. And then your system ground to a halt and the app crashed. But these problems have been fixed: the dependency issue is more clear with the GPLing of Java, GUI bindings for common platforms are a bit stronger, computers have gotten a lot faster, and perhaps most importantly the hopes of using Java as the cross platform application development environment have been dashed. I think it’s probably fair to say that most Java these days runs on the server side, so we don’t have to interact with it in the same sort of hands on way.

This isn’t to say that administering Java components in server operations is without problems: Java apps tend to run a bit hot (in terms of RAM,) and can be a bit finicky, but Java applications seem to fit in a bit better in these contexts, and certainly have been widely deployed here. Additionally, I want to be very clear, I don’t want to blame the language for the poor programs that happen to be written in it.

Here are the (hopefully not too leading) questions:

1. Is the “write once run everywhere,” thing that Java did in the beginning still relevant, for server-based applications? It’s a server application after all, you wouldn’t be loosing much by targeting a more concrete native platform.

2. Is the fact that Java is statically typed more of hindrance in terms of programmer time? And will the comparative worth of Java’s efficiency wear off as computers continue to get more powerful

   Conventional wisdom being that while statically typed apps “run faster,” but take longer to develop. This is the argument used by Python/Perl/Ruby/etc proponents, and I don’t know how the dynamics of these arguments shift in response to the action of Moore’s Law.

3. One of the great selling points of Java is that it executes code in a “managed” environment, which provides some security and safety to the operator of the system. Does the emergence of system-level visualization tools make the sandboxing features of the JVM less valuable?

4. I don’t think my experiences are particularly typical, but all of the Java applications I’ve done any sort of administrative work with have been incredibly resource intensive. This might be a product of the problem domains. Using Java is often like slinging a sledge hammer around, and so many problems these days don’t really require a sledge hammer.

5. At this point, the amount of “legacy” Java code in use is vast. I sometimes have trouble understanding if Java current state is the result of all of the tools that have already been invested in the platform or the result of actually interesting and exciting developments in the platform. Like Clojure. Is Clojure (as an example,) popular because Lisp is cool again and people have finally come to their senses (heh, unlikely) or because it’s been bootstrapped by java and provides a more pain free coding experience for Java developers?

Anyone have ideas on these points? Questions that you think I’m missing?

Things I'm Wiking On

• The Contra Purity Test: A couple friends and thought it might be cool to work on a “purity test” for contra dancing/dancers. Mostly for jokes, I said ”I’ll make a wiki page,” and then I did. It is, I’d say 70% done.

• Some writing and thinking about TumbleManager, tumblelog engine in response to this post. I’m writing this as a design document/spec sheet because I’m a writer and my brain works like this.

• The Lessons from System Administration is the beginning of an archive of posts that I’ve written recently drawing lessons learned in system administration into other more generally useful contexts.

• Sygn Networking - I’ve done some editing recently on this older project with an attempt to move it forward. Reorganization, cleaning up of text, adding new points in where they make sense. Of particular interest is, I think the data examples of what sygn profiles might look like. I hope this all helps illuminate the project a bit.

• Although this only barely counts: I did some very wiki-like editing to the tubmlelog post, which I think cleaned up the text a bit. I’m, as you’re aware, not a very polished sort of writer. But I think I’m pretty good at cleaning things up after a fashion, and once I have the right distance. Or something. It’s a bit clearer now, at any rate.

• Also not wiki related specifically, but I’ve been doing some rolling revisions on the Cyborg Institute website recently. I don’t want the CI project to die, but I want it to be more realistic, and useful to other people as a platform for some really awesome projects. I think by focusing the site, and pulling out some cruft we’re getting in that direction.

In addition to the above, I wanted to explore the possibility of doing more posts like this occasionally as a way of saying “Hey! I’m working on cool things like the stuff I write about here, but the stuff isn’t getting posted here. Check them out” is that a valuable use of this space? Or would this tedious?

Onward and Upward!

Input in the Next Wave

In response mostly to my own comentary of the iPad I’d like to lead a collective brainstorming of input and computer interact modalities in “the next wave.”

What’s the next wave? That thing that’s always coming “soon,” but isn’t quite here yet, the thing that we are starting to see glimpses of, but don’t really know. Accepting for a moment that things like Blackberries, netbooks, Kindles, iPads, iPhones and the like are these “harbingers” of the next wave.

The “make or break” feature of all these new and shiny things is the input method: how we get stuff from our heads into a format that a computer can do something with. While I’m a particularly… textual sort of guy, the “input question,” is something everyone who uses technology will eventually come to care about. Blackberry’s sell because they speak “messaging,” and because most of them have hardware keyboards. The iPad, with its bigger onscreen keyboard and external keyboard dock, is–to my mind–an admission that the little onscreen keyboard of the iPhone doesn’t work if you want enter more than 50 or 60 characters at any given time.

I love a good hardware keyboard. A lot, and I’m not just talking about the kind on the blackberry, but a real keyboard. The truth is I can’t even quite bring myself to justify one of the little “netbooks” on the principal that everything I do involves massive amounts of typing. And fundamentally, at the moment there doesn’t seem to be a good replacement for getting data into a computer system, that doesn’t involve a keyboard. Clearly this can’t hold out forever, and so I’d like to pose two questions:

1. What kind of computer interfaces will replace the command line?

   So in 2010 most people interact with their computers by way of the mouse and a lot of pretty pictures. Even mobile environments like the iPhone/iPad/etc. and the Blackberry have some sort of a pointer that the user has to manipulate.

   But the truth is that this kind of modality has always been inefficient: switching between the mouse and the keyboard is the greatest time sink in current user interfaces. Graphical environments require increasingly sophisticated graphics hardware, they require users to memorize interfaces in a visual way that may not be intuitive (even if we’re accustomed,) and they have incredibly high development costs relative to other kinds of software. Furthermore, most of us use a lot of text-based interfaces weather we know it or not. Google is a command line interface, as are most web browser’s address bars. And although my coworkers and I are hardly typical, we all have a handful of terminals open at any given time.

   Clearly shells, (e.g. bash, zsh, and the like) are not going to be around forever, but I think they’re going to be around until we find some sort of solution that can viably replace the traditional shell. We need computer interfaces that are largely textual, keyboard driven, powerful, modern, lightweight, fast, and designed to be used interactively. I’m not sure what it looks like, but I know that it needs to exist.

2. What kind of interfaces will replace the keyboard for data entry?

   When I was writing the iPad reflection, I thought it might be cool to have an input device that was mostly on the back of the device, so that you hold the device in both hands, your fingers make contact with some sort of sensors on the back, with your thumbs touching something on the front, and there’s some sort of on-screen interface that provides feedback to make up for the fact that you can’t see “the keys.”

   I’d be inclined to think that this would be QWERTY derived, but that’s as much a habit as it is anything. I’m a pretty good touch typist, not perfect, and not the fastest, but I don’t have to think at all about typing it just happens. But I don’t know or think that the QWERTY keyboard is going to be the interface modality of the future. While I do want to learn DVORAK typing–but haven’t managed to really feel inspired enough to do that–I think its more productive to think about replacements for the keyboard itself rather than alternate layouts.

Thoughts?