I read this post by Bruce Sterling about “Cloud Culture”, and haven’t quite been able to formulate an appropriate response. It’s true that I often find myself ranting about this or that thing Bruce Sterling has written, so I’m perhaps not the most impartial

To summarize, his notion is that “the cloud,” as evidenced by a wealth of usable digital information and content, combined with an uptick in digital participation will lead to some sort of minimalistic cultural singularity that he’s calling “Cloud Culture.”

Right.

So lets attack this full on. The “Cloud” is marketing dribble. There’s nothing there, it’s just a different way of organizing the servers that provide content and services on the Internet. In a lot of way the Amazon and Rackspace “Clouds” have brute forced the server and hosting business with utterly conventional hardware, off the shelf open source software, and a bunch of [insert-favorite-scripting-language-here] scripts. Oh, and a lot of capital.

That’s the back-end, and moving upwards in the stack, we’ll find that the tools (e.g. Apache, Relational Databases, Perl) that we’re using now are for the most part the same this time around as they were ten or fifteen years ago. Sure we have new practices, and there are new frameworks, new versions of some software, but the changes are mostly evolutionary. Until you get to what the users see, and now we have applications that live in web browsers, and a more robust set of interconnected network services.

Otherwise, the cloud is just the Internet bundled up in a differed way. And it’s true that the bundling matters, but at the end of the day just because the “cloud” presents a better way to do the work of the Internet (in some situations,) it doesn’t mean that any of the fundamental principals have changed.

Another phenomena that hasn’t changed with the cloud: we’ve not quite gotten the economic system for the Internet figured out in a way that’s reliable and sustainable. New things on the Internet are supported by capital from “old-world” industries by way of venture capital and spin off subsidiaries. The leading business models are advertising (very print media) and premium subscription services (which are often contrived). So eventually it all collapses, and any gains in cultural participation run up against the hard costs of keeping power flowing to the servers in some way.

A few thoughts:

• We could just as easily call this whole “cloud” thing “the matrix,” or “cyberspace,” in Gibson’s original formulation. Cloud is a horrible metaphor for most things, but particularly this.
• The apparent cohesiveness of the Internet in 2010 is almost entirely due to the success of Google. As much as we might try and avoid Google, I firmly believe if Google were to go away without warning, while the Internet would still work most of the economy around it would crash and burn: people wouldn’t discover new things, the on-line advertising industry would largely collapse, a lot of the software that a lot of people inside and outside of google use would be without maintainers.
• I’m not sure cultural participation is problem space that can be brute forced, and certainly the development of additional applications in the cloud is unlikely to increase participation in a direct manner. I’m not being an elitist and saying “cultural participation by the unschooled and unwashed masses isn’t worthwhile,” but rather cultural participation is about developing direct relationships and making/building things, and that requires forming relationships that don’t scale well.

Thoughts?

The file system is dead. Long live the File system.

We live in an interesting time. There are two technologies that aim to accomplish two very goals. On the one hand we have things like Amazon’s S3, Hadoop, NoSQL, and a host of technologies that destroy the file system metaphor as we know it today. The future, if you believe it, lays in storing all data in some sort of distributed key/value store-based system. And then, on the other hand we have things like “FUSE” that attempt to translate various kinds of interfaces and data systems onto the file system metaphor.

Ok, so the truth is that the opposition between the “lets replace file systems” with non-file based data stores folks and the “lets use the file system as a metaphor for everything,” is totally contrived. How data is stored and how we interact with data are very different (and not always connected) problems.

Let’s lay down some principals:

• There are (probably) more tools to interact with, organize, manage, and manipulate files and file system objects than there are for any other data storage system in contemporary technology.

• Most users of computers have some understanding of file systems and how they work, though clearly there are a great diversity of degrees here.

• In nearly every case, only one system can have access to a given file system at a time. In these days of such massive parallel computing, the size of computer networks, (and the associated latency) this has become a rather substantial limitation.

• From the average end user’s perspective, it’s probably the case that file systems provide too much flexibility, and can easily become disorganized.

• There are all sorts of possible problems regarding consistency, backups, and data corruption that all data storage systems must address, but that present larger problems as file systems need to scale to handle bigger sets of data, more users, and attach to systems that are more geographically disparate.

  Given these presumptions, my personal biases and outlook, and a bit of extrapolation here’s a basic feature set for “information storage system.” These features will transcend the storage engine/interface boundary a bit. You’ve been warned.

• Multiple people and systems need to be able to access and edit the same objects concurrently.

• Existing tools need to be able to work in some capacity. Perhaps using FUSE-like systems. File managers, mv, ls, and cp should just work, etc.

• There ought to be some sort of off-network capability so that a user can loose a network connection without loosing access to his or her data.

• Search indexing and capabilities should be baked into the lowest levels of the system so that people can easily find information.

• There ought to be some sort of user facing meta-data system which can affect not just sort order, but also attach to actions, to create notifications, or manipulate the data for easier use.

These sorts sorts of features are of course not new ideas. My sygn project is one example, as is haven, as is this personal information management proposal.

Now all we need to do is figure some way to build it.

I’m quite fond of saying “I’m not a programmer or software developer,” on this blog, and while I don’t think that there’s a great chance that I’ll be employed as a developer, it’s becoming more apparent that the real difference between me and a “real developer” is vanishingly small. Stealth Developer, or something. In any case, my ongoing tooling around with common lisp and more recently the tumble manager project have given me opportunities to think about lisp and to think about why I enjoy it.

This post started when a friend asked me “so should I learn common lisp.” And my first response was something to the effect of “no, are you crazy?” or, alternately “well if you really want to.” And then I came to my senses and offered a more reasonable answer that I think some of you might find useful.

Let us start by asking “Should You Study Common Lisp?”

Yes! There are a number of great reasons to use Common Lisp:

• There are a number of good open source implementations of the common lisp language including a couple of very interesting and viable options. They’re also stable: SBCL which is among the more recent entrants to this field is more than a decade old.
• There are sophisticated development tools, notably SLIME (for emacs) which connects and integrates emacs with the lisp process, as well as advanced REPLs (i.e. interactive mode). So getting started isn’t difficult.
• Common Lisp supports many different approaches to programming. Indeed, contemporary “advanced” languages like Ruby and Python borrow a lot from Lisp. So it’s not an “archaic” language by any means. Dynamic typing, garbage collection, macros, and so forth.
• CL is capable of very high performance, so the chance of saying “damn, I wish I wrote this in a faster language,” down the road isn’t terribly likely. Most implementations run on most platforms of any consequence, which is nice.
• You’re probably tired of hearing that “Learning Lisp will make your a better programmer in any language,” but it’s probably true on some level.

The reasons to not learn Lisp or to avoid using it are also multiple:

• “Compiled” Lisp binaries are large compared to similarly functional programs in other languages. While most CL implementations will compile native binaries, they also have to compile in most of themselves.
• Lisp is totally a small niche language, and we’d be dumb to assume that it’s ever going to take off. It’s “real” by most measurements, but it’s never really going to be popular or widely deployed in the way that other contemporary languages are.
• Other programmers will think you’re weird.

Having said that all of I think we should still start projects in CL, and expand the amount of software that’s written in the language. Here’s why my next programing project is going to be written in lisp:

• I enjoy it. I suspect this project like many projects you may be considering is something of an undertaking. Given that I don’t want to have to work in an environment that I don’t enjoy, simply because it’s popular or ubiquitous.
• Although Lisp isn’t very popular, it’s popular enough that all of the things that you might want to do in your project have library support. So it’s not exactly a wasteland.
• The Common Lisp community is small, but it’s dedicated and fairly close knit. Which means you may be able to get some exposure for your application in the CL community, simply because your app is written in CL. This is a question of scale, but it’s easier to stand out in a smaller niche.

Of course there are some advantages to “sticking with the crowd” and choosing a different platform to develop your application in:

• If you want other people to contribute to your project, it’s probably best to pick a language that the people who might be contributing to your application already know.
• While there are libraries for most common things that you might want to do with Common Lisp, there might not be libraries for very new or very esoteric tasks or interfaces. Which isn’t always a problem, but can be depending on your domain.
• The binary size problem will be an issue if you plan to deploy in limited conditions (we’re talking like a 15 meg base size for SBCL, which is a non issue in most cases, but might become an issue.)
• If you run into a problem, you might have a hard time finding an answer. This is often not the case, but it’s a risk.

Onward and Upward!

A few weeks ago a coworker walked into my office to talk about the architecture of a project, complete with diagrams, numbers I didn’t grasp (nor really need to,) and the examples of potential off the shelf components that would make up the stack of the application at hand. I asked scores of questions and I think it was a productive encounter. Normal day, really. I seem to be the guy developers come to and pitch ideas to for feedback. Not sure why but I thin think that the experience of talking through a programing or design problem tends to be a productive learning experience for everyone. In any case the details aren’t terribly important

What stuck in my head is that an off the self, but non-trivial part of the system was written in Java.

We all inhaled sharply.

I don’t know what it is about Java, and I don’t think it’s just me, but the moment I find out that an application is written in Java, I have a nearly visceral reaction. And I don’t think it’s just me.

Java earned a terrible reputation in the 90s, because although it was trumped as the next big thing every user facing application in Java sucked: first you had to download a lot of software (and hope that you got the right version of the dependency) and then when you ran the app it took a while to start up and looked like crap. And then your system ground to a halt and the app crashed. But these problems have been fixed: the dependency issue is more clear with the GPLing of Java, GUI bindings for common platforms are a bit stronger, computers have gotten a lot faster, and perhaps most importantly the hopes of using Java as the cross platform application development environment have been dashed. I think it’s probably fair to say that most Java these days runs on the server side, so we don’t have to interact with it in the same sort of hands on way.

This isn’t to say that administering Java components in server operations is without problems: Java apps tend to run a bit hot (in terms of RAM,) and can be a bit finicky, but Java applications seem to fit in a bit better in these contexts, and certainly have been widely deployed here. Additionally, I want to be very clear, I don’t want to blame the language for the poor programs that happen to be written in it.

Here are the (hopefully not too leading) questions:

1. Is the “write once run everywhere,” thing that Java did in the beginning still relevant, for server-based applications? It’s a server application after all, you wouldn’t be loosing much by targeting a more concrete native platform.

2. Is the fact that Java is statically typed more of hindrance in terms of programmer time? And will the comparative worth of Java’s efficiency wear off as computers continue to get more powerful

Conventional wisdom being that while statically typed apps “run faster,” but take longer to develop. This is the argument used by Python/Perl/Ruby/etc proponents, and I don’t know how the dynamics of these arguments shift in response to the action of Moore’s Law.

3. One of the great selling points of Java is that it executes code in a “managed” environment, which provides some security and safety to the operator of the system. Does the emergence of system-level visualization tools make the sandboxing features of the JVM less valuable?

4. I don’t think my experiences are particularly typical, but all of the Java applications I’ve done any sort of administrative work with have been incredibly resource intensive. This might be a product of the problem domains. Using Java is often like slinging a sledge hammer around, and so many problems these days don’t really require a sledge hammer.

5. At this point, the amount of “legacy” Java code in use is vast. I sometimes have trouble understanding if Java current state is the result of all of the tools that have already been invested in the platform or the result of actually interesting and exciting developments in the platform. Like Clojure. Is Clojure (as an example,) popular because Lisp is cool again and people have finally come to their senses (heh, unlikely) or because it’s been bootstrapped by java and provides a more pain free coding experience for Java developers?

Anyone have ideas on these points? Questions that you think I’m missing?

• The Contra Purity Test: A couple friends and thought it might be cool to work on a “purity test” for contra dancing/dancers. Mostly for jokes, I said “I’ll make a wiki page,” and then I did. It is, I’d say 70% done.
• Some writing and thinking about TumbleManager, tumblelog engine in response to this post. I’m writing this as a design document/spec sheet because I’m a writer and my brain works like this.
• The Lessons from System Administration is the beginning of an archive of posts that I’ve written recently drawing lessons learned in system administration into other more generally useful contexts.
• Sygn Networking - I’ve done some editing recently on this older project with an attempt to move it forward. Reorganization, cleaning up of text, adding new points in where they make sense. Of particular interest is, I think the data examples of what sygn profiles might look like. I hope this all helps illuminate the project a bit.
• Although this only barely counts: I did some very wiki-like editing to the tubmlelog post, which I think cleaned up the text a bit. I’m, as you’re aware, not a very polished sort of writer. But I think I’m pretty good at cleaning things up after a fashion, and once I have the right distance. Or something. It’s a bit clearer now, at any rate.
• Also not wiki related specifically, but I’ve been doing some rolling revisions on the Cyborg Institute website recently. I don’t want the CI project to die, but I want it to be more realistic, and useful to other people as a platform for some really awesome projects. I think by focusing the site, and pulling out some cruft we’re getting in that direction.

In addition to the above, I wanted to explore the possibility of doing more posts like this occasionally as a way of saying “Hey! I’m working on cool things like the stuff I write about here, but the stuff isn’t getting posted here. Check them out” is that a valuable use of this space? Or would this tedious?

Onward and Upward!

In response mostly to my own comentary of the iPad I’d like to lead a collective brainstorming of input and computer interact modalities in “the next wave.”

What’s the next wave? That thing that’s always coming “soon,” but isn’t quite here yet, the thing that we are starting to see glimpses of, but don’t really know. Accepting for a moment that things like Blackberries, netbooks, Kindles, iPads, iPhones and the like are these “harbingers” of the next wave.

The “make or break” feature of all these new and shiny things is the input method: how we get stuff from our heads into a format that a computer can do something with. While I’m a particularly… textual sort of guy, the “input question,” is something everyone who uses technology will eventually come to care about. Blackberry’s sell because they speak “messaging,” and because most of them have hardware keyboards. The iPad, with its bigger onscreen keyboard and external keyboard dock, is--to my mind--an admission that the little onscreen keyboard of the iPhone doesn’t work if you want enter more than 50 or 60 characters at any given time.

I love a good hardware keyboard. A lot, and I’m not just talking about the kind on the blackberry, but a real keyboard. The truth is I can’t even quite bring myself to justify one of the little “netbooks” on the principal that everything I do involves massive amounts of typing. And fundamentally, at the moment there doesn’t seem to be a good replacement for getting data into a computer system, that doesn’t involve a keyboard. Clearly this can’t hold out forever, and so I’d like to pose two questions:

1. What kind of computer interfaces will replace the command line?

So in 2010 most people interact with their computers by way of the mouse and a lot of pretty pictures. Even mobile environments like the iPhone/iPad/etc. and the Blackberry have some sort of a pointer that the user has to manipulate.

But the truth is that this kind of modality has always been inefficient: switching between the mouse and the keyboard is the greatest time sink in current user interfaces. Graphical environments require increasingly sophisticated graphics hardware, they require users to memorize interfaces in a visual way that may not be intuitive (even if we’re accustomed,) and they have incredibly high development costs relative to other kinds of software. Furthermore, most of us use a lot of text-based interfaces weather we know it or not. Google is a command line interface, as are most web browser’s address bars. And although my coworkers and I are hardly typical, we all have a handful of terminals open at any given time.

Clearly shells, (e.g. bash, zsh, and the like) are not going to be around forever, but I think they’re going to be around until we find some sort of solution that can viably replace the traditional shell. We need computer interfaces that are largely textual, keyboard driven, powerful, modern, lightweight, fast, and designed to be used interactively. I’m not sure what it looks like, but I know that it needs to exist.

2. What kind of interfaces will replace the keyboard for data entry?

When I was writing the iPad reflection, I thought it might be cool to have an input device that was mostly on the back of the device, so that you hold the device in both hands, your fingers make contact with some sort of sensors on the back, with your thumbs touching something on the front, and there’s some sort of on-screen interface that provides feedback to make up for the fact that you can’t see “the keys.”

I’d be inclined to think that this would be QWERTY derived, but that’s as much a habit as it is anything. I’m a pretty good touch typist, not perfect, and not the fastest, but I don’t have to think at all about typing it just happens. But I don’t know or think that the QWERTY keyboard is going to be the interface modality of the future. While I do want to learn DVORAK typing--but haven’t managed to really feel inspired enough to do that--I think its more productive to think about replacements for the keyboard itself rather than alternate layouts.

Thoughts?

I’ve been thinking recently about the relationship and dynamic between the corporations and “enterprises” which participate in and reap benefits from open source/free software and the quasi-mythic “communities” that are responsible for the creation and maintenance of the software. Additionally this post may be considered part of my ongoing series on cooperative economics.

When people, ranging from business types, to IT professionals, to programmers, and beyond, talk about open source software we talk about a community: often small to medium sized groups of people who all contribute small amounts of time to creating software. And we’re not just talking about dinky little scripts that make publishing blogs easier (or some such), we’re talking about a massive amount of software: entire operating systems, widely used implementations of nearly all relevant programing languages, and so forth. On some level the core of this question is who are these people, and how do they produce software?

On the surface the answer to these questions is straightforward. The people who work on open source software are professional programmers, students, geeks, and hacker/tinkerer-types who need their computers to do something novel, and then they write software. This works as model for thinking about who participates in open source, if we assume that the reason why people contribute to open source projects is because their individual works/contributions are too small to develop business models around. This might explain some portion of open source contributions, but it feels incomplete to me.

There are a number of software projects that use open source/free software licenses, with accessible source code, supported by “communities,” which are nonetheless developed almost entirely by single companies. MySQL, Alfresco, and Resin among others serve as examples these kinds of projects which are open source by many any definitions and yet don’t particularly strike me as “community,” projects. Is the fact that this software provides source code meaningful or important?

Other questions…

1. If there are companies making money from open source code bases, particularly big companies in a business directly related to software, does this effect participation of people who are not employed by that company in the project?

In my mind I draw distinctions between technology businesses that use/sell/support open source software (e.g. Red-Hat, the late MySQL AB, etc.) and businesses that do something else but use open source software (i.e. everyone with a Linux server in the basement, every business with a website that runs on Apache, etc.)

2. Does corporate personhood extend to the open source community. Are corporate developers contributing as people, or as representatives of their company?

I largely expect that it’s the former; however, I’d be interested in learning more about the various factors which affect the way these contributors are perceived?

3. Do people participate in open source because it is fun or for the enjoyment of programming,

4. Has software become so generic that open source is a current evolution of industry standards groups. Do we write open source software for the same reason that industries standardized the size and threading of bolts?

5. Are potential contributors disinclined to contribute to software that is controlled by a single entity, or where projects

6. Is the cost of forking a software project too high to make that a realistic outcome of releasing open source software?

Conversely, were forks ever effective?

7. Do communities actually form around software targeted at “enterprise” users, and if so in what ways are those communities different from the communities that form around niche window p managers or even community projects like Debian?

I don’t of course have answers yet, but I think these questions are important, and I’d love to hear if you have any ideas about finding answers to these questions, or additional related questions that I’ve missed.

I’ve been thinking about how ideas, projects, and ideas scale a bit in the past few weeks, and as usual, I wanted to collect a few of these thoughts. This post is generally in my series of posts of “Extrapolations from Systems Administration.” Inspirations and origins of these ideas come from, in part:

• My Tumblr Killed the Tumblelog Star post.
• My Google reader unread count.
• Savage Minds Post on Social Enterpernurship and Anthropology post.
• All the television I don’t watch.
• The nginx HTTP Proxy module.
• Seeing the Forest for the Brand

The internet is a big place, you don’t need me to tell you this, but I think that it’s really incomprehensibly big. Even the small corners of the internet that we (well, I at least,) inhabit contain vast amounts of information and it’s very difficult to keep your head above water, to feel like you’re in touch with what’s happening. Strategies for managing this information successfully are as concerned with “figuring out what to ignore,” as they are about figuring out how to absorb information successfully.

Scaling an idea or a concept (like a blog, or a piece of software or a web server) to be able to address problem sets (like an audience, or a given set of data, or both) of different sizes is just as difficult. It’s tough to get a web server to be able to host really large loads, its difficult to be able to write a blog that appeals to a huge audience: the this nexuses of related problems are quite large.

I think, however, we can begin to draw some conclusions. I do hope that you’ll be able to help me add to this list. Perhaps in a wikish page.

1. Be the biggest fish in the smallest possible pond.

   The core strategy here is to avoid having to figure out how to scale up to “full speed,” by reframing the problem set. You don’t have to become the most popular or widely consumed blogger/novelist: you just have to become the most popular blogger about cyborg philosophy, or the political economics and philosophy of the open source world. You have to become the most popular post-colonial historiography space opera novelist.

2. Don’t participate in the proliferation of crap: only build/use what you need to.

   I see lots of people say something along the lines of “I want to make a websites for all of the people interested in what I’m interested in, and we’ll need a wiki and some discussion forums, and some sort of blogs, maybe a lot of blogs, and…” This inevitably leads to a bunch of organization and building of things for their websites, and then everything is built and… no one is interested in using the crap.

   This is a classic premature optimization problem. Don’t build things that you think you might need later. Build things that you need now. Or things that you really needed last week. Focus on the thing you do, and build the infrastructure as you need it, when you need it.

3. Work in a scalable and sustainable manner, and assume that other people will need to pick up on your projects.

   While you shouldn’t expend the effort to scale before you need to, because that could end in failure, it’s common sense to approach your projects with the assumption that other people might have to finish them for you, if things take off and you need to delegate later you’ll be ready for them. Consider the possibility that you might need to scale a project when you’re in the initial planning stages and avoid getting backed into a corner by a decision.

4. Ignore everything you can possibly stand to.

   There are so many things that you could be doing with your time. There are so many distractions. Email lists, RSS feeds, the work of other people in your field. Charity projects of one sort or another. All of these things are important and you should participate fully in the communities that surround your work, but be fully aware that humans as individuals don’t scale well, and succeeding at [your project] is dependent upon your ability to ignore everything that you can stand to.

5. Consume information on your terms, in the formats that make the most sense to you.

   As a corollary to the above, the way to successfully engage and manage everything that you can’t possibly stand to ignore is to as much as possible engage on your terms. Figure out what your terms are first, and then work to consume content in these terms.

6. Use technology and media to build relationships rather than accumulate information.

   Too often, I think, the geekier among us (and I count myself among this number) are interested in technology because it’s cool, and we’re tempted to solve technological problems and learn about the inner workings of stuff because they interest us. And that’s okay, as a hobby: in the pursuit of doing work, technology is only useful insofar as it allows you to get things done. And in most cases, the core function that technology provides is to enable relationships. So focus on that, and fiddle with the technical underpinnings, only when you must.

Onward and Upward!