Posts on tychoish

New Go Module: tychoish/godmenu

Tue, 15 Jul 2025 00:00:00 +0000

A while back, I wrote a little Go wrapper library around the dmenu library and called it go-dmenu but then realized that without the hyphen the ambiguity between god-menu and go-dmenu was sort of delicious so I went with it.

This is not an exciting piece of software, and is just a wrapper that calls dmenu in a subprocess and passes it a list of options and returns to that Go program, the selection provided by the user (via dmenu.) That’s it. One function, one configuration structure, that’s it, no dependencies, no more features.

Check the repo tychoish/godmenu, and also the docs.

In some ways that’s the post. I should end here, but…

`dmenu`?

dmenu is this great, super simple piece of software that is just a menu for selecting items from a list. That’s it. You send it a list of selections on standard input, it does it’s thing and returns, on standard output, the selection. That’s it.

It comes with this dmenu_run script that gets a list of all the executable files on the PATH runs whatever command you select. So it works kind of like a launcher, and I think by and large, this is the main thing that it’s used for, but it’s so simple, and so good at doing it’s thing that there’s no reason you can’t use it for anything menu-ish or selection-ish.

What?

godmenu is just a wrapper that lets you do call dmenu from Go programs, so that it’s easy to use it from within Go programs, and so that you don’t need to write shell scripts to use it. There’s sort of nothing to the library, and maybe that’s the point: better to have it be the kind of thing that you write once, get it right and then sort of never have to think about it or do it again.

I’ve been thinking recently about what it means for a library to be useable or possible for a project to adopt or include it, and this is an experiment in a couple of these ideas: it’s simple and focused, provides a single piece of functionality, manages no state, has no dependencies, and makes it easier to do a thing that would otherwise be annoying.

Why?

A while ago I decided that I was done writing tools for my own personal use as shell scripts, because I was tired of having to fix them after years of not thinking about them, or figuring out what their dependencies were when I got a new computer, and just generally in writing the kind of very defensive code that you end up needing to write.

I’ve gotten pretty good at subprocess management (mostly because of jasper, though the godmenu wrapper doesn’t use jasper to minimize the dependency proliferation,) but adding interactivity directly in a simple program gets very complex quickly. Beyond a certain point, as soon as you you need user input or interaction, writing a little script becomes incredibly complex. Command line only interfaces–subcommands and flags–are simple but there are lots of parsing edge cases and countless competing libraries/frameworks, and when there are lots of options and the “default” options aren’t frequently used, usability falls apart. Richer interfaces–GUI or console widgets–add a lot of technical complexity and overhead (almost all of the time,) and you still have to design the interface.

Using dmenu selectors in your scripts, means you can (basically) write a bunch of small commands, stick them in a menu, and get fuzzy-text selection based on the options you provide. This gets you pretty far with minimal additional technical or conceptual complexity.

How?

It’s a normal Go package, nothing crazy, here’s a minimal example, for a user to select between the first four lowercase ASCII characters.

	output, err := godmenu.Do(ctx,
		godmenu.Operation{
			Selections: []string{"a", "c", "d", "b"},
			Sorted: true,
			DMenu: &godmenu.Configuration{
				Path:            godmenu.DefaultDMenuPath,
				BackgroundColor: godmenu.DefaultBackgroundColor,
				TextColor:       godmenu.DefaultTextColor,
				Font:            "Source Code Pro-13",
				Lines:           16,
				Prompt:          "=>>",
			}
		},
	)
	if err != nil {
		return err
	}
	// &etc...

The Do function returns the string the user selected with the menu, or the empty string ("") and the godmenu.ErrSelectionMissing error.

Interesting options here:

the “lines” option makes the number of visable lines the menu uses configurable. I only discovered this option recently and I rather like it.
I don’t use it myself, but there’s a Bottom boolean option that makes the menu appear at the bottom rather than the top of the screen.

There’s also the Run method which does the same thing, but uses the so-called “functional arguments” pattern for a bit more flexibility at the site. Since dmenu and godmenu have some reasonable default options, you don’t need to specify all the options, or even many of them, so something like the following would work.

	out, err := godmenu.Run(ctx
		godmenu.Selections("a", "b", "c", "d"), 
		godmenu.Prompt("godmenu =>>"),
		godmenu.Sorted(), 
	)
	if err != nil {
		return err 
	}
	// &etc...

There are also some helpers and aliases among the options provided, to make it easier to call this in various ways:

	out, err := godmenu.Do(ctx, 
		godmenu.ResolveOptions(
			godmenu.Items("a", "b", "c"), 
			godmenu.Prompt("=>"),
			godmenu.WithFlags(godmenu.DefaultFlags())))
	if err != nil {
		return err
	}

	// or... 
	
	out, err = godmenu.Do(ctx, MakeOptions("a", "b", "c"))
	if err != nil {
		return err
	}

	// or ... 

	out, err = godmenu.Run(ctx, WithOptions(MakeOptions("a", "b", "c")))
	if err != nil {
		return err
	}

Though possible, these exact examples might not make a lot of sense, but it becomes possible to refactor the way you call godmenu, to reuse as much of the configuration as possible.

But other than this one operation–“select item from list”–and the ways you can call things… but this that’s it!

What if..?

Nope.

I’ve been playing with building out some additional tools here, but I’m disinclined to creep the scope (or the dependencies) of the core package for this, in the same way that dmenu itself is unlikely to ever really gain new features:

For sequences of options or “nesting” of menus so that you can use dmenu either as a prompt for occasional interaction or simply as an easy way to help users navigate a tree-like structure.

I think this is cool, but it feels like something that you could do in downstream code more easily and clearly.
I think it’d be cool if you could define a bunch of options, and not have to write the glue code between “the selection” and the option you have. Write the command/options as a map of strings to functions, and then call dmenu for the keys in the map and run the options.

Seems cool to have routing/dispatching driven by dmenu, and totally possible, though again, this is probably shouldn’t be a core feature.
~~The one exception to the “no new features ever” rule, might be to have an alternate top-level call that takes configuration as functional arguments rather in the structured form.~~

Done, oops.

Anyway, give it a try!

And, if you thought this was cool, stay tuned, I’ve got a few more blog posts in the works, more thoughts on this kind of “tools for ‘personal computing’” software that I’ve been working on!

The Mid Career Shuffle

Wed, 20 Sep 2023 00:00:00 +0000

tl;dr> I’ve decided to take a new job as an early engineering manager/tech lead, at a data analytics/database startup. And I hope to begin blogging/journaling here more about software development and the way that software and databases interact with other things that grab my interest.

This post doesn’t really have a narrative. It’s just a collection of thoughts. I’m sort of out of the habit of writing blog posts, and one of the things that’s gotten me stuck on writing in the past year or so is feeling like the narrative of any given blog post is not quite right.¹ But we have to start somewhere, and the story will happen over the next few posts.

In the middle of 2020 I left a job at a company that I’d been at for almost 9 years. It seemed like the world was on fire, what it meant to work at a globally distributed tech company was changing (and still is!), and it felt like a good time to make a change. But to what? I’ve spent the last few years working on different projects and exporing different things: I’ve worked on cool distributed systems problems, I’ve worked on product-focused (backend) engineering, I’ve worked on familiar latency-sensitive compute provisioning and orchestration, and a little bit this and that between.

I’ve identified a few things about jobs and working in this process:

while remote work is (and has been for a while) definitely a reality of our world,² the one thing you can’t really accommodate for is timezones. A friend, zmagg, claims (and I believe this) that time differences between 9 hours and 16 hours are basically unworkable as there isn’t really enough overlap to really collaborate.
if the center of gravity in a company is in a place, or in an office, remote teams really have to be intentional about making sure to include people who are outside of that bubble.
the interesting parts of software engineering is what happens between people building software on a team: programming is a means to an end, and while the tools change every so often, how you build together is really fascinating (and hard)!

My new job is different from any job I’ve ever had: First, I’m going to be working on building and developing teams and helping support the engineers and the engineering teams rather than working directly/full-time on the software. I probably will end up doing some software work too. This isn’t that novel: I’ve definitely done “the management thing” a few times of times, but more of my time will be doing this, and I get to be more intentional about how I approach it. Second, my work (mostly) has been in support of products that other people--largely my coworkers develop, in the sense that the work that I’m drawn to is the internal infrastructure of software. I like infrastructure a lot, and I think I tend to focus on these kinds of problems because I feel like it gives me the biggest opportunity to make an impact. In this sense, in a lot of ways this role is very similar to my previous jobs: building the infrastructure to enable and support the people working around me. Only this time it is (not entirely) software. I’m actually looking forward to mapping concrete business goals to actual research and development problems and projects, helping to provide context and prioritize projects without needing to filter through layers of indirection.

I have been blogging in one form or another (mostly this form) to greater and lesser (mostly lesser) extent for a (really) really long time. The landscape of the way we interact on the internet is changing: twitter is really a shell of its former self, “big” social media networks (facebook, instagram, ticktock, twitter, etc.) have really segmented by generation and region, and little social media sites are still niche.

For my part, I’ve been spending more time in “group chat,” with sort of wacky combinations of random friends. This isn’t new for me, and I take a lot of joy of building out very small communities of folks. Also, as I think about it, the thing I most want to do is have coversations with people. Sometimes that conversation is longer form (like this,) and sometimes, it’s more one-to-many (like this sort of half baked telegram channel idea that’s just sort of a “things I would have said on twitter” but as a chat,) but rarely is it “I would like to become a publisher,” or I want to use blogging as a way to market some other entrepreneurial effort. Not that there are anything wrong with these reasons for blogging.

It’s easy (as a software engineer, though I think this probably holds true for anyone who process for “building” thing requires a lot of just “thinking,” about things,) to do a lot of the thinking part in private and to only show up with the “right” solution (with maybe a few alternatives for rhetorical effect,) or to mostly focus our writing on conclusions reached rather than questions asked.

So perhaps, (and we’ll see if holds,) I can use this space more as a journal and as a collection of questions and (perhaps) failed attempts and less of news feed.

I think I often get stuck on where to start, and how much exposition to provide for the topic. I think too much time as a technical writer where my job was to explain difficult concepts means I can (at least try) to start to start to early, and then feel like I get to the end of a post before I’ve said what I think needs to be said. The answer to this might be to just write more and post more, so the exposition is just in the scroll back rather than trying to do too much in one post. And to always err on the side of not enough exposition. Let me know, though if you have an opinion on that one. ↩︎
companies are often necessarily global in scope, and having a team or project spread between different offices isn’t all that different than when it is spread between a bunch of people’s homes. Once you have more than one location, n locations are pretty much the same. ↩︎

The Most Forgotten CI Feature

Tue, 22 Aug 2023 00:00:00 +0000

Developers love to complain about their tools, particularly continuous integration (CI) software, everyone has a pet idea of how to make it faster to run or make the results easier to interpret, or more useful. They’re all wrong.¹

I spent a few years as the, I guess tech lead(?),² of a team that was developing a CI tool/platform. It was a cool project, big scale (IO-bound workloads! Cloud!) in an organization that grew rapidly and was exactly the right age for the organization to believe in CI, but also predate the maturity of really mature off-the-shelf CI systems (e.g. Github Actions.)

We did a lot of cool things, but I think the most useful thing we ever built was a notification system that let people know when their tests were done running.

That’s it.

See, people were always asking “I want my test to run in 5 minutes, or 10 minutes,” so that I don’t lose too much time waiting for the results and I can avoid getting distracted or losing focus. You could spend a lot of time making things faster, and in some cases this is a great idea: slow things are bad, compute time is expensive (particularly in aggregate), and for trivial things you do want a fast response time.

The problem is really that sometimes things can’t be made all that much faster without an exceptional amount of effort, and while compute time is expensive sometimes you end up spending significantly more on faster machines or more machinesfor increased parallelism, which can result in for only modest gains.

This of course misses the point: human attention spans are incredibly fickle and while really well focused and disciplined engineers might be able to wait for 5 minutes, anything longer than that and most people will have moved on and at that point it might as well taken an hour. While there are some upper bounds and pragmatic aspects on this just because if a build takes 2 hours (say) a developer can only really has time to try 1 or 2 things out in a given work day, but it does mean that execution times between a minute and about 20 minutes are functionally the same.

So, just notify people when their build is done. Don’t beat distraction by being really fast, beat distraction by interrupting the distraction. People don’t need to spend their day looking at a dashboard if the dashboard tells them (gently) to come back when their task is done.

Why doesn’t every CI tool do this? It might be the feature that every developer wants, and yet, there’s no really good “tell me when my build is done,” feature in any of the popular tools. It’s a hard feature to get right, and there are a lot of tricky bits:

People generally don’t want to get emails; emails would be easy, but they’re not a good way to send a quick--largely ephemeral--reminder. While you can pull emails out of commits (which isn’t a great strategy most of the time,) or (presumably) usernames if you’re on a platform like GitHub, there are some important user settings that you have to store somewhere.
Who to notify for a particular build is a little hard, it turns out. People often want to opt into notifications and be able to only receive notifications that are important to them. Sometimes the person who wants the notification isn’t the set of the authors of a commit, or the user that submitted the branch/pull request.
When to send a notification isn’t clear either. Whenall tasks in the build complete? What if some tasks are blocked from starting because one thing failed? Is “do what I mean” notifications something like “notify me on the first failure, or when all (runable) tasks succeed”? If a task fails (and generates a notification,) and then a task is restarted and then the build goes on to succeed do you send a second notification (probably?) Not only are these hard questions, but different teams might want different semantics.

It’s one of those things that seems simple, but there’s just enough complexity that it’s hard to build and hard to get right. Easier, a bit to do when all of your CI platform is developed in house, but only a bit, and (probably for the better) there aren’t many specific tasks Anyway, I hope someone builds something like this, I’d certainly use it.

Well not all of them. The problem is that developers are very likely to get stuck spinning in weird local optimizations and it’s really hard to think about CI features as a user. Developing CI software is also a fun ride because all your users are also engineers, which is a tough crowd. ↩︎
In retrospect I realize I was doing tech-lead things, but my title and how we talked about I what I was doing at the time wasn’t indicative of this reality. Ah well. ↩︎

Reintroducing Grip

Thu, 17 Aug 2023 00:00:00 +0000

Once upon a time, I wrote this logging library for Go called grip and used it a lot at a couple of jobs, where it provided a pretty great API and way to think about running logging and metrics infrastructure for an application. For the last year and a half, I’ve mostly ignored it. I didn’t want to be that guy about loggers, so I mostly let it drop, but I was looking at some other logging systems and I felt inspired to unwrap it a bit and see if I could improve things and if the time away would inspire me to rethink any of the basic assumptions.

This post is about a lot of things (and maybe it will spill over,) but generally:

the practice and use of logging, and how that is changing,
adoption of libraries out side of the standard library for common infrastructure,
the process of changing grip, and also digging into some more specific changes.

Logging Trends

Grip came about sort of at the height of the structured logging craze, and I think still focuses more on being a way to provide not just the ability to do slightly cooler than a collection of print statements but also be the primary way your software transmits events. Because all of the parts of grip are pretty plugable, we used this for everything from normal application logging to our entire event notification system, and kind of everything in between, including a sort of frightful amount of managing the output of a distributed build farm. The output of these messages went to all of the usual notification targets (e.g. email, slack, github, jira,) but also directly to log aggregation services (e.g. splunk) and the system log without being written to a file, which just made the services much easier to operate.

Like me, grip is somewhat opinionated. A lot of loggers compete based on “speed” (typically of writing messages to standard output or to any file) or on features (automatic contextual data collection and message annotation, formatting tools, hierarchical filtering, etc.) but I think grip’s selling point is really:

great interfaces for doing all of the things that you normally do with a logger, and all the things you want to do.
provide ways of logging data that doesn’t involve building/formatting strings, and lets the normal data.
make logging work just as well for the small CLI process as the large distributed application, and with similar amounts of ease. Pursuant to this I really think that any logging system or architecture that requires lots of extra stuff (agents, forwarders, rotation tools, etc.) to actually be able to use the logs is overkill. There’s nothing particularly exciting about sending logs over the network (or to a file, or a local socket,) and it should be possible to reduce the amount of complexity in this part of the deployment, which is good for just about everyone.
provide good interfaces and paradigms for replacing or custimizing core logging behavior. It should be possible for applications to write their own tools for sending log messages to places, or functions to filter and format log messages. This is a logging tool kit that can grow and develop with your application.
- in the x hierarchy include implementations for many messaging formats, and tools: use grip implementations to send alerts to telegram, email, slack, jira, etc. Also, support for sending logs _directly to wherever they need to be (splunk, syslog, etc.). Whever the logs need to be, grip should be able to get you there.
- The core grip package provides logging multiplexers, buffers, and batching tools to help control and manage the flow of logging from your application code to whatever logging backend you’re targeting.'
- Implementations and bridges between grip’s interfaces and other logging framework and tools. Grip attempts to make it possible to become the logging system for existing applications without needing to replace all logging call sites immediately (or ever!)
- Tools to support using grip interfaces and backends inside of other tools: you can use a grip Sender as a writer, or access a standard-library logger implementation that writes to the underlying grip logger.
logging should be fast, but speed of logging only really matters when data volume is really high, which is usually a problem for any logger: when data volume is lower even a really slow logger is still faster than the network or the file system. Picking a logger that’s fast, given this, is typically a poor optimization. Grip should be fast enough for any real world application and contains enough flexibility to provide an optimized path for distributing log messages as needed.
provide features to support some great logging paradigms:
- lazy construction of messages: while the speed of writing messages to their output handler typically doesn’t matter, sometimes building log messages can be intensive. Recently I ran into a case where calling [Sprintf]{.title-ref} at the call site of the logger for messages that were not logged (e.g. debug messages,) had material effects on the memory usage of the program. While string formating is a common case (and grip has “[Logf]{.title-ref}” style handlers that are lazy by default,) we found a bunch of metrics collection workloads that had similar sorts of patterns and lazy execution of these metrics tended to help a lot.
- conditional logging, rather than wrapping a log statement in an [if]{.title-ref} block, I found that you could have logging handlers that
- randomized logging or sampled logging: In some cases, I want to have log messages that only get logged half the time or something similar. In some high-volume code paths logging all the time is too much, and never is also too much, but it’s possible to devise a happy medium. Grip, for years now, has implemented this in terms of the conditional logging functionality.
- structured logging, rather than logging strings, it’s sometimes nice to just pass data, in the form of a special structure or just a map and let the logging system package deal with the output formating and transmission of messages. In general, line-oriented JSON makes for a good logging format, assuming most or enough of your messages have related structures and your log viewing and processing tools support this kind of data, although alternate human-readable string formats should also be available.

None of these things changed in the rewrite: by default, grip mostly just looks and works like the Go standard library logger (and even uses the logger under the hood for a lot of things,) but it was definitely fun to look at more contemporary logging practices and tools and think about what makes a logging library compelling (or not).

Infrastructure Libraries

Infrastructure libraries are an interresting beast: ideally every dependency carries some kind of maintenance cost, so you want to minimize the number of dependencies you require. At the same time, not using libraries is also bad because it means you end up writing more code and that has even more maintenance costs. It’s also the case that software engineers love writing infrastructure code and are often quite opinionated about the ergonomics of their infrastructure libraries.

On top of that, you make infrastructure software decisions once and then are sort of stuck with them for a really long time. I’ve changed loggers in projects and it’s rough, and in general you want to choose libraries: as an application developer you have the great feeling that no one’s differentiating feature is going to have anything to do with the logger¹ and you want something that’s battle tested and familiar to everyone. Sometimes--as in the logging package in Python--the standard library has a library just works and everyone just uses that; other times, there are one or two options that most project uses (e.g. log4j in java, etc.).

Even if grip is great, it seems unlikely that everyone (or anyone?) will switch to using grip over some of the other more popular options. I’m OK with that, I’m not sure that beyond writing a few blog posts I’m really that excited about doing the kind of marketing and promotion that it might take to promote a logging library like this, and at the same time the moment for a new logging library might have already passed.

Grip Reborn

The “new grip” is a pretty substantial change. A lot of implementation details changed and I deleted big parts of the interface that didn’t quite make sense, or that I thought were a bit fussy to use. Basically just sanding off a lot of awkward edges. The big changes:

greatly simplified dependencies, with more packages and an x hierarchy for extensions. The main grip package, and it’s primary sub packages' no longer has any external dependencies (beyond github.com/tychoish/fun.) Any logging backend or message type that has additional dependencies are in x.
I deleted a lot of code. There were a lot of things that just weren’t needed, there was an extra interface, a bunch of constructors for various objects that weren’t useful. I also simplified the concept of levels/priority within the logging system.
simpler high level logging interface. I used to have an extra interface and package to hold all of the Info, Error, (and so forth), and I cut a lot of that out and just made a Logger type in the main package which just wraps an underlying interface and doesn’t need to be mutable, and doing this made it possible to simplify a lot of the code.
added some straight forward handlers to attach loggers to contexts. I think previously, I was split on the opinion that loggers should either be (functionally) global or passed explicitly around as objects, and I think I’ve come around to the idea that loggers maybe ought to hang off the context object, but contextual loggers, global loggers, and logging objects are all available.
the high level logging interface is much smaller, with handlers for all the levels and formatting, line, and conditional (e.g. f, ln, When) logging. I’m not 100% sold on having ln, but I’m pretty pleased with having things be pretty simple and streamlined. The logger interface, as with the standard logger is mirrored in the grip package, with a shim.
new message.Builder interface and methods that provides a chainable interface for building messages without needing to mess with the internals of the message package which might be ungainly at logging call sites.
new KV message type: this makes it possible to have structured logging payloads without using map types, which might prove easier easier to integrate with the new zerolog and zap backends.
I have begun exploring in the [series]{.title-ref} package, what it’d mean to have a metrics collection and publication system that is integrated into a logger. I wrote probably too much code, but I am excited to do some more work to do some more work in this area.

Arguably, in a CI platform, most of the hard problems have something to do with logging, so this is an exception, but perhaps one of the only exceptions ↩︎

New Go Module: tychoish/fun

Wed, 16 Aug 2023 00:00:00 +0000

This is a follow up to my New Go Modules post about a project that I’ve been working on for the past few months: github.com/tychoish/fun.

fun is a collection of simple libraries using generics to do a collection of relatively mundane things, with a focus on well-built tools to make it easier for developers to solve higher level problems without needing to re-implement low level infrastructure, or use some rougher parts of the go standard library. It has no dependencies outside of the standard library, and contains a number of pretty cool tools, which were fun to write. Some of the basic structures were:

I wrote linked list implementations (single and double), adapted a single-ended Queue implementation that a former coworker did as part of an abandoned branch of development, and wrote a similar Deque. (fun/pubsub)
I adapted and developed a “broker” which uses the queues above to be able to do one-to-many pubsub implementations. (fun/pubsub)
I made a few high-level standard library synchronization tools (sync.Map, sync.WaitGroup, sync.Pool, and atomic.Value) even more higher level, with the help generics and more than a few stubborn opinions. (fun/adt; atomic data types)
I revisited an error aggregation tool that I wrote years ago, and made it a bunch faster, less quirky, and more compatible with the current state of the art with regards to error handling (e.g. errors.Is and errors.As). (fun/erc). I also wrote some basic error mangling tools including a more concise errors.Join, tools to unwind/unwrap errors, and a simple error type ers.Error to provide for constant errors (fun/ers)
I wrote an iterator interface and a collection of function wrappers and types (in the top level package), for interacting with iterators (or really, streams in one way or another,) and decorating those handlers and processors with common kinds of operations, modifications, and simple semantics.

I don’t know that it will catch on, but I’ve written (and rewritten) a lot of worker pools and stream processing things to use thse tools, and it’s been a useful programming model. By providing the wrappers and the iteration, users can implement features almost functionally (which should be easy to test.)
I built a collection of service orchestration tools to manage long running application services (e.g. http servers, worker pools, long running subprocesses, etc.) and a collection of context helpers to make it easier to manage the lifecycle of the kind of long-running applications I end up working on most of the time. Every time I’ve joined a new project… ever, I end up doing some amount of service orchestration work, and this is the toolkit I would want. (fun/srv)
I wrote some simple testing frameworks and tools for assertions (fun/assert) halt-on-failure, but with a tesitfy-inspired interface, and better reporiting along with a mirrored, fail-but-continue fun/assert/check, along with fun/testt (“test-tee” or “testy”) which has a bunch of simple helpers for using contexts, and fun/ensure, which is a totally different take on an assertion library.

I don’t want this post to be documentation about the project; there are a lot of docs in the README and on the go documentation, also the implementations are meant to be pretty readable, so feel free to dive in there. I did want to call out a few ways that I’ve begun using this library and the lessons it’s taught me in my day to day work.

I set a goal of writing code that’s 100% covered by tests. This is hard, and only possible in part because it’s a stateless library without dependencies, but I learned a lot about the code I was writing, and feel quite confident in it as a result.
I also set a goal of having no dependencies outside of the module and the standard library: I didn’t want to require users opt in using a set of tools that I liked, or that would require on going maintenance to update and manage. Not only is this a good goal in terms of facilitating adoption, it also constrained what I would do, and forced me to write things with external extensibility: there had to be hooks, interfaces and function types had to be easy to implement, and I decided to limit scope for other things.
Having a more complete set of atomic types and tools (particularly the map and the typed atomic value, also the typed integer atomic types in the standard library are great), has allowed me to approach concurrent programming problems without doing crazy things with channels or putting mutexes everywhere. I don’t think either channels or mutexes are a problem in the hands of a practiced practitioner, but having a viable alternative means it’s one less thing to go wrong, and you can save the big guns (mutexes) for more complex synchronization problems.
Linked lists are super cool. I’ve previously taken the opinion that you shouldn’t implement your own sequence types ever, and mostly avoided writing one of these before now. Having now done it, and now having truly double-ended structures means things like “adding something to the beginning” or “inserting into/removing from the middle” of a sequence isn’t so awkward.

The experimental slices library makes this a little less awkward with standard library slices/arrays, and they are proably faster.
It was really fun to take the concept of an interator and then build out from this concept to build tools that would make them easy to use, and got some good filtering, parallel processing, and map/reduce tools. I definitely learned a bunch but also I think (and have found) these tools useful.
I’ve long held that most go applications should be structured so that you shouldn’t really need to think too much about concurrency when writing business logic. I’ve previously tried to posit that the solution to this was to provide robust queue processing tools (e.g. amboy), but I think that’s too heavy weight, and it was cool to be able to think about the solution to this concept from a different angle.

Anyway, give it a try, and tell me what you think! Also pull requests are welcome!

New Go Modules

Tue, 15 Aug 2023 00:00:00 +0000

I’ve been doing a little big of programming for fun in recent months, and I wanted to do a little big of blogging here to explain myself, so expect that over the next few days.

By way of creating the facade of suspense--which the astute among you will be able to spoil by way of looking at my github--this post isn’t going to mention the actual projects, but rather answer two questions that are more interesting from a higher level:

Why have you done this?
What have you learned about yourself as a programmer?

Why Have You Done This?

First, while I’ve been writing Go for a long time, I must confess that I was a long time generic skeptic and hold out. Writing some code for myself and I wanted to get a better feeling for how they worked, now that they’re here.

In my day to day work, I have found myself writing similar code again and again: while I have definitely have a class of problems that I tend to work on, I’d be ok if I never wrote another function to bridge the gap between a [context.Context]{.title-ref} and a [sync.WaitGroup]{.title-ref} (and I’m not holding my breath for this in the standard library.)

Finally, over the years I’ve written a few projects that I’ve worked on professionally have been open source, and a few of them I’ve even rather liked, and I wanted to see what it would be like to revisit some of these projects with (potentially) wiser eyes and fingers.

What Have You Learned?

I actually think I’ve learned a lot:

While I’m a big fan of the part of software development which is “deleting code” in my professional work, it was really interesting to take that to code that I knew I’d written (or directed the writing of) and been able to see what I could cut.
I’m better at writing tests and writing testable code than I was even a few years ago. While I believe in writing defensive code, I found myself writing an implementation and then going through and deleting a lot of code that wasn’t useful or was (fundamentally) superstitious when I discovered that there was no way to trigger a potential case.
Because I’ve spent a few years mostly working on relatively high level projects and not having the bandwidth to work on lower (mid?) level infrastructural code, in practice I’ve spent more time assessing other people’s libraries and I was keenly aware that I was developing code that would only be used if someone chose to, and in thinking about how those decisions are made.
This isn’t new, but I care a lot about the erognomics of software. I think 5 or 6 years ago, ergonomics meant “code which was easy to use and provented users from doing wrong things,” and I think my view of erognomic code has expanded to include interfaces that are easy and obvious to use, and promote the users of my code to write better code.

Crafts at Scale

Fri, 11 Nov 2022 00:00:00 +0000

I’ve been kntting off and on since 2002 or 2003 (or so) but have been particularly “on” in the last couple of years. When I started working as a computer programmer (without formal training as such,) I quipped that I learned how to program from hand knitting. This is a simplification--of course--but it’s not that incorrect. Knitting is a system with some basic fundamentals (stitches, yarn, needle), a lot of variables (gauge, tension), repeated procedures, and a hell of a lot of prior art. This is a lot like programming.

Spinning too, has many of the same properties, but similarities aside they feel like different kinds of crafts: where knitting feels like you’re applying a set of understood procedures to produce something that’s unique, spinning is often about figuring out how to apply the same procedures in a way that prodcues consistent result. This makes sense you want to produce a quantity of yarn that’s on average similar enough that when you knit (or I suppose weave,) you have good consistent results. In many ways, spinning leads naturally to an idea of “production” or “scale” as an aspect of craft.

Just to be clear, these kinds of crafts should be fun and rewarding on their own merits. If you want to spin and are excited and happy to make and have yarn with variable thickness, or where every skein is unique, then do that. For me, particularly now, I find the problem of figuring out how to be consistent while spinning a couple of pounds of wool over the course of a few weeks to be really exciting and entrancing.

The kind of knitting that I’ve been doing recently has had some of these production/scale aspects as well: knitting with very similar white yarns removes color and minimizes texture as a variable. While I’ve been knitting roughly the same sock at production scale, the sweater’s I’ve been working on have some bespoke aspects, though the process is broadly similar. There’s something so compelling about being able to understand my craft and procedure so thoroughly that I can make things that aren’t wonky with confidence.

Programming is also very much like thsi for me these days. I spent years as a programmer trying to figure out how code worked, and how basic fundamental systems and protocols worked (e.g. webservers, Linux, databases,) and now I know how to build most things, or feel confident in my ability to figure out how to build a new thing when needed. The exciting things about software engineering is more about making the software work at large scale, the processes that allow increasingly large teams of engineers work together effectively, and being able to figure out the right solution for the problem users have.

I’m currently somewhere on the 7th 100 gram skein of approximately worsted weight, 3-ply merino yarn. My consistency isn’t quite where I want it, but if you look at all of the skeins they seem roughly related, so I think I’ll be able to make a sweater easily from it. I have two more skeins after this one. My plan from here is to alternate spinning batches of white yarn with spinning batches of not-white/natural colored wool for variety. Probably mostly 3 ply for now, though I may give 2 ply a go for one of them.

I’m knitting a white seamless style sweater, using Elizabeth Zimmerman’s method for bottom up sweaters. I’ve changed many of the numbers and some of the proportions, but nothing particularly fundamental about the process. I’ve knit 3 sweaters back to back with this same process, though this is the first with this specific yarn. I do have enough of this yarn to knit 3 or 4 sweaters, which I find both daunting and exciting, taken as a whole. With the sleeves done, I’m about halfway to the underarms on the body. I want to try knitting a saddle shouldered garment for this one.

Why Spinning?

Sat, 05 Nov 2022 00:00:00 +0000

A few years ago, I sent my spinning wheel away because I was living in a very small apartment with two very attentive cats. While I’ve been living in an apartment with more room (and doors!) for a few years now, only this week has my wheel returned: I realized that I missed spinning, and it’s not like soothing hobbies are unwelcome these days.

I started spinning about 15 years ago, and did it a bunch for a few years and then more or less stopped for a long time. It’s been interesting to start up again, and discover that my hands/body more or less remembered exactly how to do it. I had a few hours and about 200g of yarn to spin before some of the finer points came back and now I’ve spun a couple more skeins closer to my intention.

The other human asked “What do you like about spinning?"--well the question was phrased more like “is handspun yarn better?”, but I will paraphrase to better capture intent. There are, of course, a few answers:

the act of spinning is quite satisfying. Sometimes it’s enough for things to be fun and satisfying even if they aren’t productive.
the yarn can be sort of nifty, and although I’ve spun a lot of yarn, I have mostly not knit much with handspun yarn. I tend to like consistent and fine (fingering) yarns in my own knitting, and machines just do better at making this kind of yarn, so I end up giving a lot of handspun away to friends who I know will knit it better.
spinning gives you a lot of control over the wool (and kind of sheep) that go into the yarn you get, in a way that just doesn’t scale up to larger production schemes. I quite enjoy being able to first select what kind of sheep the wool I use comes from and then decide what kind of yarn I want from it. When other people spin, you can usually only pick one of these variables.

I’m currently spinning some white merino roving that I’ve (apparently had for years.) There’s a piece of paper in the bag that says “2 lbs” but between my practice skeins and whatever I did before I stopped, there’s probably only about a pound and a half left: this is fine. Merino is great, but it’s quite common and I knit a lot of merino. I’ve been working on getting a pretty stable 3-ply worsted weight yarn, and I’m roughly there. I like 3-ply because of the round construction, and worsted weight is about the heaviest yarn I’m really interested in knitting with or using (and it’s easy to design with/for!)

My next few spinning projects are with wool from different breeds of sheep (BFL! Targhee! Rambouillet!) though mostly undyed (and largly white), and mostly in larger batches (a pound or two.) I’ve never really gotten into hand-dyed roving, and mostly really enjoy spinning undyed wool: in most cases dying the finished garment or the yarn before knitting leads to the best result anyway. I guess one of the most The thing I like about spinning, in a lot of ways, is that it lets me focus on the wool and the sheep.

As a spinner, I’m far more interested in the wool and the sheep, in much the same way that as a knitter I’ve become far more interested in the structure of what I’m knitting than the color or the yarn. This feels entirely consistent to me: as a spinner I’m far more interested in the process and the wool than I am in yarn, and as a knitter I’m far more interested in using the yarn to explore the structure. Somehow, the yarn itself isn’t the thing that compells me, despite being kind of at the center of the process.

Anyway, back to the wheel!

The Most Plain Knitting

Wed, 21 Sep 2022 00:00:00 +0000

Last night I finished knitting a sweater that I’d been working on for either a while (pictures in this twitter thread) or not all that long, and promptly started the next sweater.

Also last weekend I handed off a bag of undyed (white) knitting to a friend of mine who is way more excited about dying than I am. This includes 13 or 14 pairs of socks (in a few different batches,) and a sweater that I knit. We also found someone who the sweater is more likely to fit than me, and I always quite like finding homes for wayward sweaters.

I have a couple of long flights for work trips coming up so I wanted to make sure that I wasn’t bringing a sweater that I was two-thirds of the way through and would likely finish. The next sweater is the 4th I’ve made from this yarn, and the 3rd plain sweater. I’ve made two plain raglans, and this last one was a crew neck.

By now I have a reasonable set of numbers/patterns for a “fingering weight sweater that basically fits an adult medium/small” that I’ve been honing, and enough yarn stashed to make about 9 of these sweaters. That should get me through the winter.

The crew neck is a touch lower than I think it needed to be, but it looks pretty smart. The thing about knitting Elizabeth Zimmerman-style seamless sweaters is that for the entire time you’re knitting the yoke section it really does not seem like it’s going to work out, so you have low-key panic the entire time, and then somehow, magically it all does. The key to success is to not overthink things too much and not fuck around.

I think this last sweater had a bit too much fucking around with the neckline, so it looks a bit weird (to my somewhat exacting tastes) where the ragland decreases interact with the neck shaping. The front of the neck could have been higher, and I think I could have done like 3-4 sets of short rows near the end to get the right effect for the front. Perhaps one of the next few sweaters can be another attempt at a raglan.

My plan for the next/current sweater is to do set-in sleeves with a crew neck. I have the math all worked out, so that seems like it might be fun. I’ve also never done EZ’s saddle shoulder (or hybrid) yoke, so that seems like some fun winter knitting. Regardless, saddles and set in sleeves are mostly constructed the same way, so I can wait quite a while to make a decision. After about 18 months of mostly knitting socks (and having gotten ~30 pairs done,) a (minor) change seems good.

Isolation Reading

Mon, 19 Sep 2022 00:00:00 +0000

I spent a few days last week isolating after attending a larger social event in a friends apartment in a (mostly) unfamiliar neighborhood and I got to spend a few days enjoying (a dear friend’s) book collection.

I don’t have many paper books left: enough moves and small new York city apartments, combined with vague personal preference for e-ink have left me with only about 100 books, but I do sometimes enjoy reading paper books when I’m visiting someone else. My perfect vacation has always been some combination of “drinking too much coffee and reading books,” and given that I’m kind of in an in-between moment job-wise right now, this was actually pretty much perfect.

I started out the week reading Slouching Toward Bethlehem (Joan Diddion) and finished it reading the first half of The God of Small Things (Arundhati Roy). It was pretty much everything.

I’ve always been an admirer of Diddon, but I’ve never read Bethlehem and I’ve meant to sort of spend a few years trawlling luxuriously through her backlist, but hadn’t gotten around to it. The writing is perfect in exactly the sort of austere but precise way that I’ve come to expect. I’m doubly impressed also that she was so young when these essays were published.

I had in my mind that this was a book that was an account of the state of counter-culture in the 60s, and the title essay definitely is that, but having read the entire book over the course of a few days, I’m left with the impression that this book is really a big “why I left New York City in my late 20s” combined with a love letter to California from a returning native child, who remembers “the (really) old California” and what is by now “the (simply) old California”.

The “why I left New York City in my late 20s” story is pretty familiar, and it’s actually nice to see, now 60 years on, that people coming to New York in their 20s and then burning out or not figuring out how to be in New York sustainably is a very old story indeed. I’m also of course, heartened that she returned to the city for the last 25 years of her life. I hope that this also proves to be an enduring pattern for my generation.

I was also struck by the way that the reflection (and really, critique) of the counter culture managed to be very early but also consistent with what a lot of people were saying earlier. To my eyes, it’s not particularly surprising but the date is a bit.

The God of Small Things is, of course, lush in all the ways that Slouching is austere. Almost provoking whiplash.

I typically find these sort of lush non-linear books to be a bit Extra. Lovely, to be sure, but the lushness and non-linearity can so distract from the plot or the characters or the impact. Lush and non-linear prose has also started to feel faddish and at least for me, a signifies of a certain kind of academic/“art school” approach to prose. This is not true at all of Small Things: the story directly and explicitly explores childhood memories and trauma in ways that are reflected both in the characters and the story telling. It extremely works.

As is, I suppose, the intent, the book and writing has me thinking a lot about imperialism¹ and the history therein, and I think there’s a way that the non-linearity of the story telling manages to engage this fundamental question² “why do people fight for their servitude as if it were their salvation,” and watching this

I’m not done yet with the book, but I’m excited to dig in more.

The next book on my friend’s bookshelf that I’m excited by was a collection of Grace Paley stories and essays. I haven’t really started it, yet, but I think I will soon.

I wrote this sentence as “post/colonialism” but I think there are so many layers and intersections that expand have echos and impacts that are much larger than the history of the British in India, which isn’t (and shouldn’t!) be the at the center of the story, despite it’s outsized and unrefutable impact. ↩︎
In a bit of my own non-linearity, I’ve been working on an essay that plays with this famous quote/question from Deleuze (derived from Riech, derived from Spinoza). The full (ish) quote is, “the fundamental problem of political philosophy is still precisely the one that Spinoza saw so clearly, and that Wilhelm Reich rediscovered: ‘Why do men fight for their servitude as stubbornly as though it were their salvation?’ How can people possibly reach the point of shouting: ‘More taxes! Less bread!’? As Reich remarks, the astonishing thing is not that some people steal or that others occasionally go out on strike, but rather that all those who are starving do not steal as a regular practice, and all those who are exploited are not continually out on strike.” ↩︎

Software Engineering for 2.0

Thu, 07 Jul 2022 00:00:00 +0000

I’ve been thinking about what I do as a software engineer for a while, as there seems to be a common thread through the kinds of projects and teams that I’m drawn toward and I wanted to write a few blog posts on this topic to sort of collect my thoughts and see if these ideas resonated with anyone else.

I’ve never been particularly interested in building new and exciting features. Hackathon’s have never held any particular appeal, and the things I really enjoy are working on are on the spectrum of “stabilize this piece of software,” or “make this service easy to operate” or “refactor this code to make support future development” and less “design and build some new feature.” Which isn’t to say that I don’t like building new features or writing code, but that I’m more driven by the code and supporting my teammates than I am by the feature.

I think it’s great that I’m different from software engineers who are really focused on the features, both because I think the tension between our interests pushes both classes of software engineer to do great things. Feature development keeps software and products relevant and addresses users' needs. Stabilization work makes projects last and reduces the incidence of failures that distract from feature work, and when there’s consistent attention paid to aligning infrastructure¹ work with feature development of the long term, infrastructure engineers can significantly lower the cost of implementing a feature.

The kinds of projects that fall into these categories inculde the following areas:

managing application state and workload in larger distributed contexts. This has involved designing and implementing things like configuration management, deployment processes, queuing systems, and persistence layers.
concurrency control patterns and process lifecycle. In programming environments where threads are available, finding ways to ensure that processes can safely shut down, and errors can be communicated between threads and processes takes some work and providing mechanisms to shutdown cleanly, communicate abort signals to worker threads, and handle communication patterns between threads in a regular and expected way, is really important. Concurrency is a great tool, but being able to manage concurrency safely and predictably and in descret parts of the code are useful.
programming model and ergonomic APIs and services. No developers produces a really compelling set of abstractions on the first draft, particularly when they’re focused on delivering different kinds of functionality. The revision and iteration process helps everyone build better software.
test infrastructure and improvements. No one thinks tests should take a long time or report results non-deterministically, and yet so many test are. The challenge is that tests often look good or seem reasonable or are stable when you write them, and their slow runtimes compound overtime, or orthogonal changes make them slower. Sometimes adding an extra check in some pre-flight test-infrastructure code ends ups causing tests that had been just fine, thank you to become problems. Maintaining and structure test infrastructure has been a big part of what I’ve ended up doing. Often, however, working back from the tests, it’s possible to see how a changed interface or an alternate factoring of code would make core components easier to test, and doing a cleanup pass of tests on some regular cadence to improve things. Faster more reliable tests, make it possible to develop with greater confidence.

In practice this has included:

changing the build system for a project to produce consistent artifacts, and regularizing the deployment process to avoid problems during deploy.
writing a queuing system without any extra service level dependencies (e.g. in the project’s existing database infrastructure) and then refactoring (almost) all arbitrary parallel workloads to use the new queuing system.
designing and implementing runtime feature flagging systems so operators could toggle features or components on-and-off via configuration options rather than expensive deploys.
replacing bespoke implementations with components provided by libraries or improving implementation quality by replacing components in-place, with the goal of making new implementations more testable or performant (or both!)
plumbing contexts (e.g. Golang’s service contexts) through codebases to be able to control the lifecycle of concurrent processes.
implementing and migrating structured logging systems and building observability systems based on these tools to monitor fleets of application services.
Refactoring tests to reuse expensive test infrastructure, or using table-driven tests to reduce test duplication.
managing processes' startup and shutdown code to avoid corrupted states and efficiently terminate and resume in-progress work.

When done well (or just done at all), this kind of work has always paid clear dividends for teams, even when under pressure to produce new features, because the work on the underlying platform reduces the friction for everyone doing work on the codebase.

It’s something of an annoyance that the word “infrastructure” is overloaded, and often refers to the discipline of running software rather than the parts of a piece of software that supports the execution and implementation of the business logic of user-facing features. Code has and needs infrastructure too, and a lot of the work of providing that infrastructure is also software development, and not operational work, though clearly all of these boundaries are somewhat porous. ↩︎

Systems Administrators are the Problem

Wed, 06 Jul 2022 00:00:00 +0000

For years now, the idea of the terrible stack, or the dynamic duo of Terraform and Ansible, from this tweet has given me a huge amount of joy, basically anytime someone mentions either Terraform or Ansible, which happens rather a lot. It’s not exactly that I think that Terriform or Ansible are exactly terrible: the configuration management problems that these pieces of software are trying to solve are real and actually terrible, and having tools that help regularize the problem of configuration management definitely improve things. And yet the tools leave things wanting a bit.

Why care so much about configuration management?

Configuration matters because every application needs some kind of configuration: a way to connect to a database (or similar), a place to store its output, and inevitably other things, like a dependencies, or feature flags or whatever.

And that’s the simple case. While most things are probably roughly simple, it’s very easy to have requirements that go beyond this a bit, and it turns out that while a development team might--but only might--not have requirements for something that qualifies as “weird” but every organization has something.

As a developer, configuration and deployment often matters a bunch, and it’s pretty common to need to make changes to this area of the code. While it’s possible to architect things so that configuration can be managed within an application (say), this all takes longer and isn’t always easy to implement, and if your application requires escalated permissions, or needs a system configuration value set then it’s easy to get stuck.

And there’s no real way to avoid it: If you don’t have a good way to manage configuration state, then infrastructure becomes bespoke and fragile: this is bad. Sometimes people suggest using image-based distribution (so called “immutable infrastructure,") but this tends to be slow (images are large and can take a while to build,) and you still have to capture configuration in some way.

But how did we get here?

I think I could weave a really convincing, and likely true story about the discipline of system administration and software operations in general and its history, but rather than go overboard, I think the following factors are pretty important:

computers used to be very expensive, were difficult to operate, and so it made sense to have people who were primarily responsible for operating them, and this role has more or less persisted forever.
service disruptions can be very expensive, so it’s useful for organizations to have people who are responsible for “keeping the lights on,” and troubleshoot operational problems when things go wrong.
most computer systems depend on state of some kind--files on disks, the data in databases--and managing that state can be quite delicate.
recent trends in computing make it possible to manipulate infrastructure--computers themselves, storage devices, networks--with code, which means we have this unfortunate dualism of infrastructure where it’s kind of code but also kind of data, and so it feels hard to know what the right thing to do.

Why not just use <xyz>

This isn’t fair, really, but and you know it’s gonna be good when someone trivializes an adjacent problem domain with a question like this, but this is my post so you must endure it, because the idea that there’s another technology or way of framing the problem that makes this better is incredibly persistent.

Usually <xyz>, in recent years has been “Kubernetes” or “docker” or “containers,” but it sort of doesn’t matter, and in the past solutions platforms-as-a-service (e.g. AppEngine/etc.) or backend-as-a-service (e.g. parse/etc.) So let’s run down some answers:

“bake configuration into the container/virtual machine/etc. and then you won’t have state,” is a good idea, except it means that if you need to change configuration very quickly, it becomes quite hard because you have to rebuild and deploy an image, which can take a long time, and then there’s problems of how you get secrets like credentials into the service.
“use a service for your platform needs,” is a good solution, except that it can be pretty inflexible, particularly if you have an application that wasn’t designed for the service, or need to use some kind of off-the-shelf (a message bus, a cache, etc.) service or tool that wasn’t designed to run in this kind of environment. It’s also the case that the hard cost of using platforms-as-a-service can be pretty high.
“serverless” approaches something of a bootstrapping problem, how do you manage the configuration of the provider? How do you get secrets into the execution units?

What’s so terrible about these tools?

The tools can’t decide if configuration should be described programatically, using general purpose programming languages and frameworks (e.g. Chef, many deployment tools) or using some kind of declarative structured tool (Puppet, Ansible), or some kind of ungodly hybrid (e.g. Helm, anything with HCL). I’m not sure that there’s a good answer here. I like being able to write code, and I think YAML-based DSLs aren’t great; but capturing configuration creates a huge amount of difficult to test code. Regardless, you need to find ways of being able to test the code inexpensively, and doing this in a way that’s useful can be hard.
Many tools are opinionated have strong idioms in hopes of helping to make infrastructure more regular and easier to reason about. This is cool and a good idea, it makes it harder to generalize. While concepts like immutability and idempotency are great properties for configuration systems to have, say, they’re difficult to enforce, and so maybe developing patterns and systems that have weaker opinions that are easy to comply with, and idioms that can be applied iteratively are useful.
Tools are willing to do things to your systems that you’d never do by hand, including a number of destructive operations (terraform is particularly guilty of this), which erodes some of their trust and inspires otherwise bored ops folks, to write/recapitulate their own systems, which is why so many different configuration management tools emerge.

Maybe the tools aren’t actually terrible, and the organizational factors that lead to the entrenchment of operations teams (incumbency, incomplete cost analysis, difficult to meet stability requirements,) lead to the entrenchment of the kinds of processes that require tools like this (though causality could easily flow in the opposite direction, with the same effect.)

API Ergonomics

Tue, 05 Jul 2022 00:00:00 +0000

I touched on the idea of API ergonomics in Values for Collaborative Codebases, but I think the topic is due a bit more exploration. Typically you think about an API as being “safe” or “functionally complete,” or “easy to use,” but “ergonomic” is a bit far afield from the standard way that people think and talk about APIs (in my experience.)

I think part of the confusion is that “API” gets used in a couple of different contexts, but let’s say that an API here are the collection of nouns (types, structures,) and verbs (methods, functions) used to interact with a concept (hardware, library, service). APIs can be conceptually really large (e.g. all of a database, a public service), or quite small and expose only a few simple methods (e.g. a data serialization library, or some kind of hashing process.) I think some of the confusion is that people also use the term API to refer to the ways that services access data (e.g. REST, etc.) and while I have no objection to this formulation, service API design and class or library API design feel like related but different problems.

Ergonomics, then is really about making choices in the design of an API, so that:

functionality is discoverable during programming. If you’re writing in a language with good code completion tools, then make sure methods and functions are well located and named in a way to take advantage of completion. Chainable APIs are awesome for this.
use clear naming for functions and arguments that describe your intent and their use.
types should imply semantic intent. If your programming language has a sense of mutability (e.g. passing references verses concrete types in Go, or const (for all its failings) in C++), then make sure you use these markers to both enforce correct behavior and communicate intent.
do whatever you can to encourage appropriate use and discourage inappropriate use, by taking advantage of encapsulation features (interfaces, non-exported/private functions, etc.), and passing data into and out of the API with strongly/explicitly-typed objects (e.g. return POD classes, or enumerated values or similar rather than numeric or string types.)
reduce the complexity of the surface area by exporting the smallest reasonable API, and also avoiding ambiguous situations, as with functions that take more than one argument of a given type, which leads to cases where users can easily (and legally) do the wrong thing.
increase safety of the API by removing or reducing and being explicit about the API’s use of global state. Avoid providing APIs that are not thread safe. Avoid throwing exceptions (or equivalents) in your API that you expect users to handle. If users pass nil pointers into an API, its OK to throw an exception (or let the runtime do it,) but there shouldn’t be exceptions that originate in your code that need to be handled outside of it.

Ergonomic interfaces feel good to use, but they also improve quality across the ecosystem of connected products.

Common Gotchas

Thu, 30 Jun 2022 00:00:00 +0000

This is a post I wrote a long time ago and never posted, but I’ve started getting back into doing some work in Common Lisp and thought it’d be good to send this one off.

On my recent “(re)learn Common Lisp” journey, I’ve happened across a few things that I’ve found frustrating or confusing: this post is a collection of them, in hopes that other people don’t struggle with them:

Implementing an existing generic function for a class of your own, and have other callers specialize use your method implementation you must import the generic function, otherwise other callers will (might?) fall back to another method. This makes sense in retrospect, but definitely wasn’t clear on the first go.
As a related follow on, you don’t have to define a generic function in order to write or use a method, and I’ve found that using methods is actually quite nice for doing some type checking, at the same time, it can get you into a pickle if you later add the generic function and it’s not exported/imported as you want.
Property lists seem cool for a small light weight mapping, but they’re annoying to handle as part of public APIs, mostly because they’re indistinguishable from regular lists, association lists are preferable, and maybe with make-hash even hash-tables.
Declaring data structures inline is particularly gawky. I sometimes want to build a list or a hash map in-line an alist, and it’s difficult to do that in a terse way that doesn’t involve building the structure programatically. I’ve been writing (list (cons "a" t) (cons "b" nil)) sort of things, which I don’t love.

You could render this as:
```
`(("a" . t) ("b" . nil)) 
```
Having said that, I’ve always found the back-tick hard to read, so I tend to disprefer it.
If you have a variadic macro (i.e. that takes &rest args), or even I suppose any kind of macro, and you have it’s arguments in a list, there’s no a way, outside of eval to call the macro, which is super annoying, and makes macros significantly less appealing as part of public APIs. My current conclusion is that macros are great when you want to add syntax to make the code you’re writing clearer or to introduce a new paradigm, but for things that could also be a function, or are thin wrappers on for function, just use a function.

How to Choose a Programming Language

Wed, 29 Jun 2022 00:00:00 +0000

I talk to lots of people about software and programming: people who are trying to make a technical decision for a new project or who are interested in learning something new, and some form of the question “what programming language should I learn?” or “what’s the best language for this new project?” comes up a lot.

These are awful questions, because there is no singular right answer, and in some senses all answers are wrong. This post will be an exploration of some decent answers to this question, and some useful ways to think about the differences between programming languages.

If you already build and maintain software in one programming language, build new components in the same language you already use. Adding new tools and technologies increases maintenance burden for all engineers, and software tends to stick around for a long time, so this cost can stick around for a long time.
Sometimes the software you want to write must target a specific runtime or environment, there’s really only one reasonable choice. The prototypical examples of these are things like: iOS apps (Swift,) Android apps (Kotlin), or things that run in the browser (JavaScript,) although:
Given things like React Native and Electron, it’s reasonable to just write JavaScript for all GUI code, although often this might actually mean TypeScript in practice. While it used to be the case that it made sense to write GUI code in various native tool kits, at this point it seems like it makes sense to just figure out ways of doing it all in JS.
If you already know how to program in one language, and want to learn something new, but don’t have a specific project in mind attempt to learn something that’s quite different from you already know: if you’re comfortable in something like Python, try and learn something like Go or Rust. If you’re primarily a Java programmer, something like JavaScript or Python might be an interesting change of pace.

The same basic ideas applies to selecting languages that will be used by teams: choose a tool that’s complementary to what you’re already doing, and that could provide value.
If you’re more familiar with a few programming languages or don’t feel you need to learn a new language for professional reasons pick something fun and off the wall: Ocaml! Common Lisp! Rust! Haskell! Scheme! Elixir! It doesn’t matter and in these cases you probably can probably learn new languages when you need, the point is to learn something that’s radically different and to help you think about computers and programming in radically different ways.
Choose the language that people working on similar projects are already using. For instance, if you’re doing a lot of data science, using Python makes a lot of sense; if you’re writing tools that you expect banks (say) to use, something that runs on the JVM is a good bet. The idea here is you may be able to find more well developed tools and resources relevant to the kinds of problems you encounter.
When starting a new project and there isn’t a lot of prior art in the area that you’re working, or you want to avoid recapitulating some flaw in the existing tools, you end up having a lot of freedom. In general:
- Think about concurrency and workload characteristics. Is the workload CPU, Network, or IO bound? Is the application multithreaded, or could take advantage of parallelism within processes? There are different kinds of concurrency, and different execution models, so this isn’t always super cut-and-dry: theoretically languages that have “real threads” (C/C++, Java, Rust, Common Lisp, etc.) or a close enough approximation (Go,) are better, but for workloads that are network bound, event-driven systems (e.g. Python’s Tornado and Node.J’s) work admirably.
- How will you distribute and run the application? There are some languages that can provide static binaries that include all of their dependencies for distribution, which can simplify some aspects of distribution and execution process, but for software that you control the runtime (e.g. services deployed on some kind of container based-platform,) it might matter less.
- Are there strong real-time requirements? If so, and you’re considering a garbage collected language, make sure that the GC pauses aren’t going to be a problem. It’s also the case that all GCsare not the same, so having a clear idea of what the tolerances
- Is this software going to be maintained by a team, and if so, what kind of tools will they need in order to succeed and be productive. Would static typing help? What’s the developer tooling and experience like? Are there libraries that you’d expect to need that are conspicuously missing?

Have fun! Build cool things!

Spheres of Alignment

Tue, 28 Jun 2022 00:00:00 +0000

This is a post in my alignment series. See the introductory post Finding Alignment for more context.

I think, in practice, most of what managers do--and indeed all leadership--is about building alignment. The core concept, of alignment, having a shared understanding of the problem space and its context combined with relevant goals and objectives, and grasp of how the context contexts to these objectives. Alignment isn’t just “agreement” or “understanding the solution,” and really centers on this connection between context and goals. Alignment shows up in many different situations and interactions:

a small working group (2-4 people) who are working on building or developing something. The thing can be any kind of work product: a piece of software, documentation, a business process, a marketing campaign, a sales deal. When you have more than one person working on something, if they’re not aligned, each person may be able to work on a piece of work as delegated or assigned, but lacks the ability to (reliably) continue to work on the next piece of work after finishing a narrow task, or be able to assess if a line of work is still germane to the goals as things develop. If we view people’s roles in projects as machines, and they perform assigned tasks well, then alignment isn’t super critical, but if you need people to make decisions and act upon them, then they have to be aligned, as a group otherwise the project runs a huge risk of stalling out as each contributor pulls in an opposite direction.
one person aligning with the rest of their team to understand how their background and personal goals contribute to and interact with the team’s context and goals. Individuals all bring unique skills and interests and (hopefully) useful to teams, and teams (e.g. their leaders) need to be able to understand how to effectively use those skills and interests to support the team’s goals. This happens over conversation and in the context of someones participation in a team over time, and doesn’t need to take a lot of time on a regular basis, but cannot be entirely abandoned.
managers need to align their teams with the company’s objectives. This takes the form of making sure that the projects that the team is working on (and will work on in the future,) support the organization and company’s larger goals.
across all level each team needs to align with its peer teams and the organization that it belongs in. This is true in organizations with 30 people and 3-4 teams, and in organizations of 2000 people and dozens of teams.

Alignment is hierarchical, and largely the responsibility of leaders to monitor alignment above and below them, and understand if their teams or specific contributors are falling out of alignment. This doesn’t necessarily mean that it’s not participatory and discursive: individuals can impact the direction, goals, or alignment of their teams, but there must be well formed goals of the organization (that they can understand!) and they must be supported by their team in order to actualize in this dimension. Despite being hierarchical, and individuals and teams must align up building and maintaining alignment in all directions is actually the responsibility of leadership at all levels.

It’s easy to frame this as “you must align with the goals sent from above,” this couldn’t be further from the truth. Some organizations function like this, but it’s probably not healthy for anyone, because the kinds of alignment that it builds are fleeting and tactical. Teams and contributors do need to align with broader goals (up), but their job is not building alignment, it’s building whatever their specialty is: attending to the organizational health and alignment is the concern of leadership whose work must center on building alignment. At almost every level, the alignment goes both ways: you work with leaders above you to align your own work and team, and work with the people you collaborate and mentor to build alignment.

When it works, and even though it takes a while, it helps teams and organizations work really well.

Easy Mode, Hard Mode

Thu, 23 Jun 2022 00:00:00 +0000

I’ve been thinking more recently about that the way that we organize software development projects, and have been using this model of “hard mode” vs “easy mode” a bit recently, and thought it might be useful to expound upon it.

Often users or business interests come to software developers and make a feature request. “I want it to be possible to do thing faster or in combination with another operation, or avoid this class of errors.” Sometimes (often!) these are reasonable requests, and sometimes they’re even easy to do, but sometimes a seemingly innocuous feature or improvement are really hard sometimes, engineering work requires hard work. This isn’t really a problem, and hard work can be quite interesting. It is perhaps, an indication of an architectural flaw when many or most easy requests require disproportionately hard work.

It’s also the case that it’s possible to frame the problem in ways that make the work of developing software easier or harder. Breaking problems into smaller constituent problems make them easier to deal with. Improving the quality of the abstractions and testing infrastructure around a problematic area of code makes it easier to make changes later to an area of the code.

I’ve definitely been on projects where the only way to develop features and make improvement is to have a large amount of experience with the problem domain and the codebase, and those engineers have to spend a lot of consentrated time building features and fighting against the state of the code and its context. This is writing software in “hard mode,” and not only is the work harder than it needs to be, features take longer to develop than users would like. This mode of development makes it very hard to find and retain engineers because of the large ramping period and consistently frustrating nature of the work. Frustration that’s often compounded by the expectation or assumption that easy requests are easy to produce.

In some ways the theme of my engineering career has been work on taking “hard mode projects” reducing the barriers to entry in code bases and project so that they become more “easy mode projects”: changing the organization of the code, adding abstractions that make it easier to develop meaningful features without rippling effects in other parts of the code, improving operational observability to facilitate debugging, restructuring project infrastructure to reduce development friction. In general, I think of the hallmarks of “easy mode” projects as:

abstractions and library functions exist for common tasks. For most pieces of “internet infrastructure” (network attached services,) developer’s should be able to add behavior without needing to deal with the nitty gritty of thread pools or socket abstractions (say.) If you’re adding a new REST request, you should be able to just write business logic and not need to think about the applications threading model (say). If something happens often (say, retrying failed requests against upstream API,) you should be able to rely on an existing tool to orchestrate retries.
APIs and tools are safe ergonomic. Developers writing code in your project should be able to call into existing APIs and trust that they behave reasonably and handle errors reasonably. This means, methods should do what they say, and exported/public interfaces should be difficult to use improperly, and (e.g. expected exception handling/safety, as well as thread safety and nil semantics ad appropriate.) While it’s useful to interact with external APIs defensively, you can reduce the amount of effort by being less defensive for internal/proximal APIs.
Well supported code and operational infrastructure. It should be easy to deploy and test changes to the software, the tests should run quickly, and when there’s a problem there should be a limited number of places that you could look to figure out what’s happening. Making tests more reliable, improving error reporting and tracing, exposing more information to metrics systems, to make the behavior of the system easier to understand in the long term.
Changes are scoped to support incremental development. While there are lots of core technical and code infrastructure work that support making projects more “easy mode” a lot of this is about the way that development teams decide to structure projects. This isn’t technical, ususally, but has more to do with planning cadences, release cadences, and scoping practices. There are easier and harder ways of making changes, and it’s often worthwhile to ask yourself “could we make this easier.” The answer, I’ve found, is often “yes”.

Moving a project from hard to easy mode is often in large part about investing in managing technical debt, but it’s also a choice: we can prioritize things to make our projects easier, we can make small changes to the way we approach specific projects that all move projects toward being easier. The first step is always that choice.

Methods of Adoption

Wed, 22 Jun 2022 00:00:00 +0000

Before I started actually working as a software engineer full time, writing code was this fun thing I was always trying to figure out on my own, and it was fun, and I could hardly sit down at my computer without learning something. These days, I do very little of this kind of work. I learn more about computers by doing my job and frankly, the kind of software I write for work is way more satisfying than any of the software I would end up writing for myself.

I think this is because the projects that a team of engineers can work on are necessarily larger and more impactful. When you build software with a team, most of the time the product either finds users (or your end up without a job.) When you build software with other people and for other people, the things that make software good (more rigorous design, good test discipline, scale,) are more likely to be prevalent. Those are the things that make writing software fun.

Wait, you ask “this is a lisp post?” and “where is the lisp content?” Wait for it…

In Pave the On/Off Ramps¹ I started exploring this idea that technical adoption is less a function of basic capabilities or numbers of features in general, but rather about the specific features that support and further adoption and create confidence in maintenance and interoperability. A huge part of the decision process is finding good answers to “can I use these tools as part of the larger system of tools that I’m using?” and “can I use this tool a bit without needing to commit to using it for everything?”

Technologies which are and demand ideological compliance are very difficult to move into with confidence. A lot of technologies and tools demand ideological compliance, and their adoption depends on once-in-a-generation sea changes or significant risks.² The alternate method, to integrate into people’s existing workflows and systems, and provide great tools that work for some usecases and to prove their capability is much more reliable: if somewhat less exciting.

The great thing about Common Lisp is that it always leans towards the pragmatic rather than the ideological. Common Lisp has a bunch of tools--both in the langauge and in the ecosystem--which are great to use but also not required. You don’t have to use CLOS (but it’s really cool), you don’t have to use ASDF, there isn’t one paradigm of developing or designing software that you have to be constrained to. Do what works.

I think there are a lot of questions that sort of follow on from this, particularly about lisp and the adoption of new technologies. So let’s go through the ones I can think of, FAQ style:

What kind of applications would a “pave the exits” support?

It almost doesn’t matter, but the answer is probably a fairly boring set of industrial applications: services that transform and analyze data, data migration tools, command-line (build, deployment) tools for developers and operators, platform orchestration tools, and the like. This is all boring (on the one hand,) but most software is boring, and it’s rarely the case that programming langauge actually matters much.

In addition, CL has a pretty mature set of tools for integrating with C libaries and might be a decent alternative to other langauges with more complex distribution stories. You could see CL being a good langauge for writing extensions on top of existing tools (for both Java with ABCL and C/C++ with ECL and CLASP), depending.
How does industrial adoption of Common Lisp benefit the Common Lisp community?

First, more people writing common lisp for their jobs, which (assuming they have a good experience,) could proliferate into more projects. A larger community, maybe means a larger volume of participation in existing projects (and more projects in general.) Additionally, more industrial applications means more jobs for people who are interested in writing CL, and that seems pretty cool.
How can CL compete with more established languages like Java, Go, and Rust?

I’m not sure competition is really the right model for thinking about this: there’s so much software to write that “my langauge vs your langauge” is just a poor model for thinking about this: there’s enough work to be done that everyone can be successful.

At the same time, I haven’t heard about people who are deeply excited about writing Java, and Go folks (which I count myself among) tend to be pretty pragmatic as well. I see lots of people who are excited about Rust, and it’s definitely a cool langauge though it shines best at lower level problems than CL and has a reasonable FFI so it might be the case that there’s some exciting room for using CL for higher level tasks on top of rust fundamentals.

In line with the idea that product management and design is about identifying what people are doing and then institutionalizing this is similar to the urban planning idea of “paving cowpaths,” I sort of think of this as “paving the exits,” though I recognize that this is a bit force.d ↩︎
I’m thinking of things like the moment of enterprise “object oriented programing” giving rise to Java and friends, or the big-data watershed moment in 2009 (or so) giving rise to so-called NoSQL databases. Without these kinds of events you the adoption of these big paradigm-shifting technologies is spotty and relies on the force of will of a particular technical leader, for better (and often) worse. ↩︎

Signs of Alignment

Tue, 21 Jun 2022 00:00:00 +0000

This is a post in my alignment series. See the introductory post Finding Alignment for more context.

I really want to dig into some topics related to building alignment and figuring out when you’re aligned as a contributor, or when the people you’re working with are falling out of alignment with you and/or your team or organization, but I think it’s worth it to start slow and chew on a big question: What it feels like when you and your team are well aligned, and why that’s a good thing.

To my mind, when you have a foundation of alignment, and an understanding of what the business goals are for your organization, then it becomes really easy to work independently, because you know what’s important, you know what needs to happen next and the people your working for/with can be confident that you’ll be moving in the right direction, and don’t need to do as much monitoring. Every so often, teams find this, and can really grind on it and deliver great features and products on the basis of this. It takes a long time (months!) for a team to gel like this, and sometimes teams don’t quite get there.

This isn’t to say that needing more guidance and wokring less independently means that you’re unaligned just that you (or the people you’re working with/for) are newer to the team, or there’s been a change recently and everyone needs more touch points to build alignment. One of the risks of hiring people and growing teams that are really well aligned is that the change in team dynamic can throw off alignment, and I think this is one of the reasons that teams sometimes struggle to grow. In any case, while alignment is great and it doesn’t happen for free, and it’s fine for it to be a thing you’re working on.

Alignment also reduces a lot of potentially contentious conversations and interactions: when you have alignment within a team or between teams you have a framework for prioritizing decisions: the most possible things that have the largest positive impact on the goals that you have are more important than… everything else. It all ends up being pretty simple. Sometimes you have to spend a bit of time on something that’s locally lower priority if another team depends on it, or if you’re helping someone learn something, but for the most part alignment helps you move toward the right direction.

When teams (and contributors) lack alignment, it’s easy for low priority work to get done, or projects that don’t end up supporting the business goals and so fail to find use (projects fail for other reasons, some of which are expected, so failed projects don’t necessarily indicate miss-alignment). An unaligned team can end up competing with peer teams and internal collaborators. If some parts of a team or organization are well aligned and other’s aren’t, resentment and frustration can brew between teams. Basically, without alignment you can--if you’re lucky--skate by with a little wasted effort, but often alignment deficits are a blight that can threaten a team’s ability to be productive and make it really hard to retain great team members.

Not everything is an alignment problem: teams and projects fail for technical or logistical reasons. Sometimes conflicts emerge between collaborators who are well aligned but working on disconnected projects, or hold different concerns within a project. Alignment is a framework for understanding how organizations can move together and be productive particularly as they grow, and in this I hope that this has been helpful!

Tips for More Effective Multi-Tasking

Thu, 16 Jun 2022 00:00:00 +0000

I posted something about how I organize my own work, and I touched on “multi-tasking,” and I realized immediately that I had touched something that required a bit more explanation.

I feel like a bit of an outlier to suggest that people spend time learning how to multitask better, particularly when the prevaling conventional wisdom is just “increase focus,” “decrease multitasking,” reduce “context switches,” between different tasks. It’s as if there’s this mythical word where you can just “focus more” taking advantage of longer blocks of time, with fewer distractions, and suddenly be able to get more done.

This has certainly never been true of my experience.

I was, perhaps unsurprisingly, a bit disorganized as a kid. Couldn’t sit still, forgot deadlines, focused inconsistently on things: sometimes I was unstoppable, and sometimes nothing stuck. As an adult, I’ve learned more about myself and I know how to provide the kind of structure I need to get things done, even for work that I find less intrinsically fascinating. Also I drink a lot more caffeine. I’m also aware that with a slightly different brain or a slightly different set of coping strategies, I would struggle a lot more than I do.

There are a lot of reasons why it can be difficult to focus, but I don’t think the why matters much here: thinking pragmatically about how to make the most of the moments we do have, the focus that’s available. Working on multiple things just is, and I think to some extent its a skill that we can cultivate or at least approximate. Perhaps some of the things I do would be useful to you:

fit your tasks to the attention you have. I often write test code later in the day or during my afternoon slump between 2-3:30, and do more complicated work with my morning coffee between 9:30 and 11:30, and do more writing later in the day. There are different of tasks, and knowing what kinds of work makes sense for which part of the day can be a great help.
break tasks apart as small as you can do, even if it’s just for yourself. It’s easy to get a little thing done, and bigger tasks can be intimidating. If the units of work that you focus on are the right size it’s possible to give yourself enough time to do the work that you need to do and intersperse tasks from a few related projects.
plan what you do before you do it, and leave yourself notes about your plan. As I write code I often write a little todo list that contains the requirements for a function. This makes it easy to pick something up if you get interrupted. My writing process also involves leaving little outlines of paragraphs that I want to write or narrative elements that I want to pass.
leave projects, when possible, at a stopping point. Make it easy for yourself to pick it back up when you’re ready. Maybe this means making sure that you finish writing a test or some code, rather than leaving a function half written. When writing prose, I sometimes finish a paragraph and write the first half of the next sentence, to make it easier to pick up.
exercise control what and when you do things. There are always interruptions, or incoming mesages and alerts that could require our attention. There are rarely alerts that must cause us to drop what we’re currently working on. While there are “drop everything” tasks sometimes, most things are fine to come back to in a little while, and most emails are safe to ignore for a couple of hours. It’s fine to quickly add something to a list to come back to later. It’s also fine to be disrupted, but having some control over that is often helpful.
find non-intrusive ways to feel connected. While it should be possible to do some level of multitasking as you work, there are some kind of interruptions that take a lot of attention. When you’re focusing on work, checking your email can be a distraction (say), but it can be hard to totally turn off email while you’re working. Rather than switch to look at my email on some cadence throughout the day, I (effectively,) check my phone far more regularly just to make sure that there’s nothing critical, and can go much longer between looking at my email. The notifications I see are limited, and may messages never trigger alerts. I feel like I know what’s going on, and I don’t get stuck replying to email all day.¹

This is, more or less, what works for me, and I (hope) that there’s something generalizable here, even if we do different kinds of work!

Email is kind of terrible, in a lot of ways: there’s a lot of it, messages come in at all times, people are bad at drafting good subject lines, a large percentage of email messages are just automated notifications, historically you had to “check it” which took time, and drafting responses can take quite a while, given that the convention is for slightly longer messages. I famously opted out of email, basically for years, and gleefully used all the time I wasn’t reading email to get things done. The only way this was viable, was that I’ve always had a script that checks my mail and sends me a notification (as an IM) with the From and Subject line of most important messages, which gives me enough context to actually respond to things that were important (most things aren’t) without needing to actually dedicate time to looking at email. ↩︎

Pave the On and Off Ramps

Wed, 15 Jun 2022 00:00:00 +0000

I participated in a great conversation in the #commonlisp channel on libera (IRC) the other day, during which I found a formulation of a familar argument that felt more clear and more concrete.

The question--which comes up pretty often, realistically--centered on adoption of Common Lisp. CL has some great tools, and a bunch of great libraries (particularly these days,) why don’t we see greater adoption? Its a good question, and maybe 5 year ago I would have said “the libraries and ecosystem are a bit fragmented,” and this was true. It’s less true now--for good reasons!--Quicklisp is just great and there’s a lot of coverage for doing common things.

I think it has to do with the connectivity and support at the edges of a project, an as I think about it, this is probably true of any kind of project.

When you decide to use a new tool or technology you ask yourself three basic questions:

“is this tool (e.g. language) capable of fulfilling my current needs” (for programming languages, this is very often yes,)
“are there tools (libraries) to support my use so I can focus on my core business objectives,” so that you’re not spending the entire time writing serialization libraries and HTTP servers, which is also often the case.
“will I be able to integrate what I’m building now with other tools I use and things I have built in the past.” This isn’t so hard, but it’s a thing that CL (and lots of other projects) struggle with.

In short, you want to be able to build a thing with the confidence that it’s possible to finish, that you’ll be able to focus on the core parts of the product and not get distracted by what should be core library functionality, and finally that the thing you build can play nicely with all the other things you’ve written or already have. Without this third piece, writing a piece of software with such a tool is a bit of a trap.

We can imagine tools that expose data only via quasi-opaque APIs that require special clients or encoding schemes, or that lack drivers for common databases, or integration with other common tools (metrics! RPC!) or runtime environments. This is all very reasonable. For CL this might look like:

great support for gRPC

There’s a grpc library that exists, is being maintained, and has basically all the features you’d want except support for TLS (a moderately big deal for operational reasons,) and async method support (not really a big deal.) It does depend on CFFI, which makes for a potentially awkward compilation story, but that’s a minor quibble.

The point is not gRPC qua gRPC, the point is that gRPC is really prevalent globally and it makes sense to be able to meet developers who have existing gRPC services (or might like to imagine that they would,) and be able to give them confidence that whatever they build (in say CL) will be useable in the future.
compilation that targets WASM

Somewhat unexpectedly (to me, given that I don’t do a lot of web programming,) WebAssembly seems to be the way deploy portable machine code into environments that you don’t have full control over,¹ and while I don’t 100% understand all of it, I think it’s generally a good thing to make it easier to build software that can run in lots of situation.
unequivocally excellent support for JSON (ex)

I remember working on a small project where I thought “ah yes, I’ll just write a little API server in CL that will just output JSON,” and I completely got mired in various comparisons between JSON libraries and interfaces to JSON data. While this is a well understood problem it’s not a very cut and dry problem.

The thing I wanted was to be able to take input in JSON and be able to handle it in CL in a reasonable way: given a stream (or a string, or equivalent) can I turn it into an object in CL (CLOS object? hashmap?)? I’m willing to implement special methods to support it given basic interfaces, but the type conversion between CL types and JSON isn’t always as straight forward as it is in other languages. Similarly with outputting data: is there a good method that will take my object and convert it to a JSON stream or string? There’s always a gulf between what’s possible and what’s easy and ergonomic.

I present these not as a complaint, or even as a call to action to address the specific issues that I raise (though I certianly wouldn’t complain if it were taken as such,) but more as an illustration of technical decision making and the things that make it possible for a team or a project to say yes to a specific technology.

There are lots of examples of technologies succeeding from a large competitive feild mostly on the basis of having great interoperability with existing solutions and tools, even if the core technology was less exciting or innovative. Technology wins on the basis of interoperability and user’s trust, not (exactly) on the basis of features.

I think the one real exception is runtimes that have really good static binaries and support for easy cross-compiling (e.g. Go, maybe Rust.) ↩︎

Finding Alignment

Tue, 14 Jun 2022 00:00:00 +0000

I keep making notes for writing a series of essays about alignment, the management concept, and it’s somewhere in between a blog post and a book, so maybe I’ll make it a series of blog posts. This is the introduction.

Alignment is this kind of abstract thing that happens when you have more than one entity (a person, working group, or team) working on a project (building or doing something). Leaving aside, for a moment, “entity” and “project,” when efforts well aligned in that all of the effort is in persuit of the same goal and collaborators do work in support of each other. When efforts are out of alignment, collaborators can easily undermine eachother or persue work that doesn’t support the larger goal.

Being well aligned sounds pretty great, you may think, “why wouldn’t you always just want to be aligned?” And I think deep down people want to be aligned, but it’s not obvious: as organizations grow and the problems that the organizations address become bigger (and are thus broken down into smaller parts,) it’s easy for part of a team to fall out of alignment with the larger team. It’s also the case that two parts of an organization may have needs or concerns that appear to be at odds with each other which can cause them to fall out of alignment.

Consider building a piece of software, as I often do: you often have a group of people who are building features and fixing bugs (engineers), and another group of people who support and interact with the people who are using the software (e.g. support, sale, or product management depending). The former group wants to build the product and make sure that it works, and the later group wants to get (potential) users using the software. While their goals are aligned in the broad sense, in practice there is often tension either between engineers who want things to be correct and complete before shipping them and product people who want to ship sooner or conversely between engineers who want to ship software early and product people who want to make sure the product actually works before it sees use. In short, while the two teams might be aligned on the larger goal, these teams often struggle to find alignment on narrower issues. The tension between stability and velocity is perennial and teams must work together to find alignment on this (and other issues.)

While teams and collaborators want to be in alignment, there are lots of reasons why a collaborator might fall out of alignment. The first and most common reason is that managers/leaders forget to build alignment: collaborators don’t know what the larger goals are or don’t know how the larger goals connect to the work that they’re doing (or should be doing!) If there’s redundancy in the organization that isn’t addressed', collaborators might end up compeating against eachother or defending their spheres or fifedomes. This is exacerbated if two collaborators or groups have overlapping areas of responsibility. Also, when the businesses falter and leaders don’t have a plan, collaborators can fall out of alignment to protect their own projects and jobs. It’s also the case that collaborators interests change over time, and they may find themselves aligned in general, but not to the part of the project that they’re working on. When identified, particularly, early, there are positive solutions to all these problems.

Alignment, when you have it feels great: the friction of collaboration often falls away because you can work independently while trusting that your collaborators are working toward the same goal. Strong alignment promotes prioritization, so you can be confident that you’re always working on the parts of the problem that are the most important.

Saying “we should strive to be aligned,” is not enough of a solution, and this series of posts that I’m cooking up addresses different angles of alignment: how to build it, how to tell when you’re missing alignment, what alignment looks like between different kinds of collaborators (individuals, teams, groups, companies,) and how alignment interacts with other areas and concepts in organizational infrastructure (responsibility, delegation, trust, planning.)

Stay tuned!

Against Testify

Tue, 19 Apr 2022 00:00:00 +0000

For a long time I’ve used this go library testify, and mostly it’s been pretty great: it provides a bunch of tools that you’d expect in a testing library, in the grand tradition of jUnit/xUnit/etc., and managed to come out on top in a field similar libraries a few years ago. It was (and is, but particularly then) easy to look at the testing package and say “wouldn’t it be nice if there were a bit more higher-level functionality,” but I’ve recently come around to the idea that maybe it’s not worth it.¹ This is a post to collect and expand upon that thought, and also explain why I’m going through some older projects to cut out the dependency.

First, and most importantly, I should say that testify isn’t that bad, and there’s definitely a common way to use the library that’s totally reasonable. My complaint is basically:

The “suite” functionality for managing fixtures is a bit confusing: first it’s really easy to get the capitalization of the Setup/Teardown (TearDown?) functions wrong and have part of your fixture not run, and they’re enough different from “plain tests” to be a bit confusing. Frankly, writing test cases out by hand and using Go’s subtest functionality is more clear anyway.
I’ve never used testify’s mocking functionality, in part because I don’t tend to do much mock-based testing (which I see as a good thing,) and for the cases where I want to use mocks, I tend to prefer either hand written mocks or something like mockery.
While I know “require” means “halt on failure” and “assert” means “continue on error,” and it makes sense now, “assert” in most² other languages means “halt on failure” so this is a bit confusing. Also while there are cases where you do want continue on error semantics for test assertions, (I suppose,) it doesn’t come up that often'
There are a few warts, with the assertions (including requires,) most notably that you can create an “assertion object” that wraps a *testing.T, which is really an anti-pattern, and can cause assertion failures to be reported at the wrong level.
There are a few testify assertions that have some wonky argument structure, notably that Equal wants arguments in expected, actual form but Len wants arguments in object, expected form. I have to look that up every time.
I despise the failure reporting format. I typically run tests in my text editor and then use “jump to failure” point when a test fails, and testify assertions aren’t well formed in the way that basically every other tool are (including the standard library!)³ such that it’s fussy to find a failure when it happens.

The alternative is just to check the errors manually and use t.Fatal and t.Fatalf to halt test execution (and t.Error and t.Errorf for the continue on error case.) So we get code that looks like this: :

// with testify:
require.NoErorr(t, err)

// otherwise:
if err != nil {
     t.Fatal(err)
}

In addition to giving us better reporting, the second case looks like code that is more typical of code that you might write outside of test code, and so gives you a chance to use the production API which can help you detect any awkwardness but also serve as a kind of documentation. Additionally, if you’re not lazy, the failure messages that you pass to Fatal can be quite useful in explaining what’s gone wrong.

Testify is fine and it’s not worth rewriting existing tests to exclude the dependency (except maybe in small libraries) but for new code, give it a shot!

I must also confess that my coworker played some role in this conversion. ↩︎
I’d guess all, but I haven’t done a survey. ↩︎
Ok, the stdlib failures have the problem, where the failures are just attributed to the filename (no path) of the failure, which doesn’t work great in the situation where you have a lot of packages with similarly named files and you’re running tests from the root of the project. ↩︎

emt -- Golang Error Tools

Thu, 14 Apr 2022 00:00:00 +0000

I write a lot of Go code, increasingly so to the point that I don’t really write much code in other languages. This is generally, fine for me, and it means that most of the quirks of the language have just become sort of normal to me. There are still a few things that I find irritating, and I stumbled across some code at work a few weeks ago that was awkwardly aggregating errors from a collection of goroutines and decided to package up some code that I think solves this pretty well. This is an introduction and a story about this code.

But first, let me back up a bit.

The way that go models concurrency is very simple: you start gorountines, but you have to explicitly manage their lifecycle and output. If you want to get errors out of a thread you have to collect them somehow, and there’s no standard library code that does this so there are a million bespoke solutions to this, and while every Go programmer has or will eventually write a channel or some kind of error aggregator to collect errors from a goroutine, it’s a bit dodgy because you have to stop thinking about whatever thing you’re working on to write some thread-safe, non-deadlocking aggregation code, which inevitably means even more goroutines and channels and mutexes or some such.

Years ago, I wrote this type that I called a “catcher” that was really just a slice of errors and a mutex, wrapped up with [Add(error)]{.title-ref} and [Resolve() error]{.title-ref} methods, and a few other convenience methods. You’d pass or access the catcher from different goroutines and never really have to think much about it. You get “continue-on-error” semantics for thread pools, which is generally useful, and you never accidentally deadlock on a channel of errors that you fumbled in some way. This type worked its way into the logging package that I wrote for my previous team and got (and presumably still gets) heavy use.

We added more functionality over time: different output formats, support for error annotation when it came and also the ability to have a catcher annotate incoming errors with a timestamp for long running applications of the type. The ergonomics are pretty good, and it helped the team spend more time implementing core features and thinking about the core problems of the product’s domain and less time thinking about managing errors in goroutines.

When I left my last team, I thought that it’d be good to take a step back from the platform and tools that I’d been working on and with for the past several years, but when I saw some code a while back that implemented its own error handling again something clicked, and I wanted just this thing. '

So I dug out the old type, put it in a new package, dusted off a few cobwebs, improved the test coverage, gave it a cool name, and reworked a few parts to avoid forcing downstream users to pickup unnecessary dependencies. It was a fun project, and I hope you all find it useful!

Check out emt! Tell me what you think!

Rescoping the Engineering Interview

Tue, 12 Apr 2022 00:00:00 +0000

It’s not a super controversial to assert that the software engineering interview process is broken, but I think it’s worthwhile to do that anyway. The software engineering interview is broken. There are lots of reasons for this:

interview processes are overoptimized for rejecting candidates that aren’t good, that they often reject candidates that are good. This isn’t a problem if it happens occasionally, but it’s really routine.
it’s difficult to design an interview process that’s works consistently well across different levels and different kinds of roles, and companies/teams can easily get into a place where they really can only hire one type or level of engineer.
while many engineering teams know that the hiring process is biased, most of the attempts to mitigate this focus on the bias of the interviewer by making interview processes more consistent across candidate or easier to score objectively, while abdicating for the ways that the process can be biased toward certain kinds of candidates.

I’ve been part of lots of conversations over the years about “making interviews better,” and many of the improvements to the process that come out of these conversations don’t do much and sometimes exacerbate the biases and inefficiencies of the process. I think also, that the move toward remote work (and remote interviewing,) has presented an underrealized opportunity to revisit some of these questions and hopefully come up with better ways of interviewing and building teams.

To unwind a bit, the goals of an interview process should be:

have a conversation with a candidate to ensure that you can communicate well with them (and them with you!) and can imagine that they’ll fit into your team or desired role.
identify skills and interests, based on practical exercises, review of their past work (e.g. portfolio or open source work,) that would complement your team’s needs. Sometimes takes “figure out if the person can actually write code,” but there are lots of ways to demonstrate and assess skills.
learn about the candidates interests and past projects to figure out if there’s alignment between the candidate’s career trajectory and the team you’re building.

Most processes focus on the skills aspect and don’t focus enough on the other aspects. Additionally, there are a bunch of common skills assessments that lots of companies use (and copy from eachother!) and most of them are actually really bad. For example:

live coding exercises often have to be really contrived in order to fit within an hour interview, and tend to favor algorithims problems that folks either have memorized because they recently took a class or crammed for interviews. As engineers we almost never write code like this, and the right answer to most of these problems is “use a library function”, so while live coding is great for getting the opportunity to watch a candidate think/work on a problem, success or failure aren’t necessarily indicative of capability or fit.
take home coding problems provide a good alternative to live coding exercises, but can be a big imposition timewise particularly people on people who have jobs while interviewing. Often take home exercises also require people to focus more on buildsystems and project-level polish rather than the kind of coding that they’re likely to do more of. The impulse with take home problems is to make them “bigger,” and while these problems can be a little “bigger” than an hour, a lot of what you end up looking at with these problems is also finishing touches so keeping it shorter is also a good plan.
portfolio-style reviews (e.g. of open source contributions or public projects,) can be great in many situations, particularly when paired with some kind of session where the candidate can provide context, but lots of great programmers don’t have these kinds of portfolios because they don’t program for fun (which is totally fine!) or because their previous jobs don’t have much open source code. It can also be difficult to assess a candidate in situations where these work samples are old, or are in codebases with awkward conventions or requirements.

There isn’t one solution to this, but:

your goal is to give candidates the opportunity to demonstrate their competencies and impress you. Have an interview menu¹ rather than an interview process, and let candidates select the kind of problem that they think will be best for them. This is particularly true for more senior candidates, but I think works across the experience spectrum.
if you want to do a programming or technical problem in real time, there are lots of different kinds of great exercises, so avoid having another candidate implement bubble sort, graph search, or reverse a linked list. Things like:
- find a class (or collection of types/functions) in your codebase that you can share and have the candidate read it and try and understand/explain how it works, and then offer up suggestions for how to improve it in some way. I find this works best with 100/200 lines of code, and as long as you can explain idioms and syntax to them, it doesn’t matter if they know the language. Reading code you don’t know '
- provide the candidate with a function that doesn’t have side effects, but is of moderate length and have them write tests for all the edge cases. It’s ok if the function has a bug that can be uncovered in the course of writing tests, but this isn’t particularly important.
- provide the candidate with a set of stubs and a complete test suite and have them implement the interface that matches the test cases. This works well for problems where the class in question should implement a fairly pedestrian kind of functionality like “a hash map with versioned values for keys,” or “implement an collection/cache that expires items on an LRU basis.”
- have the candidate do a code review of a meaningful change. This is an opportunity to see what it’s like to work with them, to give them a view into your process (and code!), and most importantly ask questions, which can provide a lot of insight into their mindset and method.
I think that the menu approach also works well here: different people have different skills and different ways of framing them, and there’s no real harm in giving people a choice here.
independent/take home/asynchronous exercises tend to be better (particularly for more senior candidates,) as it more accurately mirrors the way that we, as programmers work. At the same time, it’s really easy to give people problems that are too big or too complex or just take too long to solve well. You can almost always get the same kind of signal by doing smaller problems anyway. I also believe that offering candidates some kind of honoraria for interview processes are generally a good practice.
one of the big goals of the interview processes is to introduce a candidate to the team and give them a sense for who’s on the team and how they operate, which I think has given rise to increasingly long interview sequences. Consider pairing up interviewers for some or all of your interview panel to give candidates greater exposure to the team without taking a huge amount of their time. This is also a great way to help your team build skills at interviewing.
assessing candidates should also balance the candidates skills, alignment with the team, with the team’s needs and capacity for ramping new members. Particularly for organizations that place candidates on teams late in the process, it’s easy to effectively have two processes (which just takes a while,) and end up with “good” candidates that are just haphazardly allocated to teams that aren’t a good fit.

These are hard problems, and I think its important both to be open to different ways of interviewing and reflecting on the process over time. One of the great risks is that a team will develop an interview process and then keep using it even if it turns out that the process becomes less effective as interviewers and needs change. Have (quick) retrospectives about your interview process to help make sure that stays fresh and effective.'

I think this is a follow up, in someways, to my earlier post on Staff Engineering. If you liked this, check that out!

To be clear, I think the interview menu has to be tailored to candidates and roles. There’s a danger of decision paralysis, so recruiters and hiring managers should definitely use part of their time with the candidate to select a good interview plan. The options need to make sense for the role, the interviewers need to prepare, and the hiring manager/recruiter should be able to eliminate options from the menu that don’t make sense for the candidates background. ↩︎

Doubled Hat Pattern

Tue, 08 Mar 2022 00:00:00 +0000

Last year I wrote a draft of a book about knitting that I’m working on revising and also drafting something of a sequal to. The book contains a discussion of some fundamental techniques but mostly describes the process for knitting a collection of projects, mostly sweaters, but a few other things as well. The chapters exist somewhere between an unconventional pattern and a long form account of the design and construction process of several specific garments, though I hope there’s a sort of companionable air about it, even if the details end up being mostly technical.

In any case, this post is an attempt at the same form, more or less, but focused on a hat that I recently completed.

I’m going to be knitting a hat with a sort of unconventional empirical construction. There’s not a lot of preparation work that you need: no gauge, no sizing information, no counting stitches (unless you want,) just knitting and figuring it out as you go along. The hat itself is a simple beanie-style knitted cap, with a “lining” for extra warmth and potentially comfort.

Cast on 16 stitches. Your gauge probably doesn’t matter, within reason. I chose a fingering weight wool on the heavier side of fingering, and US size 0 needles. 16 stitches is about an inch and a half or two inches: from these stitches you’ll knit a strip of fabric that will encircle your head, so better to keep it narrower than 3 or 4 inches at the outside. I cast on using the long tail method, and I made sure that there was a generous tail left over afterwards as I intended to take advantage of this tail.

Knit, in garter stitch, until the strip is long enough to fit around your head.

I, for my part, made the strip 21 inches or so, around. My head is (unfortunately) 24 inches around, and I think if or when I do it again, I’d make it shorter: maybe 19 or 20 inches around. You can figure out the length empirically, buy placing the knitting around your head and seeing what fits. It’s okay to stretch the band a bit for a closer fit, but because there’s going to be another layer of knitting on the inside of the hat, it’s even expected that the hat will be a little bit big at this point.

When the strip is large enough, bind off, but do not break the yarn. You should have the tail from the cast on be on the same side of the work as the end of the working yarn from where you cast off.

With the same working yarn that you just bound off with, pick up stitches, knitwise, along the side of the strip, creating one sititch in every garter “ridge.” When you get all the way around the strip, join and knit in the round. Knit about an inch plain, and then begin shaping the crown.

I do this weird crown shaping that I wrote about here 15 years ago (!!) that I adapted from the toe shaping of a sock. I think it works better for hats than socks, and is great when you don’t want to figure out how to evenly divide into 4 or 5 “spokes” and have a spiral decrease. Convienetly, it also structures the decreases so that you switch to double points relatively late in the process. It does something like: repeat “knit eight stitches, decrease once (e.g. knit two together),” all the way around a single round, and then knit 8 rows plain. Then replace 8 with 7: knit 7 stitches and decrease, repeating around, then knit 7 plain rows. Continue on in this manner, moving the decreases closer together in the decrease rounds, and moving the decrease rounds closer together. Eventually, all your stitches will be decreases, and you can just alternate “decrease and plain” rows until you have 8 stitches or something, and then graft the remaining stitches together. I definitely always have the feeling of totally winging the ending: worry not.

Once you take have taken care of the crown stitches, I break the yarn and weave in this end. Turning my attention back to the long tail, I sew up the cast on and bind off ends of the original strip, and have the tail ready and the lower edge of the hat. With this yarn I fuse in the remaining working yarn using a felted or sewn join, and pick up stitches along the remaining garter edge, again at a rate of one stitch for every garter ridge, all the way around.

Knit about an inch here, until the hat is your desired length: I like to have 4 or 5 inches between the lower edge of the hat and the start of the crown shaping, but this is a point of personal preference. When the hat is the proper length, purl the next row to provide a turning round, and then stop. It’s important at this point to make some decisions about the lining of the hat:

if you plan to knit the interior hat with the same color and yarn as the exterior hat, purl a second row and continue.
if you want to switch colors, knit the next row with the new color, and then purl the following row in the new color, before continuing.
if you want to switch yarns to a different weight, be careful, but proceed as if you were changing colors (even if you’re not!) and do increases or decreases as required so that the interior hat is either the same or slightly smaller than the exterior hat.
if you aren’t changing yarn, or are changing between two colors of the same yarn, then you could omit all purl rounds, and just knit plain.

For my part, I switched colors and to a different yarn type with a substantially finer gauge, and increased rather a lot at this point. I think I probably increased a bit too much, though the hat still works fine. I think I’d probably tend to keep things more simple in the future.

Finally, knit the interior hat straight away until the distance between the lower edge (purl round(s)) and the beginning of the shaping row are the same, and then repeat the shaping for the interior hat, and finish it off. Fold the inner at into the outer hat and place on head.

Observations:

the hat will be quite warm, so knitting with finer yarn is probably better. Also because the hat is so heavy, it’s viable to knit a bit loser than you might if it were single weight.
making sure that the inner hat’s total length from the brim to the crown is the same as the interior measurement of the outer hat can be a bit tricky, but getting it right avoids flaring in either direction. Avoiding the purl/turning round entirely gives you a bit of wiggle room, if you like.
this is a weird hat: while the hat looks great while on my head, it doesn’t quite lay flat. While I could have re-knit the crown to have a less aggressive decrease sequence (e.g. start with k9 k2tog, etc.) over more rows, I kind of like the flatter top look. Hats are super forgiving, and hats don’t really need to lie flat anyway because heads are three dimensional.'

Current Work

Mon, 07 Mar 2022 00:00:00 +0000

I know it seems like I write a lot about knitting, and it is the case that knitting covers a lot of the “stuff I do” it’s certainly not the only thing I’m doing, and I thought it’d be fun to quickly review a bunch of things:

As of this week, I’ve been working at Interchain GmbH on Tendermint Core which is a consensus engine for state machine replication. After spending a huge part of my career on projects that were either “enterprise technology” (e.g. writing documentation for database engines), or technical operations (e.g. systems administration), or mostly internal facing (e.g. developer tools,) it’s been really interesting to work on something that is definitely core product engineering with a great team. My work has mostly focused on what I think of as “platform concerns:” service construction, networking, workload management, and test architecture: these are the things I really enjoy, so that’s been great.
More recently I’ve begun reballancing my time work to spend some time (intentionally) on what I think of as “engineering issues” rather than “software issues.” I’m still writing basically the same amount of code as I ever did, but I’m also thinking about how to support teams as they grow and function. At basically every organization and team that I’ve worked in, the main constraining factor in shipping features has always been coordination with other engineering projects and not really “how quickly can I write code” (I’m pretty fast, all told.) I’ve always thought that challenges of how people coordinate their labor and organize their efforts in distributed (conceptually, temporally, geographically) environments, is one of the cool/hard problems.
I’ve been cooking a lot more. I’m getting better at making simple dishes that last for a few meals that I enjoy eating. I’ve been really getting into bean and letils, and have also made some good lentils making a lot of white bean and sausage dishes, lentils, pasta sauces, roast veggies.
I’m knitting a lot! Most of the time I have 2 or 3 projects going: a sweater, a pair of socks that is easily portable for travel or times when I’m not at home, and more recently a series of plain white socks. I’m quite enjoying all of this. As a backdrop to knitting, I’ve been watching Poirot recently, which has been fun.
I have a couple of “writing about knitting” projects that I’m writing and preparing drafts of. These are mostly book-length (though on the short side,) type projects, and one needs more editing (which I’m hoping to hire someone to help with,) and one is roughly half way through a first draft. The idea is to provide a lot of technical depth about the craft of knitting--techniques, skills, and design--combined with discussions of projects (mostly from a process perspective,) with some personal reflections and anecdotes sprinkled in. It’s been a fun exercise, both because writing about things you understand well is fun, but also because (as weird as this sounds) it’s been nice to sort of explore the boundary between technical writing and more creative writing.
I’ve been doing a bit more things that I think of as “general personal care/growth:” reading more books just for fun and because reading is good for inspiration generally; doing duolingo every day (Russian, which I studied as a kid in school); upgrading a bunch of my personal computing practices (new laptop, better remote editing environments, staying on top of my email, switching to tmux, etc). I definitely go in cycles of paying greater and less attention to all of these sorts of things, but I think it’s worth while to dedicate time and attention to these kinds of things.

I want to find more ways of writing little things quickly. There’s that old quip “sorry for writing a 10 page letter, I didn’t have enough time to write a one page letter,” but also I think that I do most of my writing in the morning and tend to not do this on days when I’m working, though this seems like a tractable thing to reorganize and think through ways of doing more writing (and other projects!) throughout the week.

Emacs Stability

Thu, 03 Mar 2022 00:00:00 +0000

A while ago I packaged up my emacs configuration for the world to see/use and I’m pretty proud of this thing: it works well out of the box, it’s super minimal and speedy, and has all of the features. I don’t think it’s the right solution for everyone, but I think there are a case of users for whom this configuration makes sense. I’ve definitely also benefited a lot for thinking about this “configuration” as a software project at least in terms of keeping things organized and polished and reasonably well tested. It’s a good exercise.

Historically, I’ve used my emacs configuration ans as a sort of “fun side project” and while I tried to avoid spending too much time tweaking various things, it did feel like the kind of thing that was valuable (given how much time I spend in a text editor,) without being too distracting. Particularly, early in the pandemic, or during periods over the summer when I was between jobs.

Then, I put the configuration in a public repo, and I basically haven’t made any meaningful changes since then. One part of this is clearly that I put a lot of time into polishing things in the initial push to get it released, and there haven’t been many bugs that have inspried any kind of major development effort. Another part is that, the way I use an editor isn’t really changing. I’m writing code and English and using a couple of applications (e.g. email and org-mode) within emacs, but I’m not really (often) adding new or different kinds of work, and while this isn’t exacting from a blogging perspective* it is exciting from a “things just work perspective.”

I have refered to myself as a degenerate emacs user. I’ve sometimes said unrepentant, but I think it’s basically the same. I’ve also realized that, given that I’ve basically been using emacs the same way since 2008 or so, I’m kind of an old timer, even if it doesn’t much feel like that, and there are lots of folks with longer histories.

I think I used care more about what tools other people used to edit text, even a couple of years ago, I thought that having good initial configuration and better out of the box experiences for emacs would lead to more people using emacs, which would be cool because they’d get to use a cool piece of software and we’d get more emacs users.

Increasingly, however, while I think emacs is great and people should use it, I’m less concerned: people should use what they want, and I think there will always be a enough people here and there who want to use emacs and that’s good enough for me. I think having good out of the box experiences are important, but it’s not a one-size fits all kind of situation. I also think that VS Code is pretty great software, and I like a lot of the implications for remote editing, even if I’m not particularly interested in it for myself.

Enjoy the repo, and let me know if there’s anything terrible about it. I’ve been getting back into blogging recently, and have started tweaking a few things about the ways I use computers/emacs, mostly in terms of exploring tmux (hah!) and also considering avoiding GUI emacs entirely. Stay tuned if you’re interested!

The Emacs Daemon GTK Bug, A Parable

Tue, 01 Mar 2022 00:00:00 +0000

There’s this relatively minor Emacs bug that I’ve been aware of for a long time, years. The basic drift is that on Linux systems, when running with GTK/Emacs as a daemon, and the X11 session terminates for any reason the Emacs daemon terminates. Emacs daemons are great: you start Emacs once, and it keeps running independently of what ever windows you have open. You can leave files open in Emacs buffers and not have move between different projects with minimal context switching costs.

First of all, emacs’s daemon mode is weird. I can’t think of another application that starts as a daemon (in the conventional UNIX double-forking manner,) and then a client process runs and spawns GUI (potentially) windows. If there are other applications that work this way, there aren’t many.

Nevertheless, being able to restart the window manager without loosing the current state of your Emacs session is one of the chief reasons to run Emacs in daemon mode, so this bug has always been a bit irksome. Also since it’s real, and for sure a thing, why has it taken so long to address? Lets dig a little bit deeper.

There are two GNOME bugs related to this:

What’s happening isn’t interesting or complicated: Emacs calls an API, which behaves differently than Emacs expects and needs, but not (particularly) differently than GNOME expects or needs. Which means GNOME has little incentive to fix the bug--if they even could without breaking other users of this API.

Emacs can’t fix the problem on their own, without writing a big hack around GNOME components, which wouldn’t be particularly desirable or viable, and because this works fine with the other toolkit (and is only possible in some situations,) it doesn’t feel like an Emacs bug.

We have something of a stalemate. Both party thinks the other is at fault. No one is particularly incentivized to fix the problem from their own code, and there is a work around,¹ albeit a kind of gnarly one.

This kind of issue feels, if not common, incredibly easy for a project--even one like emacs--to stumble into and quite easy to just never resolve. This kind of thing happens, in some form, very often and boundaries between libraries make it even more likely.

On the positive side, It does seem like there’s recent progress on the issue, so it probably won’t be another 10 years before it gets fixed, but who knows.

To avoid this problem either: don’t use GUI emacs windows and just use the terminal (fairly common, and more tractable as terminal emulators have improved a bunch in the past few years,) or use the Lucid GUI toolkit, which doesn’t depend on GTK at all. The lucid build is ugly (as the widgets don’t interact with GTK settings,) but its light weight and doesn’t suffer the ' ↩︎

Tips for Casting On a Sweater

Mon, 28 Feb 2022 00:00:00 +0000

Having recently started knitting a new sweater I realized that there are a lot of little things that I do, that are worth collecting in one place:

Do not tie a slip not to start, simply twist the yarn around the needle as a basis for casting on the first stitch. This twist looks like a stitch, but isn’t, you should decrease it at the end of the row, with the last stitch to complete the join.
Always used the German Twisted long tail cast on variant, which makes things a bit more elastic and just looks great, particularly when knitting ribbing, which I often do at the beginning of a sweater.
Wrap the yarn around the needle once per number of stitches that you need to cast on to estimate the length of the long tail that you’ll need to cast on. I find this overestimates a bit, but I’ve rarely regretted having too-long of a tail rather than having too short. While you can start again from the beginning in the case that you run out of tail, you can also splice in a second yarn.
If you do run out of yarn while casting on for the sweater, and you’ve been using the long tail for the finger yarn (loops around the needle,) you can sometimes get a few extra stitches out of switching to having that needle
Place markers periodically to make it easier to count, roughly every 20 stitches or so, and I try and make sure that one of the markers gets placed half way through the round. For example, to cast on 228 stitches, I placed 12 markers every 19 stitches, and the 6th marker was the “half way” point.

I did one sweater where I put markers every 32 stitches and one sweater where I put markers every 16, and found that I spent far more time casting on the one with fewer markers because I had to double check my counts more. They really help.
Cast on to a needle that’s a bit bigger than the size you intend to use. I’ve been quite happy using a US 2.5 to cast on for a US 0 sweater. I’ve been using interchangeable needles, and being able to replace the larger needle for the smaller needle before beginning to knit has made things much easier to knit for the first row. It’s also an option to hold two needles together for the cast on.

I also have to think about not pulling on the “thumb yarn” at all, as this will also make things tigheer.
While it’s good to be careful to avoid twisting the first row, if you do accidentally twist, undo the twist between the last and first stitch, which will hardly be noticeable.

Cool Things in the Alliance Union Stories

Fri, 25 Feb 2022 00:00:00 +0000

I’ve been working my way through C.J. Cherryh’s Alliance-Union stories for a few years: in part because I’ve not been reading as much, in part because some of these stories are a bit hard to get into and only available in paper books, but mostly because I’ve been hording them. These are incredibly me-type stories, and I’ve found them really inspiring. I’ve ‘wanted to collect a number of things about these stories and world that I’m quite excited by:

I really like that it’s all so concrete: Cherryh’s talked about designing the stories by taking a star chart and drawing out the “map” on stars that are near to Earth. This makes the setting feel both really big, because stars are far apart and also quite small, because there are really a small selection of planets and they’re pretty close in relative terms, so it’s
I enjoy the ways that it’s a product of its historic moment. The medium of information storage is “tape”, and I think if someone was writing this today, there’d be more planets, say. Similarly, While there are computers, the way characters interact with technology is much less ubiquitous and networked.
The scarcity of planets, and the cost of getting out of a gravity well creates these interesting economic effects which drive a lot of tension in the stories. I enjoy that Earth exists, but that most of the people on Earth are very disconnected from what happens in the world of the stores (either by distance before FTL, or by isolationism.)
While there is FTL travel, it doesn’t 100% throw relativistic effects out of the window, so “ship time” and “station/planet” time (and lives!) move at different paces, and the stories explore the impact of this on culture/society/economics/lives. FTL is also hard on the people who travel this way, so while many of the characters we interact with are spacers it’s clear that they’re a minority. Also, there aren’t ways for messages to pass faster than ships, which has the effect of making the world seem small at the same time.
These elements of the setting (distance between stars, limited goods from planets, cost of travel, etc) feed back into the imagined economics of the society, with lots of interesting thoughts about taxation policy, and what we’d call “local economies/currency” these days.

Anyway, I read a few dozen pages of the last of the “mainline” Alliance-Union book in my queue (Tripoint). I’m thinking about going back and re-reading Downbelow Station (the first I read) and Cyteen (because I love it), but also am going to enjoy reading other things for a while.

Opus Knitting

Tue, 22 Feb 2022 00:00:00 +0000

My knitting projects, recently (and currently!) have been big not just “knit sweaters out of fingering weight yarn on US 0s, which I’m certainly doing, but big on another scale. I’ve written about this kind of project as epic knitting, but I’ve really gone down the rabbit hole on this one. I’ve started thinking about knitting projects less bounded by a single object or garment, and more as a “meta project,” here are the two examples:

All socks are hand knit. Right now I mostly wear machine knit socks. I have about 10 or 12 pairs of wool socks, and I just wear them every day. The thing I like about my current socks is that they’re comfortable, easy to care for and very easy to match up when I’m folding laundry. I also have three weights of socks, and the heavy and medium socks are definitely good even in warmer months.

To avoid needing to buy new socks when the current batch wear out, I probably need about as many socks (10-12, maybe a few more to cover light weight cases,) and a slightly longer laundry cycle if I want to segregate sock laundry. But the project isn’t exactly about “just enough” socks, but also about having socks that are mostly the same, not just in terms of pattern, but also in terms of yarn content.

In persuit of this, last week I sat down with a 1.5 kilo cone of sock yarn (merino, bamboo, nylon) and cast on for a sock, and I’m already on the second sock, of what I expect will be many. Gotta figure out some situation for dying them all the same color.
I’d kind of like to explore some more corners of the seamless yoke sweater paradigm. I’ve made, I think, 2 raglan sweaters ever, and a few of the “set in sleeve” type and that’s it! I don’t know that I’ve ever made saddle shoulders! I’ve also gotten into the habit of wearing sweaters more often these days, I think I’d like to do something more comprehensive along these lines. Having really nailed down a sleeve, as well as a way to shape the body of a sweater, it seems fun to explore different shoulder shapes for variety. The truth of the matter is that I really like knitting plain sweaters, so this gives some exciting opportunity.

There are other kinds of sweaters that I’d like to get better at: cardigans, mostly, and also I think I’d enjoy exploring hems that aren’t ribbing, and also finding ways of knitting crew necks that I’d enjoy knitting.

In persuit of this, as alluded above, I’ve acquired a bunch of un-dyed yarn, in interesting fibers to knit “a few” sets of socks (maybe some gifts!) and to really get some practice in on these different sweater types. I’m definitely hunkering down into knitting at this scale and thinking about the ways that this broader

Even if it makes for somewhat less exciting blogging about knitting.

“Ah, yup, just knit another white sock.”

Anyway! I hope, if you’re knitting, you’re enjoying it as much as I am.

The Perfect Sleeve

Mon, 21 Feb 2022 00:00:00 +0000

I know I’ve been bad at blogging regularly, and it may be obvious from reading things here or talking to me that I’ve been knitting a lot. What is less obvious is that I’ve been writing a lot about knitting.

While I’d like to do some meta posting about these projects, instead I’m going to just jump in with a bit about the way I knitted the sleeve of the last sweater I made that I’m already duplicating because it was just that good.

I’m not a huge fan of well fitted sleeves on sweaters: I like to wear layers, and having a sweater sleeve hug my wrist often just means that I won’t wear the sweater: it’s good to get the sleeve right, in this context. At the same time, too bulky and the sweater looks goofy (to my sensibilities) and the sleeves take forever to knit. I’ve solved this problem, often, by knitting sleeves from the shoulder down: this way I can try things on as I go and easily re-knit the last few inches of the sleeve if things feel off.

The shoulder-down method works, but it’s bulky to carry the entire sweater around to knit a relative small piece of fabric, and it precludes knitting single-piece seamless sweaters, as you have to get too clever to make it all work.

Here’s what I’ve come to:

Start by casting on as you would for a sock: for me this means fingering weight wool with 64 stitches, US 0s (2.00mm) on 4 double pointed needles in knit 2 purl 2 ribbing. There should be 16 stitches on each needle, and knit for 2 inches.

This has several advantages not the least of which is you can chicken out after 2 inches and just knit a pair of socks. Your wrists and ankles (or calves) might be different than mine, and while my wrists are slimmer than my calves, ribbing is elastic, and I know that it won’t be too small. I know Elizabeth says to figure out the chest circumference and derive the sleeve from that, and you could do that, but chest circumferences vary more than wrists.

In the next row switch to stocking stitch, and increase to 72 stitches, or 18 stitches per needle. I use a raised bar-type increase between the knit and purl stitches on the first and third ribs.

Increasing a bit after the ribbing helps achieve both “the cuff pulls in and isn’t floppy” which is desireable, and also “sleeve isn’t too tight”. You could achieve this by using smaller needles, but I like to avoid needles smaller than US 0s whenever possible.

For sizing the rest of the sleeve, I believe that the “top” of the sleeve should be as wide as my expected yoke depth. To unpack: if the distance between the shoulder and the underarm should be 9 inches, then the sleeve should be 18 inches around. Given a gauge of 8 stitches to the inch, this means 144 stitches, and given paired increases in sleeves, 36 pairs of increases.

I do increases every 5 rounds. This means, knit five rounds, do an increase, knit five rounds. This means that there are actually 6 total rounds in any increase repeat. The increase are paired on either side of four stitches.

“Increase every 5th row” is a common knitting instruction and it’s pretty ambiguous. Do you mean “increase on the fifth row,” or “increase after the fifth row”? In computer programming we call this “a fencepost counting problem,” and over the course of 36 repeats (as is the case )

In the past I used to pair increases around an odd number of stitches so that there’d be a single “seam” stitch, but as the sleeve has an even number of stitches, and most sweater bodies have even numbers of stitches and I haven’t been particularly keen on EZ style pseudo seams, particularly since putting the increases 4 stitches away creates a clear “seam”-like effect. It definitely works.

The target size (144 stitches, or increasing 36 times for a total of 72 stitches,) means that the sleeve doubles in diameter between the end of the cuff and its final sleeve. My arms are a longer than average (particularly for my height, but that’s no matter,) but given everything, it means that after doing these increases I only need to knit 3/4s of an inch before I’m done with the sleeve. Perfect. In my estimation if you can space your increases evenly across the sleeve the result is pretty enjoyable. Having a sleeve that’s slightly longer than your arm increases the blousing effect, which by a little isn’t always a problem. If you need a longer sleeve, putting more plain knitting at the top is good too, and the increases can always be closer together at the beginning of the sleeve if needed. Having an idea of your row gauge can help figure out the math.

There you have it!

While I was writing this up, I managed to knit both sleeves for another sweater, but I have yet to figure out how I want the lower hem of the body of this sweater to go, so I think I’ll knit some socks in the mean time.

Iterative Knitting

Thu, 10 Feb 2022 00:00:00 +0000

I know I’ve been bad a blogger regularly, and it may be obvious from reading things here or talking to me that I’ve been knitting a lot. What is less obvious is that I’ve been writing a lot about knitting. While I’d like to do some meta posting about these projects, instead I’m going to just jump in with a bit.

I was getting my blood drawn last week--just routine--and the phlebotomist said “how are you doing with all the snow?” it had snowed almost five days ago, and while most of it had melted it was the first real snow of the year.

Most people didn’t spend 3 years in Wisconsin, I guess. It didn’t really register as that much snow.

We must rewind, because I didn’t actually hear her correctly the first time and though she said “how are you doing with all this,” and I just assumed that the “all this” was the omircon spike of the coronavirus pandemic in New York City.

I clarified, just because I’m not used to small talk about such enduring existential issues. After we chuckled, I said something like “oh, not much… Just knitting mostly.”

She was intensely interested, she asked what I was knitting (sweaters) and she (a crocheter it turns out,) was interested in learning to knit. I did the only thing that was reasonable and said “there are a lot of good videos on youtube, and also there’s this book called Knitting Without Tears by Elizbeth Zimmerman and you should read it, it’s great.” Good deed done.

In the course of this conversation she said “oh how many sweaters have you made?” I could tell that she was like “oh a couple,” but the truth is somewhat alarming, and I also don’t remember exactly.

“30 or 40, I think.” This has the potential to be correct, though I’m not sure.

“Oh.”

“Yeah, I dunno, I’ve been doing this for a while.”

Also true. Lots of these sweaters were total rubbish, and since I knit my first sweater in 2003 or (or something,) and while I managed to make a number of sweaters that were really great, I also made a lot of sweaters that were really terrible in one way or another. The process of making sweaters better, in my experience is to knit one, and see what works and what doesn’t work and then fix it in the next go round. While having nice sweaters to wear is a good side effect of knitting, the process is the important part.

This is one of the reasons why I buy yarn in larger lots (often by the kilo or more, so I can knit a few sweaters) and while I usually just cast on a for a new sweater (or pair of socks) as soon as I finish the preceding one. While this might be an extreme implementation of this idea, the general approach is perhaps more applicable.

I started writing this post a few days ago as a kind of introduction to what I hoped was going to be some kind of pithy instructions for knitting a sleeve, but I decided to split it out after it was clear that I’d gotten a bit carried away.

Then, last night, I came to a realization about a different sweater that I’ve been knitting for a couple of weeks. Basically I wanted to take the basic pattern for a sweater that I’ve knit a bunch of times before and update it to be more like the sweaters I’ve been knitting more recently: slim sizing, vnecks, set in sleeves for the shoulders.

I don’t think it worked out: the sweater is a touch on the small side (but not unworkable), knitting the sleeve caps with short rows in two color patterns is frightfully difficult (and likely to look really messy in a part of the sweater where the eye is drawn to.)

I’m also not wild about the v-neck (as implemented) because I didn’t account for the fact that in two-color knitting the stitches are more narrow (and therefore the row gauge is equal to the stitch gauge) and as a result the angle of the v-neck is shallower than I wanted.

Finally, and while this is tractable, I’m grumpy about the fact that I made the steeks a bit narrower than I should have and the’ve all been threatening to unravel in an unexpected way: there’s some hubris here, as I did the “just cut, don’t worry” method of steek preparation.

They don’t all work out. ‘I’ve put the sweater in a bag and I think I’m going to let it sit for a little while before I really cut my losses, but I think I’d rather knit something I’m far more excited about than slog through a sweater I’m unlikely to wear (and would be unable to give away given the sizing.)

To my mind, it’s just as important to celebrate, learn from, and write about the sweaters that didn’t turn out as it is the ones that did.

Plain Knitting Ahead

Tue, 08 Feb 2022 00:00:00 +0000

As I said in Knitting Catchup, I’ve been knitting a bunch recently--a few sweaters, and a bunch of socks--and I found myself writing a list of sweaters that I was thinking about knitting, and thought it’d be interesting to share. But first, some background…

I’ve begun a writing project that explores, in great detail, how to knit a very basic sock. I’ve knit socks here and there forever, but I think in the past year knitting lots of socks has become a thing and I think one of the most thrilling aspects of knitting is the way that repetition (at many levels,) gives you the space to develop a deep understanding of your project and I think this gives rise to craft. I definitely want to knit a bunch more socks, and also I’ve been toying more with the idea of knitting sets of socks, maybe on the order of 5 or 10 pairs (for myself and friends) so I can see what it’d be like to only wear hand knit socks.

The last sweater I finished was a bottom-up-raglan seamless number with a v-neck, and the whole process was a lot of fun and I definitely want to explore some variants on this. I’ve also been working on a colorwork sweater for a couple of weeks, and while I’ve enjoyed that a lot, I think that I’d like to mostly stick with a few more plain sweaters, to explore different shapes and also be able to have a collection of sweaters that I can wear most days during the cooler months.

Pursuant to both of these projects I’ve recently begun buying yarn in bulk, mostly un-dyed (either to stay natural or to be garment dyed at some point.) This affords reasonable and predictable options for getting really soft yarn. The first batch appeared over the weekend, so I’m itching to cast on a new sleeve!

Here’s the list of sweaters that I’m interested in knitting:

I’d like to explore knitting more bottom up yoke sweaters, I’ve never made the hybrid yoke sweater, and while i’ve done the set-in-sleeve option a few times, I haven’t done the saddle-shouldered sweater yet.
I’d like to give crew necks another go. Particularly with some of these silk blends, I think a nice crew neck could be fun to explore, both to see if I can tolerate wearing them. I’ve been doing a lot of collars with ribbing recently, and I think a crew neck with a rolled edge might be nice.
While I sometimes find lots of garter stitch to be too densen and sometimes awkward, a little garter stitch edge here and there can be really fetching, and I’d like to explore what it’d be like to put more garter stitch in places, like as a v-neck collar or maybe even the lower hem of a garment. I think, for me the challenge is mostly in finding a way so that he extra depth of the fabric doesn’t feel like it’s flaring out.
I’ve thought about doing a sweater that was entirely made out of 2 by 2 ribbing. This might be a terrible idea.
I made a sweater last year that had garter rib on the yoke, but was plane otherwise, and I really enjoyed it, though it has some flaws (Henley neck that doesn’t quite work, sleeves were a bit wrong,) and I’d like to play around with this idea, both with “garter rib for the yoke” but also more garter rib (in general,) including maybe a garter rib cardigan.

I’m excited. Looking back at this, I think I need to also pull together a list of more concrete pattern ideas rather than just a list of variables, but there’s time! And I definitely can cast on a sleeve for fun!

Knitting Catchup

Mon, 07 Feb 2022 00:00:00 +0000

I haven’t been writing or blogging much in the past few months, but I definitely have been knitting a bunch. Things have been busy and overwhelming, in totally unexceptional ways, and my knitting has been a near constant companion in providing balance and focus in various ways.

I embarked in the late spring in a kind of epic project: I bought a kilo of a lovely gray fingering weight Silk/Yak/Merino blend, and promptly started a cabled sweater based on Alice Starmore’s Na Cragga sweater (the cover of Aran Knitting). I changed a lot: the gauge (ran weight to fingering) and the construction (knit flat with pieces to in the round) and the shape (tapered body, short rows across the back, set-in sleeves and saddles,) and monkeyed with the pattern (added two repeats of the center panel, dropped one of the patterns, extended the ribbing pattern into the body of the sweater,) and made a v-neck. I kind of stalled out on the project, at roughly the underarms, in June and took a few months off of regular knitting.

In the late summer, as delta began to spike here, I started knitting socks again. I’ve always loved socks, and have gotten really into knitting really plain cuff-down socks with medium length ribbed cuffs. I’ve knit maybe 20 pairs of these socks, and while I’ve taken a bit of a sock-knitting break in the last couple of months--I guess during the depths of winter and the omicron spike--I definitely hope to knit a bunch more socks.

In the mean time I’ve been working on two sweaters:

a second sweater that I just finished knit out of the second half of the aforementioned kilo of Silk/Yak/Merino. I ran out of yarn on the collar which of course necessitated ordering more of this yarn, which is just utterly delightful. It’s a bottom-up raglan sweater with a v-neck. Nothing else special, and it’s so comfy and soft in every way. I’m not sure I’m sold on Raglan shaping, but I’d forgotten how much fun this construction is.
I started a two-color pull-over using some of my favorite color patterns from Norwegian and Turkish patterns. I’m almost to the shoulder, and the plan is to knit set in shoulders and sleeve caps (short rows!) with a v-neck. This is the first color work sweater I’ve made in years, and it’s been really fun and engaging to knit this sweater. I’d also given away almost all of my previous colorwork sweaters and they are very much my thing, and while I don’t know that it’s the kind of thing I want to focus on, making one of these every so often seems good.

There have been a few things that I’ve learned about myself as a knitter recently:

I don’t really enjoy knitting cables: the dense fabric rarely appeals to me as a sweater wearer, anything that isn’t really simple ends up breaking up the rhythm in a way that I find distracting and the result is that knitting feels like a chore.
I’m no longer afraid of grafting as I once was. I seem to only be able to really do it effectively if I hold the “right” sides together and graft from the “inside” of the garment (sock, sweater/etc,) but this seems to actually be fine in practice. This makes seamless bottom up sweater (as well as socks) much more approachable. In the past I struggled through it or tricked someone into doing it for me (or used 3 needle bind-offs.)
I did a lot of sweaters with open-henly-style collars for years, because I often find that crew neck sweaters are too warm and otherwise overwhelming. Since I started on this path, I’ve begun making sweaters out of fingering weight yarn, and have switched to the v-neck as a solution for the “more open” neck problem. I think this demands more investigation.

In an upcoming post, I’d like to explore some upcoming knitting projects and the design questions that I’ve been contemplating.

Stay tuned!

Short Row Wrapping

Thu, 15 Apr 2021 00:00:00 +0000

Short rows are this little magic thing that you can do in hand knitting where you knit a row over only some of the stitches to create a piece of fabric that is longer in one part than in another, and also when coordinated correctly a sequence of short rows can cause the fabric to curve and bend. It’s makes things possible in hand knitting that aren’t at all possible using other textile process, because you’re creating a piece of fabric that is a custom shape in three dimensions. Short rows appear for lots of reasons: dropping the bottom edge of the sweater, sloping the shoulders of a sweater, forming a top-down-sleeve cap, or to turn a sock heel, for example.

The problem with short rows, is that it can be quite difficult to hide them in an existing fabric, because there’s a little gap or hole where the short row starts. Solving this little problem has given rise to an entire discipline of knitting techniques. Most of the time, after knitting a short row, you “wrap” the next stitch, which you slip and move the yearn around, as the basis of a transition. This anchors the yarn from the short row, and helps reduces an awkward effect on the stitches that you knit.

The problem is that the “wrap” is (often) visiable in your knitting, so that’s in ideal. The options are:

in garter stitch the best thing to do is to just ignore the wrap. If the tension of the wrap itself is right, the wrap doesn’t look out of place, and you can just ignore it.
most of the time you want to “process” the wraps, by picking up the wrap and knitting it together with the (now formerly) wrapped stitch. Getting the tension on this is quite hard, though you can twist the wrap or clean things up in the next row most of the time. If you’re having trouble with wrapping:
- consider wrapping in the other direction. So that you move the yarn to the front of the piece before or after slipping the stitch, depending on what you’re presently doing. In my practice, wrapping front to back is slightly tighter than wrapping back to front, but I think this depends on your hands a bit.
- rather than wrapping the stitch, a small yarn over can serve the same purpose, as long as you’re sure to knit the yarn over with the stitch that was not part of the short row. Use this if your wraps are too tight.
for sock heels, and in some other situations, you can skip the wrap, but slip the first stitch of the short row and decease the short sliped stitch into the next (non-short row) stitch on the next row. This works only in situations where you can stand to decrease 1 stitch for every short row turn.
Some folks enjoy not processing the wraps, in situations where you’re knitting a sleeve cap off from the shoulder down in a sweater. You can see the wraps, but because you’re doing lots of sequential short rows, it looks like a pattern.

The Org Mode Product

Wed, 14 Apr 2021 00:00:00 +0000

As a degenerate emacs user, as it were, I have of course used org-mode a lot, and indeed it’s probably the mode I end up doing a huge amount of my editing in, because it’s great with text and I end up writing a lot of text. I’m not, really, an org mode user in the sense that it’s not the system or tool that I use to stay organized, and I haven’t really done much development of my own tooling or process around using orgmode to handle document production, and honestly, most of the time I use reStructuredText as my preferred lightweight markup language.

I was thinking, though, as I was pondering ox-leanpub, what even is org-mode trying to do and what the hell would a product manager do, if faced with org-mode.

In some ways, I think it sucks the air out of the fun of hacking on things like emacs to bring all of the “professionalization of making software” to things like org-mode, and please trust that this meant with a lot of affection for org-mode: this is meant as a thought experiment.

Org has a lot going on:

it provides a set of compelling tools for interacting with hierarchical human-language documents.
it’s a document markup and structure system,
the table editing features are, given the ability to write formula in lisp, basically a spreadsheet.
it’s a literate programming environment, (babel)
it’s a document preparation system, (ox)
it’s a task manager, (agenda)
it’s a time tracking system,
it even has pretty extensive calendar management tools.

Perhaps the thing that’s most exciting about org-mode is that it provides functionality for all of these kinds of tasks in a single product so you don’t have to bounce between lots of different tools to do all of these things.

It’s got most of the “office” suite covered, and I think (particularly for new people, but also for people like me,) it’s not clear why I would want my task system, my notes system, and my document preparation system to all have their data intermingled in the same set of files. The feeling is a bit unfocused.

The reason for this, historically makes sense: org-mode grew out of technically minded academics who were mostly using it as a way of preparing papers, and who end up being responsible for a lot of structuring their own time/work, but who do most of their work alone. With this kind of user story in mind, the gestalt of org-mode really comes together as a product, but otherwise it’s definitely a bit all over the place.

I don’t think this is bad, and particularly given its history, it’s easy to understand why things are the way they are, but I think that it is useful to take a step back and think about the workflow that org supports and inspires, while also not forgetting the kinds of workflows that it precludes, and the ways that org, itself, can create a lot of conceptual overhead.

There are also some gaps, in org, as a product, which I think grow out of this history, and I think there are

They are, to my mind:

importing data, and bidirectional sync. These are really hard problems, and there’ve been some decent projects over the years to help get data into org, I think org-trello is the best example I can think about, but it can be a little dodgy, and the “import story” pales in comparison to the export story. It would be great if:
- you could use the org interface to interact with and manipulate data that isn’t actually in org-files, or at least where the system-of-record for the data isn’t org. Google docs? Files in other formats?
collaborating with other people. Org-mode files tend to cope really poorly with multiple people editing them at the same time (asynchronously as with git,) and also in cases where not-everyone uses org-mode. One of the side effects of having the implementation of org-features so deeply tied to the structure of text in the org-format, it becomes hard to interact with org-data outside of emacs (again, you can understand why this happens, and it’s indeed very lispy,), which means you have to use emacs and use org if you want to collaborate on projects that use org.
- this might look like some kind of different diff-drivers for git, in addition to some other more novel tools.
- bi-directional sync might also help with this issue.
beyond the agenda, building a story for content that spans multiple-file. Because the files are hierarchical, and org provides a great deal of taxonomic indexing features, you really never need more than one org-file forever, but it’s also kind of wild to just keep everything in one file, so you end up with lots of org-files, and while the agenda provides a way to filter out the task and calendar data, it’s sort of unclear how to mange multi-file systems for some of the other projects. It’s also the case, that because you can inject some configuration at the file level, it can be easy to get stuck.
tools for interacting with org content without (interactive or apparent) emacs. While I love emacs as much as the next nerd, I tend to think that having a dependency on emacs is hard to stomach, particularly for collaborative efforts, (though with docker and the increasing size of various runtimes, this may be less relevant.) If it were trivially easy to write build processes that extracted or ran babel programs without needing to be running from within emacs? What if there were an org-export CLI tool?

Docker Isn't A Build System

Tue, 13 Apr 2021 00:00:00 +0000

I can’t quite decide if this title is ironic or not. I’ve been trying really hard to not be a build system guy, and I think I’m succeeding--mostly--but sometimes things come back at us. I may still be smarting from people proposing “just using docker” to solve any number of pretty irrelevant problems.

The thing is that docker does help solve many build problems: before docker, you had to either write code that supported any possible execution environment. This was a lot of work, and was generally really annoying. Because docker provides a (potentially) really stable execution environment, it can make a lot of sense to do your building inside of a docker container, in the same way that folks often do builds in chroot environments (or at least did). Really containers are kind of super-chroots, and it’s a great thing to be able to give your development team a common starting point for doing development work. This is cool.

It’s also the case that Docker makes a lot of sense as a de facto standard distribution or deployment form, and in this way it’s kind of a really fat binary. Maybe it’s too big, maybe it’s the wrong tool, maybe it’s not perfect, but for a lot of applications they’ll end up running in containers anyway, and treating a docker container like your executable format makes it possible to avoid running into issues that only appear in one (set) of environments.

At the same time, I think it’s important to keep these use-cases separate: try to avoid using the same container for deploying that you use for development, or even for build systems. This is good because “running the [deployment] container” shouldn’t build software, and it’ll also limit the size of your production containers, and avoid unintentionally picking up dependencies. This is, of course, less clear in runtimes that don’t have a strong “compiled artifacts” story, but is still possible.

There are some notes/caveats:

Dockerfiles are actually kind of build systems, and under the hood they’re just snapshotting the diffs of the filesystem between each step. So they work best if you treat them like build systems: make the steps discrete and small, keep the stable deterministic things early in the build, and push the more ephemeral steps later in the build to prevent unnecessary rebuilding.
“Build in one container and deploy in another,” requires moving artifacts between containers, or being able to run docker-in-docker, which are both possible but may be less obvious than some other workflows.
Docker’s “build system qualities,” can improve the noop and rebuild-performance of some operations (e.g. the amount of time to rebuild things if you’ve just built or only made small changes.) which can be a good measure of the friction that developers experience, because of the way that docker can share/cache between builds. This is often at the expense of making artifacts huge and at greatly expanding the amount of time that the operations can take. This might be a reasonable tradeoff to make, but it’s still a tradeoff.

My Code Review Comments

Thu, 08 Apr 2021 00:00:00 +0000

I end up reviewing a lot of code, and while doing code review (and getting reviews) used to take up a lot of time, I think I’ve gotten better at doing reviews, and knowing what’s important to comment on and what doesn’t

The code review process is not there to discover bugs. Write tests to catch bugs, and use the code review process to learn about a specific change, and find things that are difficult to test for.
As yourself if something is difficult to follow, and comment on that. If you can’t figure out what something is doing, or you have to read it more than once, then that’s probably a problem.
Examine and comment on the naming of functions. Does the function appear to do what the name indicates.
Think about the interface of a piece code:
- What’s exported or public?
- How many arguments do your functions take?
Look for any kind of shared state between functions, particularly data that’s mutable or stored in globally accessable, or package local structures.
Focus your time on the production-facing, public code, and less on things like tests and private/un-exported APIs. While tests are important, and it’s important that there’s good test coverage (authors should use coverage tooling to check this), and efficient test execution, beyond this high level aspect, you can keep reading?
Put yourself in the shoes of someone who might need to debug this code and think about logging as well as error handling and reporting.
Push yourself and others to be able to get very small pieces of code reviewed at a time. Shorter reviews are easier to process and while it’s annoying to break a logical component into a bunch of pieces, it’s definitely worth it.

Sleeve Survey

Wed, 07 Apr 2021 00:00:00 +0000

I somehow managed to knit the body of a sweater in like 10 days, and am once again knitting sleeves. I also want to re-knit the cuffs of the last sweater, because I’m coming to terms with the fact that I’m not really happy with them, and my current plan is to knit the sleeves of my next sweater before knitting the body, which means I have a little bit of a queue for knitting sleeves.

In preparation I measured a lot of sleeves of sweaters that I’ve knitted (and still have, to try and figure out what I like. Here are my conclusions:

I realized that regardless of the shoulder shaping, the length of the sleeve should be basically the same. This is based on the assumption that the shoulder fits, and sometimes you can compensate for an illfitting shoulder by modifying the length of the sleeve, so take it with a grain of salt.
I’ve been measuring sleeves, on drop shouldered garments from the shoulder seam to the cuff, and for other shoulder shaping, from the underarm to the cuff.
I prefer sleeves that are a little bloused (e.g. bigger, with a more aggressive cuff,) because I like to wear sweaters over shirts, so having a bit more room makes things more comfortable. Also having floppy cuffs is not great.
The sweaters that I like the most, seem to have 21 inch sleeves total with 1-2 inch cuffs. I try and spread the decreases out, as evenly as possible. The cuffs seem to be between 8 and 9 inches around (ideally,) and shoulder apertures tend to be between 18 and 20 inches around.
I’m tentatively coming out in favor of knitting all sleeves from the shoulder down to the cuff, but I do want to give it a shot going the other way at least a couple of times before I’m definitive on that subject. When you knit the sleeves first of a sweater, the process of knitting the sleeve dictates the yoke of the sweater, which means if you’re off a bit in the sleeve the whole sweater seems off. Knitting sleeves after the yoke, means you don’t have to figure out the entire sweater when you’re knitting the cuff.
Separating thinking about the sleeve cap from thinking about the sleeve, is conceptually useful (sleeve caps take a while, the process of knitting them in the round is different,) even if this doesn’t make a lot of sense when you’re actually knitting or thinking about a garment.

Values for Collaborative Codebases

Tue, 06 Apr 2021 00:00:00 +0000

After, my post on what I do for work I thought it’d be good to describe the kinds of things that make software easy to work on and collaborative. Here’s the list:

Minimal Documentation. As a former technical writer this is sort of painful, but most programning environments (e.g. languages) have idioms and patterns that you can follow for how to organize code, run tests and build artifacts. it’s ok if your project has exceptional requirements that require you to break the rules in some way, but the conventions should be obvious and the justification for rule-breaking should be plain. If you adhere to convention, you don’t need as much documentation. It’s paradoxical, because better documentation is supposed to facilitate accessibility, but too much documentation is sometimes an indication that things are weird and hard.
Make it Easy To Run. I’d argue that the most difficult (and frustrating) part of writing software is getting it to run everywhere that you might want it to run. Writing software that runs, even does what you want on your own comptuer is relatively, easy: making it work on someone else’s computer is hard. One of the big advantages of developing software that runs as web apps means that you (mostly) get to control (and limit) where the software runs. Making it possible to easily run a piece of software on any computer it might reasonably run (e.g. developer’s computers, user’s computers and/or production environments.) Your software itself, should be responsible for managing this environment, to the greatest extent possible. This isn’t to say that you need to use containers or some such, but having packaging and install scripts that use well understood idioms and tools (e.g. requirements.txt, virtualenv, makefiles, etc.) is good.
Clear Dependencies. Software is all built upon other pieces of software, and the versions of those libraries are important to record, capture, and recreate. Now it’s generally a good idea to update dependencies regularly so you can take advantage of improvements from upstream providers, but unless you regularly test against multiple versions of your dependencies (you don’t), and can control all of your developer and production environments totally (you can’t), then you should provide a single, centralized way of describing your dependencies. Typically strategies involve: vendoring dependencies, using lockfiles (requirements.txt and similar) or build system integration tends to help organize this aspect of a project.
Automated Tests. Software requires some kind of testing to ensure that it has the intended behavior, and tests that can run quickly and automatically without requiring users to exercise the software manually is absolutely essential. Tests should run quickly, and it should be possible to run a small group of tests very quickly to support iterative development on a specific area of the code. Indeed, most software development can and should be oriented toward the experience of writing tests and exercising new features with tests above pretty much everything else. The best test suites exercise the code at many levels, ranging from very small unit tests to ensure the correct behavior of the functions and methods, to higher level tests that test the functionality of higher-order functions and methods, and full integration tests that test the entire system.
Continuous Integration. Continuous integration system’s are tools that support development and ensure that changes to a code pass a more extensive range of tests than are readily available to developers. CI systems are also useful for automating other aspects of a project (releases, performance testing, etc.) A well maintained CI environment provide the basis for commonality for larger projects with a larger number for projects larger groups of developers, and is all but required to ensure a well supported automated test system and well managed dependency.
Files and Directories Make Sense. Code is just text in files, and software is just a lot of code. Different languages and frameworks have different expectations about how code is organized, but you should be able to have a basic understanding of the software and be able to browse the files, and be able to mostly understand what components are and how they relate to each other based on directory names, and should be able to (roughly) understand what’s in a file and how the files relate to eachother based on their names. In this respect, shorter files, when possible are nice, are directory structures that have limited depth (wide and shallow,) though there are expections for some programming language.
Isolate Components and Provide Internal APIs. Often when we talk about APIs we talk about the interface that users access our software, but larger systems have the same need for abstractions and interfaces internally that we expose for (programmatic) users. These APIs have different forms from public ones (sometimes,) but in general:
- Safe APIs. The APIs should be easy to use and difficult to use incorrectly. This isn’t just “make your API thread safe if your users are multithreaded,” but also, reduce the possibility for unintended side effects, and avoid calling conventions that are easy to mistake: effects of ordering, positional arguments, larger numbers of arguments, and complicated state management.
- Good API Ergonomics. The ergonomics of an API is somewhat ethereal, but it’s clear when an API has good ergonomics: writing code that uses the API feels “native,” it’s easy to look at calling code and understand what’s going on, and errors that make sense and are easy to handle. It’s not simply enough for an API to be safe to use, but it should be straightforward and clear.

The One Page Brainstorm

Wed, 31 Mar 2021 00:00:00 +0000

For a few years, rather than really double down on some kind of electronic note taking and task management tool, I’ve mostly done that kind of work long hand, using pen a paper. I’m not great at using a pen to write, but it can be fun, and I quite enjoy the opportunity to take a step back, slow down, and focus on some brain work. I don’t really have a system, as such, though I did read the bullet journal website and I guess I probably stole some of those ideas.

I definitely keep a pad on my desk that’s some kind of loose idea of “things I want to get done today or in the next couple of days.” Its less a “todo list,” in the sense that it’s not really an exhaustive list of everything that I need to get done, and I don’t use so that I can write things down and then forget them, but more so that when I sit down at my desk, I can more quickly find something useful to do, and that every few days when look over the previous list, I can get a sense of what I’ve done (or not done.)

I also, and this is perhaps more interesting, have a notebook that’s just for brainstorming. I’ve sort of intentionally selected “half-size” (e.g. 6.5" x 8.25") notebooks for this purpose. My routine is pretty simple: make a point of sitting down with a pen and a blank sheet of paper periodically and just brainstorming or writing about a topic. It’s really easy to fill a page and often easy enough to fill the facing page with a bunch of ideas about whatever topic it is, usually something I’m writing but also sometimes code or another kind of project.

I think I developed this practice when writing fiction, which I don’t seem to do much of, any more, (though I’ve been making notes for a couple of years so I suspect it’ll happen,) but the idea is less to be systematic about the notes, or to collect them in ways that will be textually useful in the future, but more as a way of focusing and putting all or most of your attention on a problem for a few minutes. Often, though, the notes are useful, both because they force you to answer questions about what you’re working on and can function as a creative jumpstart, both when doing work as a thing to review and also because sometimes once you start doing the mental work moving into actual work (typically for me, typing) becomes compelling. I suppose there is almost a meditative quality to the activity.

When thinking about writing fiction, the “write a half sized page full of notes” about an anecdote from a character, or a bit of worldbuilding, or a description of a plot progression (at any scale,) makes a lot of sense and use a useful exercise.

In some ways, I suppose, this is a little bit if a follow up to my Get More Done post, as one of the things I do help begin to make progress or make sure that it’s easy to focus and have a clear idea of what to do when you’re ready to write more formally.

How to Become a Self-Taught Programmer

Tue, 30 Mar 2021 00:00:00 +0000

i am a self taught programmer. i don’t know that i’d recommend it to anyone else there are so many different curricula and training programs that are well tested and very efficacious. for lots of historical reasons, many programmers end up being all or mostly self taught: in the early days because programming was vocational and people learned on the job, then because people learned programming on their own before entering cs programs, and more recently because the demand for programmers (and rate of change) for the kinds of programming work that are in the most demand these days. and knowing that it’s possible (and potentially cheaper) to teach yourself, it seems like a tempting option.

this post, then, is a collection of suggestions, guidelines, and pointers for anyone attempting to teach themselves to program:

focus on learning one thing (programming language and problem domain) at a time. there are so many different things you could learn, and people who know how to program seem to have an endless knowledge of different things. knowing one set of tools and one area (e.g. “web development in javascript,” or “system administration in python,") gives you the framework to expand later, and the truth is that you’ll be able to learn additional things more easily once you have a framework to build upon.
when learning something in programming, always start with a goal. have some piece of data that you want to explore or visualize, have a set of files that you want to organize, or something that you want to accomplish. learning how to program without a goal, means that you don’t end up asking the kinds of questions that you need to form the right kinds of associations.
programming is all about learning different things: if you end up programming for long enough you’ll end up learning different languages, and being able to pick up new things is the real skill.
being able to clearly capture what you were thinking when you write code is basically a programming super power.
programming is about understanding problems¹ abstractly and building abstractions around various kinds of problems. being able break apart these problems into smaller core issues, and thinking abstractly about the problem so that you can solve both the problem in front of you and also solve it in the future are crucial skills.
collect questions or curiosities as you encounter them, but don’t feel like you have to understand everything, and use this to guide your background reading, but don’t feel like you have to hunt down the answer to every term you hear or see that you don’t already know immediately. if you’re pretty rigorous about going back and looking things up, you’ll get a pretty good base knowledge over time.
always think about the users of your software as you build, at every level. even if you’re building software for your own use, think about the future version of yourself that will use that software, imagine that other people might use the interfaces and functions that you write and think about what assumptions they might bring to the table. think about the output that your program, script, or function produces, and how someone would interact with that output.
think about the function as the fundamental building block of your software. lower level forms (i.e. statements) are required, but functions are the unit where meaning is created in the context of a program. functions, or methods depending, take input (arguments, ususally, but sometimes also an object in the case of methods) and produce some output, sometimes with some kind of side-effect. the best functions:
- clearly indicate side-effects when possible.
- have a mechanism for reporting on error conditions (exceptions, return values,)
- avoid dependencies on external state, beyond what is passed as arguments.
- are as short as possible.
- use names that clearly describe the behavior and operations of the function.
if programming were human language (english,) you’d strive to construct functions that were simple sentences and not paragraph’s, but also more than a couple of words/phrases, and you would want these sentences to be clear to understand with limited context. if you have good functions, interfaces are more clear and easier to use, code becomes easier to read and debug, and easier to test.
avoid being too weird. many programmers are total nerds, and you may be too, and that’s ok, but it’s easier to learn how to do something if there’s prior art that you can learn from and copy. on a day-to-day basis, a lot of programming work is just doing something until you get stuck and then you google for the answer. If you’re doing something weird--using a programming language that’s less widely used, or in a problem space that is a bit out of mainstream, it can be harder to find answers to your problems.

Notes

I use the term “problem” to cover both things like “connecting two components internally” and also more broadly “satisfying a requirement for users,” and programming often includes both of these kinds of work. ↩︎

Quarantine Knitting Television

Thu, 25 Mar 2021 00:00:00 +0000

I’ve always enjoyed watching television while knitting, knitting itself is easy and doesn’t take much thought most of the time, and I’ve never needed to watch it for every stitch, so it makes sense. I’ve never really felt as engaged with audio-only media while knitting for long stretches, though I do try sometimes.

My tastes, in this context, tend toward procedurals: not particularly because I love crime and/or medical shows, but because the cadence of the story telling is a good fit for my attentional needs while knitting, and the overall predictability means my attention can drift in and out as need be. It’s also helpful that a lot of procedurals have had long runs which means there’s a lot of material to keep me busy. Things that I’ve watched recently:

Bones has been quite fun to watch because it sort of fits the bill correctly, there are a lot of episodes. It’s also very interesting because I remember watching some of the early seasons more or less when they aired, and it totally feels dated now.
Stargate Universe, which couldn’t stick with, despite a lot of affection for earlier Stargate shows, I couldn’t stick it out. I think that some combination of the show being very dark and isolating, with a few of the characters being deeply unlikeable.
The last few years of Star Trek TNG and Voyager. Which were nice, but it’s super weird to watch television that’s this episodic, and while I’ve watched a lot of Star Trek here and there, I must confess that I didn’t really watch them systematically before.
Eureka is totally absurd, but was fun to watch, and was the right kind of “delightful, but mostly lighthearted.”
White Collar has been fun. I’m particularly partial to shows set in New York City, and it’s fun to be able to pick out the settings from familiar places. Sometimes the characters are great, and sometimes some of the “character growth” stories are a little bit ham-fisted. I also wonder what’d be like if there was a buddy-cop show where there characters were actually gay and not just incredibly homoerotic all the time.

I’m not super sure where to go after this, and would gladly take suggestions if people have favorites. I haven’t tended to enjoy things like CSI very much, for whatever that’s worth.

Knitting Gadgets

Wed, 24 Mar 2021 00:00:00 +0000

While I took a bunch of time off knitting, I didn’t quite take enough time off to completely divest myself of all of my knitting things, which means when I decided rather abruptly that I wanted to start knitting again, I just had to run and pull a couple of boxes down off a shelf and I was off to the races. At the same time, after many years of small apartment living I didn’t have a very large collection of gadgets or yarn.

While I don’t really have a great interest in building a collection of knitting things, or yarn outside of material I have proximal use for, in the nearly 10 years, since I was a regular knitter, the state of the craft has evolved or at least changed a bit. While my collection of things hasn’t changed much, the following objects

knitting needles, have always been complicated. I tend to work at fairly small sizes which makes needle flex/bend an issue, and my skin tends to react with nickle which rules out a lot of options.
- at my mother’s recommendation I got a few Dyak Craft knitting needles. The small-sized interchangeable needles are great, and the US 0s (which has been my primary needle) don’t bend or flex at all, and have good cable/needle joins.
- I have a small collection of 5 inch carbon fiber, double pointed needles in some small sizes and I think they’re just perfect. I’m a loose knitter, so the little bit of grip that they have is great. Somehow, metal needles and six inch needles end up hurting my hands, and I’ve never used a pair of wooden needles that haven’t broken tragically.
After a couple of projects where I was just breaking the yarn by hand, or using kitchen scissors, I gave in and bought a 4 inch pair of very plain Gingher embroidery scissors (for 20 bucks,) and they’re brilliant for trimming threads and cutting open steeks. As a left handed human, scissors have always been something of a sore spot, and these are quite good.
I bought a couple of boxes of those lightbulb-shaped coil-less safety pins. I think the going rate is 6 bucks for a pack of 120, and they come in a few different colors. These are great both as stitch markers for the needle, and also to mark rows while knitting.
When I was going through my knitting things, I very quickly found a little cone of 8/2 mercerized cotton that I’ve had for years in lime green--a color I’ve never even gotten close to knitting with--that’s really perfect for setting stitches aside or for provisional cast-ons. It’s nice to use smooth, non-wool yarns for this purpose, and you don’t use very much of it, but it’s great to have around. While I’ve had this cone for 16 years or more, and it’s conceivable that I may never finish it, it’s great.
I’ve also given in a started storing in-progress knitting projects in draw-string canvas bags purpose built for knitting/crafting, as opposed to the vinyl bags that bed-linens come in, which had previously been my default. It’s pretty essential that I be able to keep things safe from cats or the other things in my backpack (not that I leave the house much these days,) so containment is necessary, and avoiding velcro and zippers is ideal.

And that’s about it!

Test Multi-Execution

Tue, 23 Mar 2021 00:00:00 +0000

Editoral Note: this is a follow up to my earlier Principles of Test Oriented Software Development post.

In software development, we write tests to make sure the code we write does what we want it to do. Great this is pretty easy to get behind.

Tests sometimes fail.

The goal, is that, most of the time when tests fail, it’s because the code is broken: you fix the code and the test passes. Sometimes when test fail there’s a bug in the test, it makes an assertion that can’t or shouldn’t be true: these are bad because they mean the test is broken, but all code has bugs, and test code can be broken so that’s fine.

Ideally either pass or fail, and if a test fails it fails repeatedly, with the same error. Unfortunately, this is of course not always true, and tests can fail intermittently if they test something that can change, or the outcome of the test is impacted by some external factor like “the test passes if the processor is very fast, and the system does not have IO contention, but fails sometimes as the system slows down.” Sometimes tests include (intentionally or not) some notion of “randomnesses,” and fail intermittently because of this.

A test suite with intermittent failures is basically the worst. A suite that never fails isn’t super valuable, because it probably builds false confidence, a test suite that always fails isn’t useful because developers will ignore the results or disable the tests, but a test that fails intermittently, particularly one that fails 10 or 20 percent of the time, means that developers always will always look at the test, or just rerun the test until it passes.

There are a couple of things you can do to fix your tests:

write better tests: find sources of non-determinism in your test and rewrite tests to avoid these kinds of “flaky” outcomes. Sometimes this means restructuring your tests in a more “pyramid-like” structure, with more unit tests and fewer integration tests (which are likely to be less deterministic.)
run tests more reliably: find ways of running your test suite that produce more consistent results. This means running tests in more isolated environments, changing the amount of test parallelism, ensure that tests clean up their environment before they run, and can be as logically isolated as possible.

But it’s hard to find these tests and you can end up playing wack-a-mole with dodgy tests for a long time, and the urge to just run the tests a second (or third) time to get them to pass so you can merge your change and move on with your work is tempting. This leaves:

run tests multiple times: so that a test doesn’t pass until it passes multiple times. Many test runner’s have some kind of repeated execution mode, and if you can combine with some kind of “stop executing after the first fail,” then this can be reasonably efficient. Use multiple execution to force the tests to produce more reliable results rather than cover-up or exacerbates the flakiness.
run fewer tests: it’s great to have a regression suite, but if you have unreliable tests, and you can’t use the multi-execution hack to smoke out your bad tests, then running a really full matrix of tests is just going to produce more failures, which means you’ll spend more of your time looking at tests, in non-systematic ways, which are unlikely to actually improve code.

Pattern Fragment 4

Thu, 04 Mar 2021 00:00:00 +0000

This is the follow up to Pattern Fragment 0, Pattern Fragment 1, Pattern Fragment 2, and Pattern Fragment 3.

Cut open the arm hole steeks, and joining new yarn at the top of the opening (after the shoulder seam), pick up stitches around the arm hole, at a rate of 2 stitches for every three rows. I picked up two extra stitches in the “corners” at the bottom of the sleeve and did not pick up a stitch from the shoulder seam.

Also, place markers where the shoulder decreases start at the bottom of the opening.

Knit 9 stitches, move yarn to front, slip the next stitch, move the yarn to the back, change directions,¹ slip the first stitch (again,) and work back to the “beginning of the row,” at the shoulder seam. Work the next nine stitches, slip the next stitch, move the yarn to the front, change directions, slip the first stitch (again), move the yarn to the back, and knit across.

When you get to a “wrapped stitch” at the end of this “short row,” pick up the wrap and knit it together with the stitch it wrapped, and then wrap the next stitch.

Continue in this manner until you get to the markers where the arm hole shaping ends, before switching to knitting in the round for the main body of the sleeve.

You could turn the work, but I do this part just by knitting back backwards. ↩︎

A Selection of Foutain Pen Recommendations

Wed, 03 Mar 2021 00:00:00 +0000

In addition to all of my other weird hobbies and practices, I have a minor fascination with and enjoyment of fountain pens, and have for a long time. After many years of not really writing very much long hand, I decided to get back to it a couple of years ago, and have amassed some pens, but perhaps more relevantly here, some opinions, which I thought I’d reflect on here.

A few notes and pieces of context first:

I am left handed, which I think makes me more intolerant of slow drying inks, and pens that have unexceptional writing experiences.
All of my long-hand writing is for myself. Aside from occasionally writing my name in the cover of a tune books, or something similar.
I’m color blind, and have stayed away from ink colors in the red/purple spectrum, opting for mostly blue/black, blues, and blue-green, colors (with one exception.)
None of these recommendations are sponsored. If you enjoy them, let me know!

So here we go.

I mostly enjoy using Clairefontaine notebooks, mostly wirebound, and mostly as it turns out in the A5 size. I keep a steno pad on my desk for todolists and notes, and a side-bound notebook around for longer form notes.
At least Somewhat to my own surprise, my favorite pens appear to be piston-filled German made pens with medium nibs. European pens tend to run more broad than average, and that’s definitely true of my favorites:
- Lamy 2000 is very modern looking and feeling, and it feels great to write with. The nib is very wet, so I’ve made a habit of using the quickest drying ink I have, and it’s worked out well. If there’s a complain, it’s that the cap-seal is less than perfect if you are (as I am) an infrequent writer.
- Pelikan m405 looks like a fountain pen, and while I suppose that’s classic, and it is a very smart pen, I wasn’t expecting this to be such a perfect pen. It’s a joy to write with, the gold nib felt worth it, and the small size is actually an asset (I also have the larger m805, and find myself going for the smaller pen more often.)
Also compelling are Namiki Vanishing Point which write really well. and are just really damn cool. The VP pens are great for regular use and short-bursts of writing. As a Japanese pen, the nibs run fine, and the gold nib versions are worth it. The Decimo version is great (smaller and thinner,) but the carbon-fiber ones are pretty compelling. I use the bladder converters, and I like them, but it’s a bit weird. Namiki cartrages are great as well, particularly the blue-black ones and they’re particularly easy to eye-dropper, if that’s your thing. Additionally, I have a Lamy Studio with a fine (steel) nib that’s way more compelling.
Surprisingly compelling also is the TWSBI Diamond Mini and TWSBI Eco. I think I have a slight preference for the Mini over its larger version, and the Eco as well, but they’re all great pens, particularly for the price, and as piston fillers, they’re great if you’re new to fountain pens or you want to have a color of ink available to you but not in regular rotation. The nibs are steel and you can tell, but it’s not a huge detriment.
The most compelling inks are, to my mind: Noodler’s Q-Eternity, which is a nice blue-black color and dries really fast. Having said that, it’s probably still second in my mind to Diamine Twilight, which also dries quite quickly, is a very similar color, and has really compelling shading. Thankfully there’s no need to pick favorites.
I have a short list of inks that I think are really good and worth being aware of if you’re not. First Pilot’s Standard Blue Black is a great ink, it dries pretty quick, the bottles are huge, and is somehow not super wet but also wet enough. I have a steel-nibbed Vanishing point pen, and this ink turned it from “meh” to compelling. I wasn’t expecting to really like Diamine Oxblood which is a red-black ink, I suppose, but it’s very nice. Finally, the Pilot Iroshizuku inks are very nice, and the colors are cool. I’m particularly fond of Tuski-Yo and Shin-Kai, but I suspect it’s hard to go wrong.

That’s it! I probably won’t blog much about this, but would definitely be down to chat more about it, or pull together some posts if it’s not too for you all.

Principles of Test Oriented Software Development

Tue, 02 Mar 2021 00:00:00 +0000

I want to like test-driven-development (TDD), but realistically it’s not something that I ever actually do. Part of this is because TDD, as canonically described is really hard to actually pratice: TDD involves writing tests before writing code, writing tests which must fail before the implementation is complete or correct, and then using the tests to refactor the code. It’s a nice idea, and it definitely leads to better test coverage, but the methodology forces you to iterate inefficiently on aspects of a design, and is rarely viable when extending existing code bases. Therefore, I’d like to propose a less-dogmatic alternative: test-oriented-development.¹

I think, in practice, this largely aligns with the way that people write software, and so test oriented development does not describe a new way of writing code or of writing tests, but rather describes the strategies we use to ensure that the code we write is well tested and testable. Also, I think providing these strategies in a single concrete form will present a reasonable and pragmatic alternative to TDD that will make the aim of “developing more well tested software” more achievable.

Make state explicit. This is good practice for all kinds of development, but generally, don’t put data in global variables, and pass as much state (configuration, services, etc.) into functions and classes rather than “magicing” them.
Methods and functions should be functional. I don’t tend to think of myself as a functional programmer, as my tendencies are not very ideological, in this regard, but generally having a functional approach simplifies a lot of decisions and makes it easy to test systems at multiple layers.
Most code should be internal and encapsulated. Packages and types with large numbers of exported or public methods should be a warning sign. The best kinds of tests can provide all desired coverage, by testing interfaces themselves,
Write few simple tests and varry the data passed to those tests. This is essentially a form of “table driven testing,” where you write a small sequence of simple cases, and run those tests with a variety of tests. Having test infrastructure that allows this kind of flexibility is a great technique.
Begin writing tests as soon as possible. Orthodox TDD suggests that you should start writing tests first, and I think that this is probably one of the reasons that TDD is so hard to adopt. It’s also probably the case that orthodox TDD emerged when prototyping was considerably harder than it is today, and as a result TDD just feels like friction, because it’s difficult to plan implementations in a test-first sort of way. Having said that, start writing tests as soon as possible.
Experiment in tests. Somehow, I’ve managed to write a lot of code without working an interactive debugger into my day-to-day routine, which means I do a lot of debugging by reading code, and also by writing tests to try and replicate production phenomena in more isolated phenomena. Writing and running tests in systems is a great way to learn about them.

Sorry that this doesn’t lead to a better acronym. ↩︎

Pattern Fragment 3

Thu, 25 Feb 2021 00:00:00 +0000

This is the follow up to Pattern Fragment 0, Pattern Fragment 1, and Pattern Fragment 2.

Starting at the “end” of the back of the neck, join new yarn, and with a short circular needle pick up stitches around the steek. Be sure to pick up the first/last stitch of the steek. If you did not set aside a stitch at the bottom of the neck, increase one stitch at the bottom point of the neck.

When you’ve completed picking up stitches and have knit (plain) across the back of the neck, begin knitting in Knit 1 Purl One Ribbing, being sure to mirror left-to-right at the bottom of the knit (i.e. if the last stitch before your “point” or “bottom of the neck stitch” is a knit, then you should knit the stitch right after the point.) Always knit the “point” stitch. Additionally, on this first row you should do a centered double decrease at the point.

My favorite centered double decrease is a “slip 2 together, knit 1, lift the two slipped stitches over the knit stitch.”

Continue knitting the collar, in this ribbing, doing one double decrease every other row, for an inch and a half. Bind off normally.

Editor Themes

Wed, 24 Feb 2021 00:00:00 +0000

It’s not real secret that I’m red-green colorblind. It’s not really a major life obstacle: I’ve got a wardrobe of clothes in colors that I can easily tell apart, I have developed a number of heuristics for guessing colors that are right enough, and mostly it just creates funny stories where I get a confused look if I try to describe something that might be purple, or have to convince a coworker into reading a graph for me.

One challenge historically, however, has been various kinds of text editing color themes: so often they end up having some kind of low contrast situation that’s hard to read, or two different aspects of code that should be highlighted differently but aren’t. I’ve tried lots of themes out, and I would always end up just going back and using default emacs themes, which might not have been great, but I always found it useable.

Until Protesilaos Stavrou’s Modus Themes, that is.

These are super compelling and I never really knew how good an editor could look until I started using it. Like is this what it’s really like for the rest of you all the time? Things are clear: I never get confused between type names and function names any more. There are rarely situations where I feel like the highlighting color and text color are the same, which used to happen all the time.

The real win, I think, is that Modus' makes dark themes accessible to me, in a way that they never were before. For the most part “dark themes” which have been so popular recently, are just impossible to see clearly (for me), I also find that it’s less taxing to spend time in front of screens when darker backgrounds, so being able to spend most of my time doing work in an environment that’s easy to read. I tend to keep to light backgrounds when using a laptop and dark backgrounds otherwise.

The second piece is that, I think I’ve caved in and decided to increase the size of the font, by default in my text editor, at least when I’m using my desktop/external monitor. I think my vision is as good as it’s been, though I should probably get that checked out post-pandemic. I think there’s a balance between “small fonts let you see more of the file you’re working on,” and “larger fonts let you focus on the area that you’re editing.” When I’m writing English, the focus is great, and when writing software I tend to want more context. There’s a balance also in wanting to keep an entire line length viable at once, and an ideal words-per-line limit for text that’s useful for making things easier to read. So there’s some tuning there, depending on what your workload looks like.

I guess if there is any lesson in this it’s that: Comfort matters, and you shouldn’t push yourself into uncomfortable display situations if you can.

Does Anyone Actually Want Serverless?

Tue, 23 Feb 2021 00:00:00 +0000

Cloud computing, and with it most of tech, has been really hot on the idea of “serverless” computing, which is to say, services and applications that are deployed, provisioned, and priced separate from conventional “server” resources (memory, storage, bandwidth.) The idea is that we can build and expose ways of deploying and running applications and services, even low-level components like “databases” and “function execution”, in ways that mean that developers and operators can avoid thinking about computers qua computers.

Serverless is the logical extension of “platform as a service” offerings that have been an oft missed goal for a long time. You write high-level applications and code that is designed to run in some kind of sandbox, with external services provided in some kind of ala carte model via integrations with other products or services. The PaaS, then, can take care of everything else: load balancing incoming requests, redundancy to support higher availability, and any kind of maintains on the lower level infrastructure. Serverless is often just PaaS but more: provide a complete stack of services to satisfy needs (databases, queues, background work, authentication, caching, on top of the runtime,) and then change the pricing model to be based on request/utilization rather than time or resources.

Fundamentally, this allows the separation of concerns between “writing software,” and “running software,” and allows much if not all of the running of software to be delegated to service providers. This kind of separation is useful for developers, and in general runtime environments seem like the kind of thing that most technical organizations shouldn’t need to focus on: outsourcing may actually be good right?

Well maybe.

Let’s be clear, serverless platforms primarily benefit the provider of the services for two reasons:

serverless models allow providers to build offerings that are multi-tenant, and give provider the ability to reap the benefit of managing request load dynamically and sharing resources between services/clients.
utilization pricing for services is always going to be higher than commodity pricing for the underlying components. Running your on servers (“metal”) is cheaper than using cloud infrastructure, over time, but capacity planning, redundancy, and management overhead, make that difficult in practice. The proposition is that while serverless may cost more per-unit, it has lower management costs for users (fewer people in “ops” roles,) and is more flexible if request patterns change.

So we know why the industry seems to want serverless to be a thing, but does it actually make sense?

Maybe?

Makers of software strive (or ought to) make their software easy to run, and having very explicit expectations about the runtime environment, make software easier to run. Similarly, being able to write code without needing to manage the runtime, monitoring, logging, while using packaged services for caching storage and databases seems like a great boon.

The downsides to software producers, however, are plentiful:

vendor lock-in is real, not just because it places your application at the mercy of an external provider, as they do maintenance, or as their API and platform evolves on their timeframe.
hosted systems, mean that it’s difficult to do local development and testing: either every developer needs to have their own sandbox (at some expense and management overhead), or you have to maintain a separate runtime environment for development.
application’s cannot have service levels which exceed the service level agreements of their underlying providers. If your serverless platform has an SLA which is less robust than the SLA of your application you’re in trouble.
when something breaks, there are few operational remedies available. Upstream timeouts are often not changeable and most forms of manual intervention aren’t available.
pricing probably only makes sense for organizations operating at either small scale (most organizations, particularly for greenfield projects,) but is never really viable for any kind of scale, and probably doesn’t make sense in any situation at scale.
some problems and kinds of software just don’t work in a serverless model: big data sets that exceed reasonable RAM requirements, data processing problems which aren’t easily parallelizable, workloads with long running operations, or workloads that require lower level network or hardware access.
most serverless systems will incur some overhead over dedicated/serverfull alternatives and therefore have worse performance/efficiency, and potentially less predictable performance, especially in very high-volume situations.

Where does that leave us?

Many applications and bespoke tooling should probably use serverless tools. Particularly if your organization is already committed to a specific cloud ecosystem, this can make a lot of sense.
Prototypes, unequivocally make sense to rely on off-the-shelf, potentially serverless tooling, particularly for things like runtimes.
If and when you begin to productionize applications, find ways to provide useful abstractions between the deployment system and the application. These kinds of architectural choices help address concerns about lock-in and making it easy to do development work without dependencies.
Think seriously about your budget for doing operational work, holistically, if possible, and how you plan to manage serverless components (access, cost control, monitoring and alerting, etc.) in connection with existing infrastructure.

Serverless is interesting, and I think it’s interesting to say “what if application development happened in a very controlled environment with a very high level set of APIs.” There are clearly a lot of cases where it makes a lot of sense, and then a bunch of situations where it’s clearly a suspect call. And it’s early days, so we’ll see in a few years how things work out. In any case, thinking critically about infrastructure is always a good plan.

The Kubernetes Cloud Mainframe

Thu, 18 Feb 2021 00:00:00 +0000

I made a tongue-in-cheek comment on twitter a while back that, k8s is just the contemporary API for mainframe computing., but as someone who is both very skeptical and very excited about the possibilities of kube, this feels like something I want to expand upon.

A lot of my day-to-day work has some theoretical overlap with kube, including batch processing, service orchestration, and cloud resource allocation. Lots of people I encounter are also really excited by kube, and its interesting to balance that excitement with my understanding of the system, and to watch how Kubernetes(as a platform) impacts the way that we develop applications.

I also want to be clear that my comparison to mainframes is not a disparagement, not only do I think there’s a lot of benefit to gain by thinking about the historic precedents of our paradigm. I would also assert that the trends in infrastructure over the last 10 or 15 years (e.g. virtualization, containers, cloud platforms) have focused on bringing mainframe paradigms to a commodity environment.

Observations

clusters are static ususally functionally. I know that the public clouds have autoscaling abilities, but having really elastic infrastructure requires substantial additional work, and there are some reasonable upper-limits in terms of numbers of nodes, which makes it hard to actually operate elastically. It’s probably also the case that elastic infrastructure has always been (mostly) a pipe dream at most organizations.
some things remain quite hard, chiefly in my mind:
- autoscaling, both of the cluster itself and of the components running within the cluster. Usage patterns are don’t always follow easy to detect patterns, so figuring out ways to make infrastructure elastic may take a while to converse or become common. Indeed, VMs and clouds were originally thought to be able to provide some kind of elastic/autoscaling capability, and by and large, most cloud deployments do not autoscale.
- multi-tenancy, where multiple different kinds of workloads and use-cases run on the same cluster, is very difficult to schedule for reliably or predictably, which leads to a need to overprovision more for mixed workloads.
kubernettes does not eliminate the need for an operations team or vendor support for infrastructure or platforms.
decentralization has costs, and putting all of the cluster configuration in etcd imposes some limitations, mostly around performance. While I think decentralization is correct, in many ways for Kubernetes, applications developers may need systems that have lower latency and tighter scheduling abilities.
The fact that you can add applications to an existing cluster, or host a collection of small applications is mostly a symptom of clusters being over provisioned. This probably isn’t bad, and it’s almost certainly the case that you can reduce the overprovisioning bias with kube, to some degree.

Impact and Predictions

applications developed for kubernettes will eventually become difficult or impossible to imagine or run without kubernettes. This has huge impacts on developer experience and test experience. I’m not sure that this is a problem, but I think it’s a hell of a dependency to pick up. This was true of applications that target mainframes as well.
Kubernetes will eventually replace vendor specific APIs for cloud infrastructure for most higher level use cases.
Kubernetes will primarily be deployed by Cloud providers (RedHat/IBM, Google, AWS, Azure, etc.) rather than by infrastructure teams.
Right now, vendors are figuring out what kinds of additional services users and applications need to run in Kubernetes, but eventually there will be another layer of tooling on top of Kubernetes:
- logging and metrics collection.
- deployment operations and configuration, particularly around coordinating dependencies.
- authentication and credential management.
- low-latency offline task orchestration.
At some point, we’ll see a move multi-cluster orchestration, or more powerful tools approach to workload isolation within a single cluster.

Conclusion

Kubernetes is great, and it’s been cool to see how, really in the last couple of years, it’s emerged to really bring together things like cloud infrastructure and container orchestration. At the same time, it (of course!) doesn’t solve all of the problems that developers have with their infrastructure, and I’m really excited to see how people build upon Kubernetes to achieve some of those higher level concerns, and make it easier to build software on top of the resulting platforms.

Programming in the Common Lisp Ecosystem

Tue, 16 Feb 2021 00:00:00 +0000

I’ve been writing more and more Common Lips recently and while I reflected a bunch on the experience in a recent post that I recently followed up on .

Why Ecosystems Matter

Most of my thinking and analysis of CL comes down to the ecosystem: the language has some really compelling (and fun!) features, so the question really comes down to the ecosystem. There are two main reasons to care about ecosystems in programming languages:

a vibrant ecosystem cuts down the time that an individual developer or team has to spend doing infrastructural work, to get started. Ecosystems provide everything from libraries for common tasks as well as conventions and established patterns for the big fundamental application choices, not to mention things like easily discoverable answers to common problems.

The more time between “I have an idea” to “I have running (proof-of-concept quality) code running,” matters so much. Everything is possible to a point, but the more friction between “idea” and “working prototype” can be a big problem.
a bigger and more vibrant ecosystem makes it more tenable for companies/sponsors (of all sizes) to choose to use Common Lisp for various projects, and there’s a little bit of a chicken and egg problem here, admittedly. Companies and sponsors want to be confidence that they’ll be able to efficiently replace engineers if needed, integrate or lisp components into larger ecosystems, or be able to get support problems. These are all kind of intangible (and reasonable!) and the larger and more vibrant the ecosystem the less risk there is.

In many ways, recent developments in technology more broadly make lisp slightly more viable, as a result of making it easier to build applications that use multiple languages and tools. Things like microservices, better generic deployment orchestration tools, greater adoption of IDLs (including swagger, thrift and GRPC,) all make language choice less monolithic at the organization level.

Great Things

I’ve really enjoyed working with a few projects and tools. I’ll probably write more about these individually in the near future, but in brief:

chanl provides. As a current/recovering Go programmer, this library is very familiar and great to have. In some ways, the API provides a bit more introspection, and flexibility that I’ve always wanted in Go.
lake is a buildsystem tool, in the tradition of make, but with a few additional great features, like target namespacing, a clear definition between “file targets” and “task targets,” as well as support for SSH operations, which makes it a reasonable replacement for things like fabric, and other basic deployment tools.
cl-docutils provides the basis for a document processing system. I’m particularly partial because I’ve been using the python (reference) implementation for years, but the implementation is really quite good and quite easy to extend.
roswell is really great for getting started with CL, and also for making it possible to test library code against different implementations and versions of the language. I’m a touch iffy on using it to install packages into it’s own directory, but it’s pretty great.
ASDF is the “buildsystem” component of CL, comparable to setuptools in python, and it (particularly the latest versions,) is really great. I like the ability to produce binaries directly from asdf, and the “package-inferred” is a great addition (basically, giving python-style automatic package discovery.)
There’s a full Apache Thrift implementation. While I’m not presently working on anything that would require a legit RPC protocol, being able to integrate CL components into larger ecosystem, having the option is useful.
Hunchensocket adds websockets! Web sockets are a weird little corner of any stack, but it’s nice to be able to have the option of being able to do this kind of programming. Also CL seems like a really good platform to do
make-hash makes constructing hash tables easier, which is sort of needlessly gawky otherwise.
ceramic provides bridges between CL and Electron for delivering desktop applications based on web technologies in CL.

I kept thinking that there wouldn’t be good examples of various things, (there’s a Kafka driver! there’s support for various other Apache ecosystem components,) but there are, and that’s great. There’s gaps, of course, but fewer, I think, than you’d expect.

The Dark Underbelly

The biggest problem in CL is probably discoverability: lots of folks are building great tools and it’s hard to really know about the projects.

I thought about phrasing this as a kind of list of things that would be good for supporting bounties or something of the like. Also if I’ve missed something, please let me know! I’ve tried to look for a lot of things, but discovery is hard.

Quibbles

rove doesn’t seem to work when multi-threaded results effectively. It’s listed in the readme, but I was able to write really trivial tests that crashed the test harness.
Chanl would be super lovely with some kind of concept of cancellation (like contexts in Go,) and while it’s nice to have a bit more thread introspection, given that the threads are somewhat heavier weight, being able to avoid resource leaks seems like a good plan.
There doesn’t seem to be any library capable of producing YAML formated data. I don’t have a specific need, but it’d be nice.
it would be nice to have some way of configuring the quicklisp client to be able to prefer quicklisp (stable) but also using ultralisp (or another source) if that’s available.
Putting the capacity in asdf to produce binaries easily is great, and the only thing missing from buildapp/cl-launch is multi-entry binaries. That’d be swell. It might also be easier as an alternative to have support for some git-style sub-commands in a commandline parser (which doesn’t easily exist at the moment'), but one-command-per-binary, seems difficult to manage.
there are no available implementations of a multi-reader single-writer mutex, which seems like an oversite, and yet, here we are.

Bigger Projects

There are no encoders/decoders for data formats like Apache Parquet, and the protocol buffers implementation don’t support proto3. Neither of these are particular deal breakers, but having good tools dealing with common developments, lowers to cost and risk of using CL in more applications.
No support for http/2 and therefore gRPC. Having the ability to write software in CL with the knowledge that it’ll be able to integrate with other components, is good for the ecosystem.
There is no great modern MongoDB driver. There were a couple of early implementations, but there are important changes to the MongoDB protocol. A clearer interface for producing BSON might be useful too.
I’ve looked for libraries and tools to integrate and manage aspects of things like systemd, docker, and k8s. k8s seems easiest to close, as things like cube can be generated from updated swagger definitions, but there’s less for the others.
Application delievery remains a bit of an open. I’m particularly interested in being able to produce binaries that target other platforms/systems (cross compilation,) but there are a class of problems related to being able to ship tools once built.
I’m eagerly waiting and concerned about the plight of the current implementations around the move of ARM to Darwin, in the intermediate term. My sense is that the transition won’t be super difficult, but it seems like a thing.

Pattern Fragment 2

Thu, 11 Feb 2021 00:00:00 +0000

This is the follow up to Pattern Fragment 0 and Pattern Fragment 1.

After the yoke decreases, in addition to the steeks, there should be 196 stitches in total, or 98 stitches on the front and back of the neck.

Knit the yoke section plain, until it is--in total--3 inches deep. On the front of the sweater, knit 49 stitches (half), cast on 10 steek stitches, and continue knitting round marking the stitches. Knit the next round plan, and then decrease one stitch on either side of the steek, every other round, 21 or 22 times to shape the neck (42 or 44 rounds). Knit plain from here to the end of the sweater. After the first 2-3 inches of decreases, you may choose to space out the decreases more for a gradual slope, though I wouldn’t.

Meanwhile, when the yoke is 7.5 inches deep, set aside at least 26 stitches in the middle of the back for back-of-neck-shaping, cast on a 10 stitch steak, and then decrease on alternating sides of the steek over the next inch and a half, until the number of stitches decreased at the front is exactly equal to the number of stitches decreased at the back.

When the yoke is 9 inches deep, in the last round bind off the middle two stitch of both of the armhole steeks, ending with knitting across the back one last time. Turn the work inside out and using a three-needle bind off, join and bind off the shoulders.

Knitting off the Cone

Wed, 10 Feb 2021 00:00:00 +0000

I have, for a long time, done rather a lot of knitting from yarn directly off of cones, which is maybe a bit weird or at least uncommon, so I thought I’d elaborate a bit more:

Theoretically a cone of yarn, which often contains at least 250 grams or more of yarn, has fewer breaks in it than you’d have with an equivalent weight of yarn packaged in skeins or balls. This isn’t always true, as cones of yarn do have breaks, sometimes, but if you have a construction that doesn’t require you break the yarn very much you can probably save a lot of weaving in by knitting off of a cone.
Cones of yarn are often not quite ready for use: most often the yarn hasn’t received its final wash, which often means that the spinning oil is still in the wool. This is potentially only true for yarn that’s undyed or dyed before being spun, and not the case for yarn that’s dyed after being spun. It’s also likely the case that the yarn will be wound onto the cone slightly tighter than it would be otherwise. The effect is that the yarn will be a bit limp relative to it’s final state. The color can also change a bit. You can knit with the unwashed yarn, but know that the final product will require a bit more washing, and the texture can change.
Typically the kind of yarn that’s available on cones is boring, which is to say that there are less varieties in general but also of different colors. I think this is actually a great thing: knitting in more plain colors and simple smooth yarns draws attention to the knitting itself, which is often my goal.
Cones of yarn feel like buying yarn in bulk, and buying yarn by the pound or kilo (!) means that you can really get a feel for the yarn and it’s behavior and knit with it for more than one project. Make a few sweaters, or many pairs of socks. See what happens!
Because yarn on cones is often used as a method of distributing undyed yarn to dyers in bulk, you can select materials on the basis of fiber content in a way that can be difficult when you also have to balance color considerations.

The clear solution to this problem is, of course, to wind the yarn off the cone into a hank (typically using a niddy nody or similar,) avoid tying the yarn too tightly, soak and wash the yarn gently with wool wash, and then hang it up to dry, and then wind it back into balls. I never do this. I should, but realistically I never do.

How to Write Performant Software and Why You Shouldn't

Tue, 09 Feb 2021 00:00:00 +0000

I said a thing on twitter that I like, and I realized that I hadn’t really written (or ranted) much about performance engineering, and it seemed like a good thing to do. Let’s get to it.

Making software fast is pretty easy:

Measure the performance of your software at two distinct levels:
- figure out how to isolate specific operations, as in unit test, and get the code to run many times, and measure how long the operations take.
- Run meaningful units of work, as in integration tests, to understand how the components of your system come together.
If you’re running a service, sometimes tracking the actual timing of actual operations over time, can also be useful, but you need a lot of traffic for this to be really useful. Run these measurements regularly, and track the timing of operations over time so you know when things actually chair.
When you notice something is slow, identify the slow thing and make it faster. This sounds silly, but the things that are slow usually fall into one of a few common cases:
- an operation that you expected to be quick and in memory, actually does something that does I/O (either to a disk or to the network,)
- an operation allocates more memory than you expect, or allocates memory more often than you expect.
- there’s a loop that takes more time than you expect, because you expected the number of iterations to be small (10?) and instead there are hundreds or thousands of operations.
Combine these and you can get some really weird effects, particularly over time. An operation that used to be quick gets slower over time, because the items iterated over grows, or a function is called in a loop that used to be an in-memory only operation, now accesses the database, or something like that. The memory based ones can be trickier (but also end up being less common, at least with more recent programming runtimes.)

Collect data, and when something gets slower you should fix it.

Well no.

Most of the time slow software doesn’t really matter. The appearance of slowness or fastness is rarely material to user’s experience or the bottom line. If software gets slower, most of the time you should just let it get slower:

Computers get faster and cheaper over time, so most of the time, as long as your rate of slow down is slow and steady over time, its usually fine to just ride it out. Obviously big slow downs are a problem, but a few percent year-over-year is so rarely worth fixing.

It’s also the case that runtimes and compilers are almost always getting faster, (because compiler devlopers are, in the best way possible, total nerds,) so upgrading the compiler/runtime regularly often offsets regular slow down over time.
In the vast majority of cases, the thing that makes software slow is doing I/O (disks or network,) and so your code probably doesn’t matter and so what your code does is unlikely to matter much and you can solve the problem by changing how traffic flows through your system.

For IX (e.g. web/front-end) code, the logic is a bit different, because slow code actually impacts user experience, and humans notice things. The solution here, though, is often not about making the code faster, but in increasingly pushing a lot of code to “the backend,” (e.g. avoid prossing data on the front end, and just make sure the backend can always hand you exactly the data you need and want.)
Code that’s fast is often harder to read and maintain: to make code faster, you often have to be careful and avoid using certain features of your programming language or runtime (e.g. avoiding ususing heap allocations, or reducing the size of allocations by encoding data in more terse ways, etc,) or by avoiding libraries that are “slower,” or that use certain abstractions, all of which makes your code less conventional more difficult to read, and harder to debug. Programmer time is almost always more expensive than compute time, so unless it’s big or causing problems, its rarely worth making code harder to read.

Sometimes, making things actually faster is actually required. Maybe you have a lot of data that you need to get through pretty quickly and there’s no way around it, or you have some classically difficult algorithm problem (graph search, say,), but in the course of generally building software this happens pretty rarely, and again most of the time pushing the problem “up” (from the front end to the backend and from the backend to the database, similar,) solves whatever problems you might have.
While there are obviously pathological counter-examples, ususally related to things that happen in loops, but a lot of operations never have to be fast because they sit right next to another operation that’s much slower:
- Lots of people analyze logging tools for speed, and this is almost always silly because all log messages either have to be written somewhere (I/O) and generally there has to be something to serialize messages (a mutex or functional equivalent,) somewhere because you want to write only one message at a time to the output, so even if you have a really “fast logger” on its own terms, you’re going to hit the I/O or the serializing nature of the problem. Use a logger that has the features you have and is easy to use, speed doesn’t matter.
- anything in HTTP request routing and processing. Because request processing sits next to network operations, often between a database as well as to the client, any sort of gain by using a “a faster web framework,” is probably immeasurable. Use the ones with the clearest feature set.

Pattern Fragment 1

Thu, 04 Feb 2021 00:00:00 +0000

This is the follow up to Pattern Fragment 0

After all of the shaping for the body of the sweater, you’ll have 256 stitches. The goal is to have 196 stitches total for the yoke section, or 98 stitches front and back. This gives me a yoke width of 14 inches, which I know fits me well. Your shoulder width may turn out to be deeply personal, modify as needed to accommodate your personal shoulders.

Put 14 stitches on holders at each underarm, this should be the 7 stitches before and after your round beginning and middle markers. Cast on 10 steek stitches using the backward loop (e-wrap) method above each steek. These are the underarms.

On the next row, after creating the steeks, decrease one body stitch into the first and last steek stitches, and continue these decreases in alternating rows, 7 times (14 total rows), until there are 98 stitches ready for the yoke.

The division between “stitches set aside” and “stitches decreased” at the beginning of the yoke are flexible, as long as you’ve finished shaping the yoke before its about 2 inches long.

Finished a Sweater

Wed, 03 Feb 2021 00:00:00 +0000

I finished knitting a sweater a bit ago, and it’s pretty cool. Some thoughts:

The cuffs ended up being a touch too wide, but it’s workable. I think this sweater is really good for wearing over an oxford, and as such slightly wider cuffs may be fine.
I used the placket / open neck line reminiscent of 1/4 zip sweaters, but chose for the first time to do garter stitch rather than ribbing for the horizontal parts of the plackets, which worked pretty well, though I might choose to execute them differently in the future. Having said that, I think I want to explore different neck shapes.
I didn’t do any kind of lower body shaping, which is fine, particularly on such a boxy garment, but waist shaping is a good thing that I’d use again in the future.
This is the first drop shouldered garment I’ve made since the knitting hiatus. It was comforting, but I suspect I’ll not knit another for quite a while.
I’d knit this sweater before using exactly these colors, albeit in a thicker weight yarn, and a few times with different color combinations. It was really fun and familiar.
HD Shetland Yarn is pretty awesome. This was the first time I’d used it for non-stranded knitting, and it was great fun to knit.
The previous couple of sweaters that I’d made were both knit at about 9 stitches to the inch, and this was about 7 stitches to the inch, which means that it felt like it went really fast. It’s wild how we acclimate to things.

Continuous Integration is Harder Than You Think

Tue, 02 Feb 2021 00:00:00 +0000

I’ve been working on continuous integration systems for a few years, and while the basic principle of CI is straightforward, it seems that most CI deployments are not. This makes sense: project infrastructure is an easy place to defer maintenance during the development cycle, and projects often prioritize feature development and bug fixing over tweaking the buildsystem or test infrastructure, but I almost think there’s something more. This post is a consideration of what makes CI hard and perhaps provide a bit of unsolicited advice.

The Case for CI

I suppose I don’t really have to sell anyone on the utility or power of CI: running a set of tests on your software regularly allows developers and teams to catch bugs early, and saves a bucket of developer time, and that is usually enough. Really, though, CI ends up giving you the leverage to solve a number of really gnarly engineering problems:

how to release software consistently and regularly.
how to support multiple platforms.
how to manage larger codebases.
anything with distributed systems.
how to develop software with larger numbers of contributors.

Doing any of these things without CI isn’t really particularly viable, particularly at scale. This isn’t to say, that they “come free” with CI, but that CI is often the right place to build the kind of infrastructure required to manage distributed systems problems or release complexity.

Buildsystems are Crucial

One thing that I see teams doing some times is addressing their local development processes and tooling with a different set of expectations than they do in CI, and you can totally see and understand how this happens: the CI processes always start from a clean environment, and you often want to handle failures in CI differently than you might handle a failure locally. It’s really easy to write a shell script that only runs in CI, and then things sort of accumulate, and eventually there emerge a class of features and phenomena that only exist for and because of CI.

The solution is simple: invest in your buildsystem,¹ and ensure that there is minimal (or no!) indirection between your buildsystem and your CI configuration. But buildsystems are hard, and in a lot of cases, test harnesses aren’t easily integrated into build systems, which complicates the problem for some. Having a good build system isn’t particularly about picking a good tool, though there are definitely tradeoffs for different tools, the problem is mostly in capturing logic in a consistent way, providing a good interface, and ensuring that the builds happen as efficiently as possible.

Regardless, I’m a strong believer in centralizing as much functionality in the buildsystem as possible and making sure that CI just calls into build systems. Good build systems:

allow you to build or rebuild (or test/subtest) only subsets of work, to allow quick iteration during development and debugging.
center around a model of artifacts (things produced) and dependencies (requires-type relationships between artifacts).
have clear defaults, automatically detect dependencies and information from the environment, and perform any required set up and teardown for the build and/or test.
provide a unified interface for the developer workflow, including building, testing, and packaging.

The upside, is that effort that you put into the development of a buildsystem pay dividends not just for managing to complexity of CI deployments, but also make the local development stable and approachable for new developers.

T-Shaped Matrices

There’s a temptation with CI systems to exercise your entire test suite with a comprehensive and complete range of platforms, modes, and operations. While this works great for some smaller projects, “completism” is not the best way to model the problem. When designing and selecting your tests and test dimensions, consider the following goals and approaches:

on one, and only one, platform run your entire test suite. This platform should probably be very close to the primary runtime of your environment (e.g. when developing a service that runs on Linux service, your tests should run in a system that resembles the production environment,) or possibly your primary development environment.
for all platforms other than your primary platform, run only the tests that are either directly related to that runtime/platform (e.g. anything that might be OS or processor specific,) plus some small subset of “verification” or acceptance tests. I would expect that these tests should easily be able to complete in 10% of the time of a “full build,”
consider operational variants (e.g. if your product has multiple major-runtime modes, or some kind of pluggable sub-system) and select the set of tests which verifies these modes of operations.

In general the shape of the matrix should be t-shaped, or “wide across” with a long “narrow down.” The danger more than anything is in running too many tests, which is a problem because:

more tests increase the chance of a false negative (caused by the underlying systems infrastructure, service dependencies, or even flakey tests,) which means you risk spending more time chasing down problems. Running tests that provide signal is good, but the chance of false negatives is a liability.
responsiveness of CI frameworks is important but incredibly difficult, and running fewer things can improve responsiveness. While parallelism might help some kinds of runtime limitations with larger numbers of tests, this incurs overhead, is expensive.
actual failures become redundant, and difficult to attribute failures in “complete matrices.” A test of certain high level systems may pass or fail consistently along all dimensions creating more noise when something fails. With any degree of non-determinism or chance of a false-negative, running tests more than once just make it difficult to attribute failures to a specific change or an intermittent bug.
some testing dimensions don’t make sense, leading to wasted time addressing test failures. For example when testing an RPC protocol library that supports both encryption and authentication, it’s not meaningful to test the combination of “no-encryption” and “authentication,” although the other three axes might be interesting.

The ultimate goal, of course is to have a test matrix that you are confident will catch bugs when they occur, is easy to maintain, and helps you build confidence in the software that you ship.

Conclusion

Many organizations have teams dedicated maintaining buildsystems and CI, and that’s often appropriate: keeping CI alive is of huge value. It’s also very possible for CI and related tools to accrued complexity and debt in ways that are difficult to maintain, even with dedicated teams: taking a step back and thinking about CI, buildsystems, and overall architecture strategically can be very powerful, and really improve the value provided by the system.

Canonically buildsystems are things like makefiles (or cmake, scons, waf, rake, npm, maven, ant, gradle, etc.) that are responsible for converting your source files into executable, but the lines get blurry in a lot of languages/projects. For Golang, the go tool plays the part of the buildsystem and test harness without much extra configuration, and many environments have a pretty robust separation between building and testing. ↩︎

Get More Done

Thu, 21 Jan 2021 00:00:00 +0000

It’s really easy to over think the way that we approach our work and manage our own time and projects. There are no shortage of tools, services, books, and methods to organizing your days and work, and while there are a lot of good ideas out there, it’s easy to get stuck fiddling with how you work, at the expense of actuallying getting work done. While I’ve definitely thought about this a lot over time, for a long time, I’ve mostly just done things and not really worried much about the things on my to-do list.¹

I think about the way that I work similarly to the way that I think about the way I work with other people. The way you work alone is different from collaboration, but a lot of the principles of thinking about big goals, and smaller actionable items is pretty transferable.

My suggestions here are centered around the idea that you have a todo list, and that you spend a few moments a day looking at that list, but actually I think the way I think about my work is really orthogonal to any specific tools. For years, most of my personal planning has revolved around making a few lists in a steno pad once or twice a day,² though I’ve been trying to do more digital things recently. I’m not sure I like it. Again, tools don’t matter.

Smaller Tasks are Always Better

It’s easy to plan projects from the “top down,” and identify the major components and plan your work around those components, and the times that I run in to trouble are always the times when my “actionable pieces” are too big. Smaller pieces help you build momentum, allow to move around to different areas as your attention and focus change, and help you use avalible time effectively (when you want.)

It’s easy to find time in-between meetings, or while the pasta water is boiling, to do something small and quick. It’s also very easy to avoid starting something big until you have a big block of unfettered time. The combination of these factors makes bigger tasks liabilities, and more likely to take even longer to complete.

Multi-Task Different Kinds of Work

I read a bunch of articles that suggest that the way to be really productive is to figure out ways of focusing and avoiding context switches. I’ve even watched a lot of coworkers organize their schedules and work around these principles, and it’s always been something of a mystery for me. It’s true that too much multi-tasking and context switching can lead to a fragmented experience and make some longer/complicated tasks harder to really dig into, but it’s possible to manage the costs of context switching, by breaking apart bigger projects into smaller projects and leaving notes for your (future) self as you work.

Even if you don’t do a lot of actual multitasking within a given hour or day of time, it’s hard to avoid really working on different kinds of projects on the scale of days or weeks, and I’ve found that having multiple projects in flight at once actually helps me get more done. In general I think of this as the idea that more projects in flight means that you finish things more often, even if the total number of projects completed is the same in the macro context.

Regardless, different stages of a project require different kind of attention and energy and having a few things in flight increases the chance that when you’re in the mood to do some research, or editing, or planning, you have a project with that kind of work all queued up. I prefer to be able to switch to different kinds of work depending on my attention and mood. In general my work falls into the following kinds of activities:

planning (e.g. splitting up big tasks, outlining, design work,)
generative work (e.g. writing, coding, etc.)
organizational (email, collaboration coordination, user support, public issue tracking, mentoring, integration, etc.)
polishing (editing, writing/running tests, publication prepping,)
reviewing (code review, editing, etc.)

Do the Right Things

My general approach is “do lots of things and hope something sticks,” which makes the small assumption that all of the things you do are important. It’s fine if not everything is the most important, and it’s fine to do things a bit out of order, but it’s probably a problem if you do lots of things without getting important things done.

So I’m not saying establish a priority for all tasks and execute them in strictly that priority, at all. Part of the problem is just making sure that the things on your list are still relevant, and still make sense. As we do work and time passes, we have to rethink or rechart how we’re going to complete a project, and that reevaluation is useful.

Prioritization and task selection is incredibly hard, and it’s easy to cast “prioritization” in over simplified terms. I’ve been thinking about prioritization, for my own work, as being a decision based on the following factors:

deadline (when does this have to be done: work on things that have hard deadlines or expected completion times, ordered by expected completion date, to avoid needing to cram at the last moment.)
potential impact (do things that will have the greatest impact before lesser impact, this is super subjective, but can help build momentum, and give you a chance to decide if lower-impact items are worth doing.)
time availability fit (do the biggest thing you can manage with the time you have at hand, as smaller things are easier to fit in later,)
level of understanding (work on the things that you understand the best, and give yourself the opportunity to plan things that you don’t understand later. I sometimes think about this as “do easy things first,” but that might be too simple.)
time outstanding (how long ago was this task created: do older things first to prevent them from becoming stale.)
number of things (or people) that depend on this being done (work on things that will unblock other tasks or collaborators before things that don’t have any dependencies, to help increase overall throughput.)

Maintain a Pipeline of Work

Productivity, for me, has always been about getting momentum on projects and being able to add more things. For work projects, there’s (almost) always a backlog of tasks, and the next thing is ususally pretty obvious, but sometimes this is harder for personal projects. I’ve noticed a tendency in myself to prefer “getting everything done” on my personal todo list, which I don’t think particularly useful. Having a pipleine of backlog of work is great:

there’s always something next to do, and there isn’t a moment when you’ve finished and have to think about new things.
keeping a list of things that you are going to do in the more distant future lets you start thinking about how bigger pieces fit together without needint to starting to work on that.
you can add big things to your list(s) and then break them into smaller pieces as you make progress.

As an experiment, think about your todo list, not as a thing that you’d like to finish all of the items, but as list that shouldn’t be shorter than a certain amount (say 20 or 30?) items with rate of completion (10 a week?) though you should choose your own numbers, and set goals based on what you see yourself getting done over time.

Though, to be clear, I’ve had the pleasure and benefit of working in an organization that lives-and-dies by a bug tracking system, with a great team of folks doing project management. So there are other people who manage sprints, keep an eye on velocity, and make sure that issues don’t get stuck. ↩︎
My general approach is typically to have a “big projects” or “things to think about” list and a “do this next list”, with occasional lists about all the things in a specific big project. In retrospect these map reasonable well to SCRUM/Agile concepts, but it also makes sense. ↩︎

Lessons from the Knitting Hiatus

Wed, 20 Jan 2021 00:00:00 +0000

I took years and years off of knitting: life and priorities change and I must confess that a few years of living in a very small apartment with very active cats made it difficult to have the space to really get into knitting. Anyway, it was really nice to have a hobby sitting on the metaphoric shelf that I could get right back into without a big learning curve.

The interesting thing, I think is in observing is that the hiatus made some subtle changes to the way that I approach knitting things, at least relative to what I remember.

I’m less opposed to garter stitch, and have been using little bits of it here and there in some projects.
I’ve gotten much better at wrap-and-turn short rows in stocking stitch., they now look pretty good and I remember them always looking terrible.
I knit the yoke of a sweater back and forth, which is a thing that I would have found unimaginable.
Knitting plain stocking stitch in the round has always been a great joy of mine, but in the last couple of months I’ve done it rather a lot, knitting 3, or so, plain sweaters, which I’ve found quite captivating. I always seemed to feel like I needed some kind of patterning (color work, lace, cables etc) to keep things interesting, and that doesn’t really seem to be the case.
I’ve yet to knit anything post-hiatus on needles other than US 0s (which are quite small,) and it doesn’t seem to bug me very much. I continue to make progress on projects and rounds with 250-340 (or so) stitches don’t seem oppressively long.
The problem of having little gaps between the sleeve of a sweater and the body at the “bottom corner,” always used to be a big problem, and these days I haven’t need to sew up these gaps at all, which is kind of novel.
My cast on edges have gotten better: I’ve managed to get edges that are as elastic as they need to be, and all of the usual problems (a twist, mistakes in rubbing, problems in counting, misjudging the length of the long-tail) haven’t been a problem at all.

Of course some things didn’t change:

I still don’t really like to do things that involve knitting rows very much, and would prefer to knit as much as possible in the round.
My taste in yarns seems to be heavy on the “boring fine wool” and while I’ve been looking around at the kinds of yarns that are available and popular, I am (for the most part,) pretty content to stick to the really simple and boring yarn.
I haven’t yet vanquished a number of old fears/struggles like making an EPS-style sweater that I really like, knitting sleeves from the cuff-up, cardigans. Many of these things are on my list of things to explore more in the future, but we’ll see how I fair.

It’s all very curious! I’m excited to see if anything else changes!

Open Source Emacs Configuration Improvements

Tue, 19 Jan 2021 00:00:00 +0000

In retrospect I’m not totally sure why I released my emacs configuration to the world. I find tweaking Emacs Lisp to be soothing, and in 2020 these kinds of projects are particularly welcome. I’ve always thought about making it public: I feel like I get a lot out of Emacs, and I’m super aware that it’s very hard for people who haven’t been using Emacs forever to get a comparable experience.¹

I also really had no idea of what to expect, and while it’s still really recent, I’ve noticed a few things which are worth remarking:

Making your code usable for other people really does make it easy for people to find bugs. While it’s likely that there are bugs that people never noticed, I found a few things very quickly:
- Someone reported higher than expected CPU use, and I discovered that there were a number of functions that ran regularly in timers, and I was able to quickly tune some knobs in order to reduce average CPU use by a lot. This is likely to be great both for the user in question, but also because it’ll help battery life.
- The config includes a git submodule (!) with the contents of all third-party packages, mostly to reduce friction for people getting started. Downloading all of the packages fresh from the archive would take a few minutes, and the git clone is just faster. I realized, when someone ran into some problems when running with emacs 28 (e.g. the development/mainline build,) that the byte-compilation formats were different, which made the emacs27 files not work on emacs28. I pushed a second branch.
More than anything the experience of getting bug reports and feedback has been great. It both makes it possible to focus time because the impact of the work is really clear, and it also makes it clear to me that I’ve accumulated some actually decent Emacs Lisp skills, without really noticing it.²
I was inspired to make a few structural improvements.
- For a long time, including after the initial release, I had a “settings” file, and a “local functions” file that held code that I’d written or coppied from one place or another, and I finally divided them all into packages named tychoish-<thing>.el which allowed me to put all or most of the configuration into use-package forms, which is more consistent and also helps startup time a bit, and makes the directory structure a bit easier.
- I also cleaned up a bunch of local snippets that I’d been carrying around, which wasn’t hurting anything but is a bit more clear in the present form.
I believe that I’ve hit the limit, with regards to startup speed. I’d really like to get a GUI emacs instance to start (with no buffers) in less than a second, but it doesn’t seem super plausible. I got really close. At this point there are two factors that constrain:
- Raw CPU speed. I have two computers, and the machine with the newer CPU is consistently 25% faster than the slow computer.
- While the default configuration doesn’t do this, my personal configuration sets a font (this is reasonable,) but seems that the time to do this is sometimes observable, and proportional to the number of fonts you have installed on the system.³
- Dependencies during the early load. I was able to save about 10% time by moving a function between package to reduce the packages that startup code depended upon. There’s just a limit to how much you can clean up here.
Having said that, these things can drift pretty easily. I’ve added some helper macros with-timer and with-slow-op-timer that I can use to report the timing of operations during startup to make sure that things don’t slow down.

Interestingly, I’ve experimented with byte-compiling my local configuration and I haven’t really noticed much of a speedup at this scale, so for ease I’ve been leaving my own lisp directory unbytecompiled.
With everything in order, there’s not much to edit! I guess I’ll have other things to work on, but I have made a few improvements, generally:
- Using the alert package for desktop notification, which allowed me to delete a legacy package I’ve been using. Deleting code is awesome.
- I finally figured out how to really take advantage of projectile, which is now configured correctly, and has been a lot of help in my day-to-day work.
- I’ve started using ERC more, and only really using my irssi (in screen) session as a fallback. My IRC/IM setup is a bit beyond the scope of this post but ERC has been a bit fussy to use on machines with intermittent connections, but I think I’ve been able to tweak that pretty well and have an experience that’s quite good.

It’s been interesting! And I’m looking forward to continuing to do this!

Sure, other editors also have long setup curves, but Emacs is particularly gnarly in this regard, and I think adoption by new programmers is definitely constrained by this fact. ↩︎
I never really thought of myself as someone who wrote Emacs Lisp: I’ve never really written a piece of software in Emacs, it’s always been a function here or there, or modifying some snippet from somewhere. I don’t know if I have a project or a goal that would involve writing more emacs software, but it’s nice to recognize that I’ve accidentally acquired a skill. ↩︎
On Windows and macOS systems this may not matter, but you may have more fonts installed than you need. I certianly did. Be aware that webbrowsers often downlaod their own fonts separately from system fonts, so having fonts installed is really relevant to your GTK/QT/UI use and not actually to the place where you’re likely doing most of your font interaction (e.g. the browser.) ↩︎

Knitting Pictures

Mon, 18 Jan 2021 00:00:00 +0000

I’ve never been really good at the blogging+picture game, and while maybe once upon a time it was technical limitation--taking photos and getting them online was complicated--anymore it’s probably not. To this end, I’ve started a knitting specific Instagram account as a kind of photoblog for knitting things. It’s @gestaltknitting, if you’re interested.

While I took this picture a while ago, I must confess that my knitting basically looks the same now.

The same, not because I’ve made no progress, but because sleeves take a while and it’s just plain knitting, so unless you have a very discerning eye, you might miss the details.

Indeed, I really want my next project to also have a lot of plain knitting with black yarn: I expect the photographs will be captivating. Perhaps it will be enjoyable for people to be able to spot the different patterns of embedded cat hair in the sweaters.

I get that knitting is visual for a lot of people, and I do like a smart looking sweater as much as the next guy, but I’ve always felt somewhat resistant to this view: knitting is about the process and the act more than it is about the product, and so the things that are most exciting aren’t the visuals.

While it’s gotten much easier to take high quality pictures, my intention for this book that I’ve been writing is that it mostly would not be a book with a lot of picture, though we’ll see: If anything, I suspect that diagrams and cartoons may be more effective for this kind of application.

Having said that, it’s nice to see what other people are knitting, and I like the way that the ephemeral nature of instagram stories make it less daunting to post in-progress updates on projects. So I’ve definitely been enjoying that.

We’ll see!

Learning Common Lisp Again

Mon, 18 Jan 2021 00:00:00 +0000

In a recent post I spoke about abandoning a previous project that had gone off the rails, and I’ve been doing more work in Common Lisp, and I wanted to report a bit more, with some recent developments. There’s a lot of writing about learning to program for the first time, and a fair amount of writing about lisp itself, neither are particularly relevant to me, and I suspect there may be others who might find themselves in a similar position in the future.

My Starting Point

I already know how to program, and have a decent understanding of how to build and connect software components. I’ve been writing a lot of Go (Lang) for the last 4 years, and wrote rather a lot of Python before that. I’m an emacs user, and I use a Common Lisp window manager, so I’ve always found myself writing little bits of lisp here and there, but it never quite felt like I could do anything of consequence in Lisp, despite thinking that Lisp is really cool and that I wanted to write more.

My goals and rational are reasonably simple:

I’m always building little tools to support the way that I use computers, nothing is particularly complex, but it’d enjoy being able to do this in CL rather than in other languages, mostly because I think it’d be nice to not do that in the same languages that I work in professionally.¹
Common Lisp is really cool, and I think it’d be good if it were more widely used, and I think by writing more of it and writing posts like this is probably the best way to make that happen.
Learning new things is always good, and I think having a personal project to learn something new will be a good way of stretching my self as a developer. Most of my development as a programmer has focused on
Common Lisp has a bunch of features that I really like in a programming language: real threads, easy to run/produce static binaries, (almost) reasonable encapsulation/isolation features.

On Learning

Knowing how to program makes learning how to program easier: broadly speaking programming languages are similar to each other, and if you have a good model for the kinds of constructs and abstractions that are common in software, then learning a new language is just about learning the new syntax and learning a bit more about new idioms and figuring out how different language features can make it easier to solve problems that have been difficult in other languages.

In a lot of ways, if you already feel confident and fluent in a programming language, learning a second language, is really about teaching yourself how to learn a new language, which you can then apply to all future languages as needed.

Except realistically, “third languages” aren’t super common: it’s hard to get to the same level of fluency that you have with earlier languages, and often we learn “third-and-later” languages are learned in the context of some existing code base or project4, so it’s hard to generalize our familiarity outside of that context.

It’s also the case that it’s often pretty easy to learn a language enough to be able to perform common or familiar tasks, but fluency is hard, particularly in different idioms. Using CL as an excuse to do kinds of programming that I have more limited experience with: web programming, GUI programming, using different kinds of databases.

My usual method for learning a new programming language is to write a program of moderate complexity and size but in a problem space that I know pretty well. This makes it possible to gain familiarity, and map concepts that I understand to new concepts, while working on a well understood project. In short, I’m left to focus exclusively on “how do I do this?” type-problems and not “is this possible,” or “what should I do?” type-problems.

Conclusion

The more I think about it, the more I realize that when we talk about “knowing a programming language,” inevitably linked to a specific kind of programming: the kind of Lisp that I’ve been writing has skewed toward the object oriented end of the lisp spectrum with less functional bits than perhaps average. I’m also still a bit green when it comes to macros.

There are kinds of programs that I don’t really have much experience writing:

GUI things,
the front-half of the web stack,²
processing/working with ASTs, (lint tools, etc.)
lower-level kind of runtime implementation.

There’s lots of new things to learn, and new areas to explore!

Notes

There are a few reasons for this. Mostly, I think in a lot of cases, it’s right to choose programming languages that are well known (Python, Java+JVM friends, and JavaScript), easy to learn (Go), and fit in with existing ecosystems (which vary a bit by domain,) so while it might the be right choice it’s a bit limiting. It’s also the case that putting some boundaries/context switching between personal projects and work projects could be helpful in improving quality of life. ↩︎
Because it’s 2020, I’ve done a lot of work on “web apps,” but most of my work has been focused on areas of applications including including data layer, application architecture, and core business logic, and reliability/observability areas, and less with anything material to rendering web-pages. Most projects have a lot of work to be done, and I have no real regrets, but it does mean there’s plenty to learn. I wrote an earlier post about the problems of the concept of “full-stack engineering” which feels relevant. ↩︎

Pattern Fragment 0

Fri, 15 Jan 2021 00:00:00 +0000

I was doing some knitting pattern math,¹ and I thought I’d share it without a lot of context:

Cast on 228 stitches using the “German Twisted” method,², placing a marker half way, after 114 stitches. Knit 2 inches of knit 1 purl 1 ribbing.

After two inches, switch to stocking stitch: knit 21 stitches, increase 1 stitch, place a marker, knit 72 stitches, place a marker, increase 1 stitch, knit 21 more stitches. You should have arrived at the “half way” marker from before.

Over the next half (115 stitches), space out 14 increases. This doesn’t divide evenly, so try: knit 5, increase 1 stitch and then knit 8 stitches, increase 1 stitch 13 times, or in short hand: K5 M1, * K8 M1, repeat from * 13 times, K5).

The “first half” is the back of the sweater and the “second” half is the front. Increase one stitch before and after the markers on the back of the sweater 7 times, every 1.5 or 2 inches (somewhere between 10 or 20 rows,) depending on how you’d like the taper.

Meanwhile³ insert 3 sets of short rows across the back of the sweater, which should get wider. For the first short row stop 3 inches from the edges, for the second 2 inches, and for the last 1 inch. I’d put an inch or two between each short row, maybe half way between the first three increases.

Notes

I’ve not, to be clear, actually knit this yet, though I plan to soon. ↩︎
As in this video, though there are many videos that may be more clear for you. I’m pretty sure that learned this method from Meg Swansen and/or Amy Detjin. ↩︎
I have to say, that the “meanwhile” part of knitting patterns is always my favorite. ↩︎

Reknitting Projects

Thu, 14 Jan 2021 00:00:00 +0000

I’m presently in the middle of knitting a sweater that I knit and designed years and years ago, with only minor modifications, and I have a number of projects that I’m thinking about that involve “reknitting” past projects. While I don’t think that I’ve peaked, or am out of ideas for knitting, it’s very clear to me that novelty isn’t exactly my guiding principle as a knitter: I enjoy the process and the act above all else, and the pleasure of wearing handknits is (for me) mostly about custom fit and less about novelty or fashion, exactly.

The chance to re-knit things, removes a lot of the questions of a design from the process and not only fix mistakes, but also polish and iterate on a garment with less guess work. It’s also the case that these projects often feel like returning to an old friend, which is incredibly comforting. Some of these projects, on my backlog include:

This basic two-color sweaters (colorblock, I suppose,) that I’m presently knitting and have/will knit again where the lower part of the body is in black--or similar very dark--except for the top 3-4 inches of the body in a contrasting color, matched by the sleeves and the collar, which I try and push into the black section.
Alice Starmore’s Faroe Sweater, from Fishermen’s Sweaters, but scaled to actually fit and maybe with a more fitted shoulder. I’ve also, apparently knit a very heavy weight version of the Norway sweater that I never wore, and they’re such great classic designs that are very fun to knit that knitting them again to modernize them sounds like a fun project.
A round pi shawl in a dark color, with no lace work (including using raised bar increases rather than yarn overs), and a contrasing set of stripes along the outer edge. There’s this stripe pattern that I think of as “Calvin Klein” stripes, but I don’t kno what the origin of that association is, the basic plan is three stripes, two wide stripes in the contrasting color, and a thin stripe of the original color in between, with the wide stripes being 3 times the width of the interior stripe.
I’ve knit two sweaters from Joyce Willams' Latvian Dreams book, the sweater on the cover and one that I knit from several charts, using yarn that ended up pilling a lot. They were delightful to knit: the patterns were originally weaving charts rather than knitting patterns, and thus had a 4-way radial summary symmetry that was just fun to knit. I’d like to try some of these again with better yarn and perhaps use this as a space to explore color work again, but in ways that might be more subtle and also well suited to cardigans and the like.
I’ve knit a handful of sweaters with all-over mitten or stocking patterns from various extant knitting traditions, mostly Scandinavian and Turkish, and I think it would be fun to revisit these patterns.

For and Against Garter Stitch

Wed, 13 Jan 2021 00:00:00 +0000

I never used to like garter stitch¹ very much, and hadn’t really knit things with a lot of garter stitch. Sure, a scarf here or there in the beginning, and I think I used it for the hem of an early sweater that didn’t turn out particularly well. There are so many clever patterns that use a lot of garter stitch, and I’d never really felt it. While I don’t know that I’m rushing to knit or design patterns out of a lot of garter stitch, I’ve definitely discovered that I’ve softened on it over my hiatus.

My earlier discontent with garter stitch was the combination of:

garter stitch is quite dense, because the fabric pulls in so much vertically, so it takes a lot of yarn and a lot of time, and results in a warmer fabric that I often don’t like'
the vertical pull in of the fabric can get pulled out by blocking or by the weight of the fabric which can be rather uneven.
normal tension irregularity is super apparent.
I’ve never much liked the way that knitting things with rows require you to flip the knitting and I don’t like the way that this can break up the rhythm of the knitting.
the strong horizontal line of the garter ridges always feels awkward to work with.
I always struggled to get a selvage edge that I really liked that wasn’t totally sloppy.

These, however, are tractable problems I realized, and I’ve always used a few garter stitches for selvage on the edge of sock heel flaps. The things that I’ve realized:

garter stitch often works best with very fine yarn, which helps ameliorate the additional bulk, and at least for me, helps provide for more even tension.
the look of garter stitch sideways is quite compelling, for me, and in most cases it won’t stretch out in the same way.
a little bit goes a long way, particularly when embedded in another piece of knitting.
I’ve settled down and find that knitting, rather than slipping, the first stitch and giving the yarn a slight tug when knitting the second stitch leads to a pretty clean edge.
designing with garter stitch is quite compelling, because the ratio of stitches to rows is basically 2:1, because of the way the ridges pull in, you can sort of approach it as “square,” picking up one stitch for every garter ridge lays very flat, so the math is never very complicated.

I’m working on a hat where I knit a ~2 inch wide garter stitch strip to fit around my head and then picked up to knit the crown of the hat along one of the sides of the strip, and along the other to knit a lining. I could have used a provisional cast on, of course, but the strip allowed me to be more confident about sizing, and it ends up being pretty sharp.

I’m not sure I’m going to plan to knit things out of primarily garter stitch, but I’ve definitely softened rather a lot.

The fabric that results from knitting all stitches on both the front and back of the fabric. The fabric is dense, and it grows slowly, because the “ridges” account for two rows of knitting and it pulls in rather a lot. ↩︎

Sweater Measurements

Tue, 12 Jan 2021 00:00:00 +0000

Hand knitting provides the opportunity to customize sizing and shaping to fit your body (or that of whomever you’re knitting for,) and it’s possible to produce garments that really fit, but even though it’s possible it’s not always easy.

First, measuring a body directly is complicated:

posture impacts the measurements, and it’s difficult to get measurements of the body in the kinds of shapes and positions that you’re likely to hold while wearing the garment.
ease, or the difference between the actual measurement of your body and the actual measurement of the garment, is both subjective and a matter of preference.

For this reason, I normally recommend measuring another sweater that has a fit that you enjoy as a starting point, but there are challenges:

measurements for different styles of sweaters can have different internal proportions: the length of the sleeve depends on the width of the shoulders, and the depth of the armhole
most machine produced garments and conventional knitting patterns are based on typical measurements and proportions which are good as starting points but typically leave something to be desired.

While people’s measurements are broadly similar, and proportional, they’re not the same, so if you have slightly longer arms or shoulders that are a bit more broad or angular, the “average” might be off by an inch or two, which might be enough to care about.

I’d still recommend starting from a garment that you know fits well, and record the garment’s measurements as clearly as possible, but also note modifications separately. The basic idea is lay the garment out as flat as possible and measure the garment which is less likely to move than a person. There are three or four measurements that are really critical:

width of body at across the chest below the arms.
width of the body at the bottom hem/edge.
distance from the middle of the back of the neck to the cuff.
length of the sweater from the top of the shoulder to the bottom hem.

Sleeve length is pretty stable when measured from the bottom of the sleeve (where it joins the body at the underarm) to the cuff, as this avoids the impact of shoulder shape on the sleeve. Measuring arm length from a common point, the middle back of the neck, to the cuff is also a stable way to take this measurement. You may also require additional measurement’s if you want the body of the garment to have contores.

While it’s true that you can deduce other measurements from the four basic measurements, there are other fit considerations that are worth noting: width of the sleeve at/above the cuff and at the shoulder; depth, height, and aperture of the collar; as well as “true” shoulder width. May of these details I’ve figured out empirically and iteratively for myself: it’s sometimes difficult to get these measurements correctly from a model garment.

Better Company

Fri, 08 Jan 2021 00:00:00 +0000

I’ve been using company mode, which is a great completion framework for years now, and in genreal, it’s phenomenal. For a while, however, I’ve had the feeling that I’m not getting completion options at exactly the right frequency that I’d like. And a completion framework that’s a bit sluggish or that won’t offer completions that you’d expect is a drag. I dug in a bit, and got a much better, because of some poor configuration choices I’d made, and I thought I write up my configuration.

Backend Configuration

Company allows for configurable backends, which are just functions that provide completions, many of which are provided in the main company package, but also provided by many third (fourth?) party packages. These backends, then, are in a list which is stored in the company-backends, that company uses to try and find completions. When you get to a moment when you might want to complete things, emacs+company iterate through this list and build a list of expansions. This is pretty straight forward, at least in principle.

Now company is pretty good at making these backends fast, or trying, particularly when the backend might be irrelevant to whatever you’re currently editing--except in some special cases--but it means that the order of things in the list matters sometimes. The convention for configuring company backends is to load the module that provides the backend and then push the new backend onto the list. This mostly works fine, but there are some backends that either aren’t very fast or have the effect of blocking backends that come later (because they’re theoretically applicable to all modes.) These backends to be careful of are: company-yasnippet, company-ispell, and company-dabbrev.

I’ve never really gotten company-ispell to work (you have to configure a wordlist,) and I’ve never been a dabbrev user, but I’ve definitely made the mistake to putting the snippet expansion near the front of the list rather than the end. I’ve been tweaking things recently, and have settled on the following value for company-backends: :

(setq company-backends '(company-capf
             company-keywords
             company-semantic
             company-files
             company-etags
             company-elisp
             company-clang
             company-irony-c-headers
             company-irony
             company-jedi
             company-cmake
             company-ispell
             company-yasnippet))

The main caveat is that everything has to be loaded or have autoloads registered appropriately, particularly for things like jedi (python,) clang, and irony. The “capf” backend is the integration with emacs' default completion-at-point facility, and is the main mechanism by which lap-mode interacts with company, so it’s good to keep that at the top.

Make it Fast

I think there’s some fear that a completion framework like company could impact the perceived responsiveness of emacs as a whole, and as a result there are a couple of knobs for how to tweak things. Having said that, I’ve always run things more aggressively, because I like seeing possible completions fast, and I’ve never seen any real impact on apparent performance or battery utilization. use these settings: :

(setq company-tooltip-limit 20)
(setq company-show-numbers t)
(setq company-idle-delay 0)
(setq company-echo-delay 0)

Configure Prompts

To be honest, I mostly like the default popup, but it’s nice to be able to look at more completions and spill over to helm when needed. It’s a sometimes thing, but it’s quite nice: :

(use-package helm-company
   :ensure t
   :after (helm company)
   :bind (("C-c C-;" . helm-company))
   :commands (helm-company)
   :init
   (define-key company-mode-map (kbd "C-;") 'helm-company)
   (define-key company-active-map (kbd "C-;") 'helm-company))

Full Configuration

Some of the following is duplicated above, but here’s the full configuration that I run with: :

(use-package company
  :ensure t
  :delight
  :bind (("C-c ." . company-complete)
         ("C-c C-." . company-complete)
         ("C-c s s" . company-yasnippet)
         :map company-active-map
         ("C-n" . company-select-next)
     ("C-p" . company-select-previous)
     ("C-d" . company-show-doc-buffer)
     ("M-." . company-show-location))
  :init
  (add-hook 'c-mode-common-hook 'company-mode)
  (add-hook 'sgml-mode-hook 'company-mode)
  (add-hook 'emacs-lisp-mode-hook 'company-mode)
  (add-hook 'text-mode-hook 'company-mode)
  (add-hook 'lisp-mode-hook 'company-mode)
  :config
  (eval-after-load 'c-mode
    '(define-key c-mode-map (kbd "[tab]") 'company-complete))

  (setq company-tooltip-limit 20)
  (setq company-show-numbers t)
  (setq company-dabbrev-downcase nil)
  (setq company-idle-delay 0)
  (setq company-echo-delay 0)
  (setq company-ispell-dictionary (f-join tychoish-config-path "aspell-pws"))

  (setq company-backends '(company-capf
               company-keywords
               company-semantic
               company-files
               company-etags
               company-elisp
               company-clang
               company-irony-c-headers
               company-irony
               company-jedi
               company-cmake
               company-ispell
               company-yasnippet))

  (global-company-mode))

(use-package company-quickhelp
  :after company
  :config
  (setq company-quickhelp-idle-delay 0.1)
  (company-quickhelp-mode 1))

(use-package company-irony
  :ensure t
  :after (company irony)
  :commands (company-irony)
  :config
  (add-hook 'irony-mode-hook 'company-irony-setup-begin-commands))

(use-package company-irony-c-headers
  :ensure t
  :commands (company-irony-c-headers)
  :after company-irony)

(use-package company-jedi
  :ensure t
  :commands (company-jedi)
  :after (company python-mode))

(use-package company-statistics
  :ensure t
  :after company
  :config
  (company-statistics-mode))

Distributed Systems Problems and Strategies

Thu, 07 Jan 2021 00:00:00 +0000

At a certain scale, most applications end up having to contend with a class of “distributed systems” problems: when a single computer or a single copy of an application can’t support the required throughput of an application there’s not much to do except to distribute it, and therein lies the problem. Taking one of a thing and making many of the thing operate similarly can be really fascinating, and frankly empowering. At some point, all systems become distributed in some way, to a greater or lesser extent. While the underlying problems and strategies are simple enough, distributed systems-type bugs can be gnarly and having some framework for thinking about these kinds of systems and architectures can be useful, or even essential, when writing any kind of software.

Concerns

Application State

Applications all have some kind of internal state: configuration, runtime settings, in addition to whatever happens in memory as a result of running the application. When you have more than one copy of a single logical application, you have to put state somewhere. That somewhere is usually a database, but it can be another service or in some kind of shared file resource (e.g. NFS or blob storage like S3.)

The challenge is not “where to put the state,” because it probably doesn’t matter much, but rather in organizing the application to remove the assumption that state can be stored in the application. This often means avoiding caching data in global variables and avoiding storing data locally on the filesystem, but there are a host of ways in which application state can get stuck or captured, and the fix is generally “ensure this data is always read out of some centralized and authoritative service,” and ensure that any locally cached data is refreshed regularly and saved centrally when needed.

In general, better state management within applications makes code better regardless of how distributed the system is, and when we use the “turn it off and turn it back on,” we’re really just clearing out some bit of application state that’s gotten stuck during the runtime of a piece of software.

Startup and Shutdown

Process creation and initialization, as well as shutdown, is difficult in distributed systems. While most configuration and state is probably stored in some remote service (like a database,) there’s a bootstrapping process where each process gets enough local configuration required to get that configuration and startup from the central service, which can be a bit delicate.

Shutdown has its own problems set of problems, as specific processes need to be able to complete or safely abort in progress operations.

For request driven work (i.e. HTTP or RPC APIs) without statefull or long-running requests (e.g. many websockets and most streaming connections), applications have to stop accepting new connections and let all in progress requests complete before terminating. For other kinds of work, the process has to either complete in progress work or provide some kind of “checkpointing” approach so that another process can pick up the work later.

Horizontal Scalability

Horizontal scalability, being able to increase the capability of an application by adding more instances of the application rather than creasing the resources allotted to the application itself, is one of the reasons that we build distributed systems in the first place,¹ but simply being able to run multiple copies of the application at once isn’t always enough, the application needs to be able to effectively distribute it’s workloads. For request driven work this is genreally some kind of load balancing layer or strategy, and for other kinds of workloads you need some way to distribute work across the application.

There are lots of different ways to provide loadbalancing, and a lot depends on your application and clients, there is specialized software (and even hardware) that provides loadbalancing by sitting “in front of” the application and routing requests to a backend, but there are also a collection of client-side solutions that work quite well. The complexity of load balancing solutions varies a lot: there are some approaches that just distribute responses “evenly” (by number) to a single backend one-by-one (“round-robin”) and some approaches that attempt to distribute requests more “fairly” based on some reporting of each backend or an analysis of the incoming requests, and the strategy here depends a lot on the requirements of the application or service.

For workloads that aren’t request driven, systems require some mechanism of distributing work to workers, ususally with some kind of messaging system, though it’s possible to get pretty far using a just a normal general purpose database to store pending work. The options for managing, ordering, and distributing the work, is the meat of problem.

Challenges

When thinking about system design or architecture, I tend to start with the following questions.

how does the system handle intermittent failures of particular components?
what kind of downtime is acceptable for any given component? for the system as a whole?
how do operations timeout and get terminated, and how to clients handle these kinds of failures?
what are the tolerances for the application in terms of latency of various kinds of operations, and also the tolerances for “missing” or “duplicating” an operation?
when (any single) node or component of the system aborts or restarts abruptly, how does the application/service respond? Does work resume or abort safely?
what level of manual intervention is acceptable? Does the system need to node failure autonomously? If so how many nodes?

Concepts like “node” or “component” or “operation,” can mean different things in different systems, and I use the terms somewhat vaguely as a result. These general factors and questions apply to systems that have monolithic architectures (i.e. many copies of a single type of process which performs many functions,) and service-based architectures (i.e. many different processes performing specialized functions.)

Solutions

Ignore the Problem, For Now

Many applications run in a distributed fashion while only really addressing parts of their relevant distributed systems problems, and in practice it works out ok. Applications may store most of their data in a database, but have some configuration files that are stored locally: this is annoying, and sometimes an out-of-sync file can lead to some unexpected behavior. Applications may have distributed application servers for all request-driven workloads, but may still have a separate single process that does some kind of coordinated background work, or run cron jobs.

Ignoring the problem isn’t always the best solution in the long term, but making sure that everything is distributed (or able to be distributed,) isn’t always the best use of time, and depending the specific application it works out fine. The important part, isn’t always to distribute things in all cases, but to make it possible to distribute functions in response to needs: in some ways I think about this as the “just in time” approach.

Federation

Federated architectures manage distributed systems protocols at a higher level: rather than assembling a large distributed system, build very small systems that can communicate at a high level using some kind of established protocol. The best example of a federated system is probably email, though there are others.²

Federated systems have more complex protocols that have to be specification based, which can be complicated/difficult to build. Also, federated services have to maintain the ability to interoperate with previous versions and even sometimes non-compliant services, which can be difficult to maintain. Federated systems also end up pushing a lot of the user experience into the clients, which can make it hard to control this aspect of the system.

On the upside, specific implementations and instances of a federated service can be quite simple and have straight forward and lightweight implementations. Supporting email for a few users (or even a few hundred) is a much more tractable problem than supporting email for many millions of users.

Distributed Locks

Needing some kind of lock (for mutual exclusion or mutex) is common enough in programming, and provide some kind of easy way to ensure that only a single actor has access to a specific resource. Doing this within a single process involves using kernel (futexes) or programming language runtime implementations, and is simple to conceptualize, and while the concept in a distributed system is functionally the same, the implementation of distributed locks are more complicated and necessarily slower (both the lock themselves, and their impact on the system as a whole).

All locks, local or distributed can be difficult to use correctly: the lock must be acquired before using the resource, and it must fully protect the resource, without protecting too much and having a large portion of functionality require the lock. So while locks are required sometimes, and conceptually simple, using them correctly is hard. With that disclaimer, to work, distributed locks require:³

some concept of an owner, which must be sufficiently specific (hostname, process identifier,) but that should be sufficiently unique to protect against process restarts, host renaming and collision.
lock status (locked/link) and if the lock has different modes, such as a multi-reader/single-writer lock, then that status.
a timeout or similar mechanism to prevent deadlocks if the actor holding a lock halts or becomes inaccessible, the lock is eventually released.
versioning, to prevent stale actors from modifying the same lock. In the case that actor-1 has a lock and stalls for longer than the timeout period, such that actor-2 gains the lock, when actor-1 runs again it must know that its been usurped.

Not all distributed systems require distributed locks, and in most cases, transactions in the data layer, provide most of the isolation that you might need from a distributed lock, but it’s a useful concept to have.

Duplicate Work (Idempotency)

For a lot of operations, in big systems, duplicating some work is easier and ultimately faster than coordinating and isolating that work in a single location. For this, having idempotent operations⁴ is useful. Some kinds of operations and systems make idempotency easier to implement, and in cases where the work is not idempotent (e.g. as in data processing or transformation,) the operation can be, by attaching some kind of clock to the data or operation.⁵

Using clocks and idempotency makes it possible to maintain data consistency without locks. At the same time, some of the same considerations apply. Having all operations duplicated is difficult to scale so having ways for operations to abort early can be useful.

Consensus Protocols

Some operations can’t be effectively distributed, but are also not safe to duplicate. Applications can use consensus protocols to do “leader election,” to ensure that there’s only one node “in charge” at a time, and the protocol. This is common in database systems, where “single leader” systems are useful for balancing write performance in distributed context. Consensus protocols have some amount of overhead, and are good for systems of a small to moderate size, because all elements of the system must communicate with all other nodes in the system.

The two prevailing consensus protocols are Paxos and Raft--pardoning the oversimplification here--with Raft being a simpler and easier to implement imagination of the same underlying principles. I’ve characterized consensus as being about leader election, though you can use these protocols to allow a distributed system to reach agreement on any manner of operations or shared state.

Queues

Building a fully generalized distributed application with consensus is a very lofty proposition, and commonly beyond the scope of most applications. If you can characterize the work of your system as discrete units of work (tasks or jobs,) and can build or access a queue mechanism within your application that supports workers on multiple processes, this might be enough to support a great deal of your distributed requirements for the application.

Once you have reliable mechanisms and abstractions for distributing work to a queue, scaling the system can be managed outside of the application by using different backing systems, or changing the dispatching layer, and queue optimization is pretty well understood. There are lots of different ways to schedule and distribute queued work, but perhaps this is beyond the scope of this article.

I wrote one of these, amboy, but things like gearman and celery do this as well, and many of these tools are built on messaging systems like Kafka or AMPQ, or just use general purpose databases as a backend. Keeping a solid abstraction between the applications queue and then messaging system seems good, but a lot depends on your application’s workload.

Delegate to Upstream Services

While there are distributed system problems that applications must solve for themselves, in most cases no solution is required! In practice many applications centralize a lot of their concerns in trusted systems like databases, messaging systems, or lock servers. This is probably correct! While distributed systems are required in most senses, distributed systems themselves are rarely the core feature of an application, and it makes sense to delegate these problem to services that that are focused on solving this problem.

While multiple external services can increase the overall operational complexity of the application, implementing your own distributed system fundamentals can be quite expensive (in terms of developer time), and error prone, so it’s generally a reasonable trade off.

Conclusion

I hope this was as useful for you all as it has been fun for me to write!

In most cases, some increase in reliability, by adding redundancy is a strong secondary motivation. ↩︎
xmpp , the protocol behind jabber which powered/powers many IM systems is another federated example, and the fediverse points to others. I also suspect that some federation-like features will be used at the infrastructure layer to coordinate between constrained elements (e.g. multiple k8s clusters will use federation for coordination, and maybe multi-cloud/multi-region orchestration as well…) ↩︎
This article about distributed locks in redis was helpful in summarizing the principles for me. ↩︎
An operation is idempotent if it can be performed more than once without changing the outcome. For instance, the operation “increment the value by 10” is not idempotent because it increments a value every time it runs, so running the operation once is different than running it twice. At the same time the operation “set the value to 10” is idempotent, because the value is always 10 at the end of the operation. ↩︎
Clocks can take the form of a “last modified timestamp,” or some kind of versioning integer associated with a record. Operations can check their local state against a canonical record, and abort if their data is out of date. ↩︎

Yarn Thoughts: HD Shetland

Tue, 05 Jan 2021 00:00:00 +0000

I’ve been knitting a sweater out of HD (Harrisville Designs) Shetland yarn for the past week or so and it’s been great, but there’s not a lot to look at because it’s just a plain sweater in black yarn, but I thought I’d write a bit about the experience.

I’ve knit a lot out of this yarn, mostly in stranded color work, and it’s probably the yarn that I have the most of in my possesion, but I’ve never really used it alone until recently, and hadn’t really knit anything with it in years. I’m a bit more than half way through a plain sweater in this yarn, and I find myself entranced.

It’s a simple 2-ply yarn, woolen spun, dyed before spinning, and it comes in hanks (which I’ve never used,) and on half pound cones. In color work, I tend to get 8 or 8.5 stitches to the inch (US 2.5/3mm), against a plain 7 stitches to the inch (US 0/2mm), and the fabric is light but solid. There are a bunch of colors, which is why I started using it for color work, including a number of heathers as well as natural colors. I would by a pound (2 cones) of each color to make a stranded sweater, but I always ended up with a lot of left overs. A plain sweater (for me) is under a pound, though I expect fewer left overs.

The name “Shetland” describes the weight, not the fiber contribution: the wool is a blend of unspecified breeds (probably some collection of Corriedale, another Merino cross, and/or Merino), but the effect is quite similar to actual Shetland Wool. While the wool is imported, the Mill is in New England, and the yarn is stocked by many yarn stores that supply weavers (though you can buy directly from the mill as well.) There’s something classic about the yarn: it smells like wool (probably the spinning oil, but still,) and the way that the fibers cling to each other makes it a jot to knit with.

HD Shetland isn’t exactly soft, but it isn’t rough either. I think part of this is about expectation management: because we know that this isn’t going to be yarn to wear against more sensitive skin (wrists, etc.), the fact that it’s actually pretty soft is a pleasant surprise. I also think that because the yarn is lofty and woolen spun the ends of the individual fibers end up less likely to be irritating or trigger reactions in the same way that smoother yarns can.

Conclusion: heartily recommend!

Epic Knitting

Mon, 04 Jan 2021 00:00:00 +0000

One of the things I’ve realized about myself is a knitter is that I really like epic projects, or sequences of projects.

Before my knitting hiatus, I bought a kilo of undyed sock yarn--because I liked the fiber content, and I wanted to explore knitting sweaters at finer gauges, because heavier weight yarns always made sweaters that felt too warm. Seemed like a good project. Due to a sizing error, I was only really able to get 2 sweaters out of the cone of yarn, but I think in the future three sweaters from a kilo seems useful. There’s something epic about this as a project, and I definitely intend to knit more sweaters in this vein. I was thinking about “epic knitting projects,” and came up with the following ideas:

10 identical socks: After many years of having a somewhat hodgepodge and recently well worn collection of socks, this year, I purchased 14 pairs of new socks and it’s been great, both because nothing is teetering on the edge of being worn out, and I decided to buy 5 pairs of two different kinds of socks that I like a lot, plus 4 of a heavier weight, and I like that every day is a “good sock” day. I also like that having a bunch of identical socks socks makes it easier to wash and pair them up. I’ve never really knit more than one pair of socks that matched so it seems like a fun challenge.
Knit all of Elizabeth Zimmerman’s Yoke Sweater shaping variations. I think I’ve made about three of them, and none of these yoke sweaters have really ever entered more regular rotation. I think knitting them in lighter weights, and reducing the yoke depth a bit to fit more could help a lot.
Knit a collection of sweater’s inspired by classic Alice Starmore patterns, but modified for modern sensibilities in terms of fit and shaping. I’ve knit three patterns (Henry VIII, Norway, and Faroe) and would gladly knit them all again and I’d love to try the cabled sweaters as well. While the designs and stitch patterns of these sweaters are compelling the fit is not, which is probably a feature of these patterns being written in the late 80s and early 90s. I think the changes would be mostly to knit things at a finer gauge (Faroe, Norway, and the cabled sweaters,) but also to modify the shaping for better fits at the shoulder, and maybe modify neck shape for v-necks or open necklines.

I’ve also thought about knitting a bunch of lace, but I lack the floor space in my current apartment to actually block any kind of lace shawl reasonably.

Anyone else have Epic-scale knitting projects for this list?

Pandemic New Years

Fri, 01 Jan 2021 00:00:00 +0000

It seems obligatory to mark the new year, and what a year it was, for me and everyone else: pandemic, quarantine, changing jobs, new hobbies, totally different routines. The backdrop of the pandemic makes looking back (and forward!) so weird, and I find myself asking: will the things that changed in my life still be true when there’s less quarantine? did the things that happen in 2020 happen because of the pandemic or would they have happened anyway? The certainty that most (but not all!) of 2021 will, in practical terms, look a lot like 2020 is intense and makes this whole “obligatory new years” thing a bit harder. Instead of doing something like “resolutions”/“goals”--which are always fraught--or some kind of lofty synthetic review of the last year, I think I’d like to muse on features of 2020 that I hope and expect to continue in 2021

Knitting: I took a lot of time off of knitting, but I found that I’d been missing it, and I think there are ways that it fits well into my life quarantine or no. It’s also been quite fun to write about knitting and knitting projects, and become more engaged with other knitters, and I look forward to knitting things for friends and family. I have list in my head of some nifty ideas for sweaters and some other things to knit, so I expect this to stay.
Blogging: I’ve been writing blog posts for years and while I’ve always found it rewarding, but everything else related to blogging has been hard. During the summer, while I was interviewing for jobs, I wrote blog posts most days, and sort of fell down on posting them, and still have 25 (or so) in the draft folder. Writing is the easy part, it’s editing (or letting go!) promotion, and remembering to move things out of draft. I’ve spruced up the blog and done some work to automate regular publishing, and it seems like I might be able to make this work! I definitely want to.
Pickling: I’ve really enjoyed pickling things, and if anything I expect that I’ll enjoy doing this more after quarnainte when it’s easier to share these things. Right now I have some cranberries on the go, and a lot of sauerkraut in the fridge that I’m slowly eating. It’ll be fun to have more people to share it with! I’m excited to explore radishes as well as nappa cabbage.
Coffee: I started drinking coffee in 2014 and have mostly had coffee made by other people: tech jobs in NYC have pretty great office coffee, and the process was something of a mystery to me at the beginning. While I had a Chemex and made coffee for myself sometimes, it was definitely a special occasion sort of thing and not part of my regular routine. Now making a pot of coffee is part of my morning routine, and I find it pretty satisfying, and while I definitely look forward to drinking coffee that other people make more often in the future, I really like getting up making a pot of coffee and sitting down to write nearly every morning.

Sweater Backlog

Thu, 31 Dec 2020 00:00:00 +0000

While I’ve been working on this knitting book project, I’ve realized that I’ve developed something of a backlog of sweaters that I want to knit (for this project, and others,) and I thought I’d write them all down for our collective enjoyment.

A second version of the favorite sweater I knit during college: a plain sweater with a black body, gray at shoulders, drop shoulder, steeks, and gray sleeves. I want to do this out of fingering weight wool, probably HD Shetland. [in progress]
The same sweater as above, except with set in sleeves, and probably light blue as the contrasting color at the sleeve.
Sock yarn sweater, superwash, with v-neck, and some kind of textrued stitch pattern at the yoke that would be knitted without. I made one of these sweaters already, but it came out a touch smaller than I think I really want, and there are a collection of small modifications that I’d like to make, again for verification purposes.
A fully gray sweater, also out of HD Shetland, with the sleeves knit cuff-up, and with the yoke knit with “set-in sleeves,” in the round. I knit a sweater like this in the fall, and I want to verify that a basic a set of changes would make the sweater really wearable.
I have the first 3-5 inches of a color work cardigan that I started before my hiatus from knitting: I think there are a number of flaws with this sweater: the sizing is off, I didn’t handle the bottom hem correctly, and I don’t think I have enough yarn in these specific collors, but I think I’d like to attempt the pattern again.
A color work sweater with set in sleeves. Because knitting rows (back and forth) in stranded is annoying and a bit fussy, most patterns use drop shoulder shaping, but I’d like to experiment and see if I can perfect the technique a more fitted style for these sweaters.
Using a “garter rib” stitch for the full body of a sweater, both because I like the idea of a gentle rib pulling in to provide more fit, while also being fun and keeping a fairly simple shape over the entire length of the sweater.

I think that’s enough for now!

Towards Faster Emacs Start Times

Wed, 30 Dec 2020 00:00:00 +0000

While I mostly run emacs as a daemon managed by systemd I like to keep my configuration nimble enough that I can just start emacs and have an ad-hoc session for editing some files.¹ Or that’s the goal. This has been a long running project for me, but early on in quarantine times, I’ve gotten things tightened up to the point that I feel comfortable using EDITOR=emacs just about all the time. On my (very old) computers, emacs starts and in well under 1 or 2 seconds, and does better on faster machines. If your emacs setup takes a long time to start up, this article is for you! Let’s see if we can get the program to start faster.

Measure!

The emacs-init-time function in emacs will tell you how long it took emacs to start up. This tends to be a bit under the actual start time, because some start-up activity gets pushed into various hooks, after startup. Just to keep an eye on things, I have a variant of the following as the first line in my config file: :

(add-to-list 'after-init-hook
      (lambda ()
        (message (concat "emacs (" (number-to-string (emacs-pid)) ") started in " (emacs-init-time)))))

It’s also possible to put additional information in this hook (and in reality I use something that pushes to a desktop notification tool, rather than just to message/logging.

You can also lower this number by putting more work into the after-init hook, but that sort of defeats the purpose: if you get the init time down to a second, but have 10 seconds of init-hook runtime, then that’s probably not much of a win, particularly if the init-hook tasks are blocking.

The second thing to do is have a macro² available for use that allows to measure the runtime of a block of code, something like: :

(defmacro with-timer (name &rest body)
  `(let ((time (current-time)))
 ,@body
 (message "%s: %.06f" ,name (float-time (time-since time)))))

You can then do something like: :

(with-timer "mode-line-setup"
  (set-up-mode-line))

And then look in the *Message* buffer to see how long various things take so you know where focus your efforts.

Strategies

use-package makes it possible to do much less during applciation startup, and makes it possible to load big lisp packages only when you’re going to use them, and registers everything so you never feel the difference. Audit your use-package forms and ensure that they all declare one of the following forms, which insures that things are loaded upon use, rather than at start time:
- :bind registers specific keybindings, but as autoloads so that the pacakge is only loaded when you use the keybinding.
- :command registers function names as autoloads, like keybinding, so the commands/functions are available but again the package isn’t loaded until it’s used.
- :mode attaches the package to a specific file extension or extensions so again, the package doesn’t get loaded until you open a file of that type.
- :after for loading a package after another package it depends on or is only used after.
- :hook this allows you to associate something from this package into another package’s hook, with the same effect of deferring loading of one package until after another is used.
Avoid having the default-scratch buffer trigger loading as much as possible. It’s very tempting to have *scratch* default to org-mode or something, but if you keep it in fundamental-mode you can avoid loading most of your packages until you actually use them, or can avoid pulling in too much too quickly.
Keep an eye on startup both for GUI, terminal, and if you use daemons, daemon instances of emacs. There are some subtleties that may be useful or important. It’s also the case that terminal startup times are often much less than GUI times.
Think about strategies for managing state-management (e.g. recentf session-mode and desktop-mode.) I’ve definitley had times when I really wanted emacs to be able to restart and leave me exactly where I was when I shut down. These session features are good, but often it’s just crufty, and causes a lot of expense at start up time. I still use desktop and session, but less so.
Fancy themes and modelines come at some cost. I recently was able to save a lot of start up time by omiting a call to spaceline-compile in my configuration. Theme setup also can take a bit of time in GUI mode (I think!), but I dropped automatic theme configuration so that my instances would play better with terminal mode.³

Helpful Settings

I think of these four settings as the “start up behavior” settings from my config: :

(setq inhibit-startup-echo-area-message "tychoish")
(setq inhibit-startup-message 't)
(setq initial-major-mode 'fundamental-mode)
(setq initial-scratch-message 'nil)

I’ve had the following jit settings in my config for a long time, and they tend to make things more peppy, which can only help during startup. This may also just be cargo-cult, stuff: :

(setq jit-lock-stealth-time nil)
(setq jit-lock-defer-time nil)
(setq jit-lock-defer-time 0.05)
(setq jit-lock-stealth-load 200)

If you use desktop-mode to save state and re-open buffers, excluding some modes from this behavior is good. In my case, avoiding reopening (and rereading) potentially large org-mode buffers was helpful for me. The desktop-modes-not-to-save value is useful here, as in the following snippet from my configuration: :

(add-to-list 'desktop-modes-not-to-save 'dired-mode)
(add-to-list 'desktop-modes-not-to-save 'Info-mode)
(add-to-list 'desktop-modes-not-to-save 'org-mode)
(add-to-list 'desktop-modes-not-to-save 'info-lookup-mode)
(add-to-list 'desktop-modes-not-to-save 'fundamental-mode)

Notes

Ok, ok, I live with a vim user and the “quick startup time,” situation is very appealing. ↩︎
I tweaked this based on what I found this on stack overflow which stole it from this listserv post. ↩︎
Mixed GUI/Terminal mode with themes can be kind of finicky because theme settings are global and color backgrounds often won’t play well with terminal frames. You can say “if this is a gui mode,” but that doesn’t play well with daemons, and hooking into frame-creation has never worked as seamlessly as I would expect, because the of the execution context the hook functions (and slowing down frame creation can defeat part of the delight of the daemon.) ↩︎

New Project: Gestalt Knitting

Tue, 29 Dec 2020 00:00:00 +0000

A couple of months ago I started writing a thing that I think will be a book, or something close, about knitting. A few weeks ago, I jokingly called it “Gestalt Knitting,” and the name stuck, at least for myself for now. The idea is to write a book, that’s mostly about knitting, that’s less a collection of patterns, and more of a meditation on the process of designing and knitting some sweaters, hats, and socks.

As a knitter, I’ve never really followed patterns: even when I see a design that I like, I always end up changing it somehow: changing the process to be something that I’d enjoy more (e.g. knitting in the round,) or modifying the size a bit (I’ve always been slim and I know I like to avoid wearing things that feel like tents,) or adding features that I know increase the comfort (e.g. making the neckline more open, or the sleeves a touch wider.) Patterns, thus, for me are sort of an opening volley in a conversation about a project rather than a description of a project, and I think in practice this is pretty common, though I think that often, knitters think about their modification as minor and inconsequential when really they’re super cool and important.

I’m less interested in any specific design or pattern, and more interested in these kinds of basic patterns that we can reach to when faced with a pile of yarn and a desire to knit a sweater (or socks, etc.). Sort of the knitting equivalent of a family recipe that you can make without looking at any instructions and maybe without really measuring the ingredients. After a while, I suspect many knitters have a few things that they can just make on their own--like a hat or some simple socks--and one of my goals of this project is to help expand the collection of “house patterns” by explaining and exploring some of the basic garments that I tend to knit.

As a writer, I have a lot of practice writing about the details of complicated ideas and processes in unambiguous and clear terms. I’m interested in seeing if I can take my style for writing about procedures and concepts can translate to another subject area. Knitting patterns are, by convention, super concise and linear in a way that gives people just enough information to reproduce a specific garment, but ends up leaving out so much useful information about why you would knit something in a specific way, options for modification, or interactions between the pattern, the shape, and the process of knitting. When I’m knitting I get a lot of delight from the way all of details of a project come together, and I want to frame and present knitting projects in ways that really highlight that aspect of knitting. I also want to write something that’s knitters would find engaging to read all on its own, even without overlapping any of my specific knitting projects.

While I’ve made a lot of progress on the draft, I still have a bunch of work to do on this project: both in terms of more writing and also knitting “research.” I suspect I’ll write a bit about it here. Stay tuned!

I made a new instagram account, because that seems to be a thing, for knitting specific things: @gestaltknitting

Knitting Hiatus and Knitting Again

Mon, 28 Dec 2020 00:00:00 +0000

While I knit a lot in college, and for a few years afterwords, my attention to knitting as a hobby kind of trailed off, and I have basically not knit at all in the last five years, but suddenly a bit before Thanksgiving, I found myself listening to an audio book and playing a silly game on my phone really wishing that I could knit, and given that it was 2020 I did, and it’s been a lot of fun and pretty satisfying to get back into knitting.

There were lots of reasons for the hiatus, but the leading reasons were:

My professional work, first technical writing for software engineers, and then software engineering itself, was actually quite similar to knitting at least conceptually: knitting and software require lots of problem solving and similar kinds of iterative math. Particularly for the first few years where I was teaching myself how to program, I felt like knitting ended up being more like work than I wanted.
I lived in southern Wisconsin in college, and it was actually cold. Between climate change, and ending up living in New York City--being a huge city that holds heat, and being on a island with the harbor as a heat sink, most buildings in the City have very aggressive heating systems--it felt like I never really wanted to wear wool, because I was often too warm.

These problems seem solveable: my learning curve as a programmer has become less steep, and my day-to-day engineering work tends has shifted to be higher level and organizational in some ways that feel less like knitting. I think the “always too warm” feeling about my city has changed a bit as I’ve acclimated and I think it will be possible to just knit very light weight things and combine them with light weight jackets for increased wearability.

So I’m back.

I’ve finished the sweater I started in 2015 (it’s not exceptional, but I know how to fix it for the future,) knit another sweater that I’m pretty pleased with, and I’ve started a third and I’ve been working on a book about knitting that I’m quite excited about. I never kept a really extensive collection of yarn, but I have enough to keep me busy for a while, and using the yarn I already have has been interesting as a constraining function in planning new projects.

I suspect I’ll be blogging about knitting a bit more over the next little bit, about both specific projects and maybe some higher level things. I hope you don’t mind!

Staff Engineering

Sun, 27 Sep 2020 00:00:00 +0000

In August of 2019 I became a Staff Engineer, which is what a lot of companies are calling their “level above Senior Engineer” role these days. Engineering leveling is a weird beast, which probably a post onto itself. Despite my odd entry into a career in tech, my path in the last 4 or 5 years has been pretty conventional; however, somehow, despite having an increasingly normal career trajectory, explaining what I do on a day to day basis has not gotten easier.

Staff Engineers are important for scaling engineering teams, but lots of teams get by with out them, and unlike more junior engineers who have broadly similar job roles, there are a lot of different ways to be a Staff Engineer, which only muddies things. This post is a reflection on some key aspects of my experience organized in to topics that I hope will be useful for people who may be interested in becoming staff engineers or managing such a person. If you’re also a Staff Engineer and your experience is different, I wouldn’t be particularly surprised.

Staff Engineers Help Teams Build Great Software

Lots of teams function just fine without Staff Engineers and teams can build great products without having contributors in Staff-type roles. Indeed, because Staff Engineers vary a lot, the utility of having more senior individual contributors on a team depends a lot of the specific engineer and the team in question: finding a good fit is even harder than usual. In general, having Senior Technical leadership can help teams by:

giving people managers more space and time to focus on the team organization, processes, and people. Particularly in small organizations, team managers often pick up technical leadership.
providing connections and collaborations between groups and efforts. While almost all senior engineers have a “home team” and are directly involved in a few specific projects, they also tend to have broader scope, and so can help coordinate efforts between different projects and groups.
increasing the parallelism of teams, and can provide the kind of infrastructure that allows a team to persue multiple streams of development at one time.
supporting the career path and growth of more junior engineers, both as a result of direct mentoring, but also by enabling the team to be more successful by having more technical leadership capacity creates opportunities for growth for everyone on the team.

Staff Promotions Reflect Organizational Capacity

In addition to experience and a history of effusiveness, like other promotions, getting promoted to Staff Engineer is less straight forward than other promotions. This is in part because the ways we think about leveling and job roles (i.e. to describe the professional activities and capabilities along several dimensions for each level,) become complicated when there are lots of different ways to be a Staff Engineer. Pragmatically, these kind of promotions often depend on other factors:

the existence of other Staff Engineers in the organization make it more likely that there’s an easy comparison for a candidate.
past experience of managers getting Staff+ promotions for engineers. Enginering Managers without this kind of experience may have difficulty creating the kinds of opportunities within their organizations and for advocating these kinds of promotions.
organizational maturity and breadth to support the workload of a Staff Engineer: there are ways to partition teams and organizations that preclude some of the kinds of higher level concerns that justify having Staff Engineers, and while having senior technical leadership is often useful, if the organization can’t support it, it won’t happen.
teams with a sizable population of more junior engineers, particularly where the team is growing, will have more opportunity and need for Staff Engineers. Teams that are on the balance more senior, or are small and relatively static tend to have less opportunity for the kind of broadly synthetic work that tends to lead to Staff promotions.

There are also, of course, some kinds of technical achievements and professional characteristics that Staff Engineers often have, and I’m not saying that anyone in the right organizational context can be promoted, exactly. However, without the right kind of organizational support and context, even the most exceptional engineers will never be promoted.

Staff Promotions are Harder to Get Than Equivalent Management Promotions

In many organizations its true that Staff promotions are often much harder to get than equivalent promotions to peer-level management postions: the organizational contexts required to support the promotion of Engineers into management roles are much easier to create, particularly as organizations grow. As you hire more engineers you need more Engineering Managers. There are other factors:

managers control promotions, and it’s easier for them to recapitulate their own career paths in their reports than to think about the Staff role, and so more Engineers tend to be pushed towards management than Senior IC roles. It’s also probably that meta-managers benefit organizationally from having more front-line managers in their organizations than more senior ICs, which exacerbates this bias.
from an output perspective, Senior Engineers can write the code that Staff Engineers would otherwise write, in a way that Engineering Management tends to be difficult to avoid or do without. In other terms, management promotions are often more critical from the organization’s perspective and therefore prioritized over Staff promotions, particularly during growth.
cost. Staff Engineers are expensive, often more expensive than managers particularly at the bottom of the brackets, and it’s difficult to imagine that the timing of Staff promotions are not impacted by budgetary requirements.

Promoting a Staff Engineer is Easier than Hiring One

Because there are many valid ways to do the Staff job, and so much of the job is about leveraging context and building broader connections between different projects, people with more organizational experience and history often have an advantage over fresh industry hires. In general:

Success as a Staff Engineer in one organization does not necessarily translate to success at another.
The conventions within the process for industry hiring, are good at selecting junior engineers, and there are fewer conventions for more Senior roles, which means that candidates are not assessed for skills and experiences that are relevant to their day-to-day while also being penalized for (often) being unexceptional at the kind of problems that junior engineering interviews focus on. While interview processes are imperfect assessment tools in all cases, they’re particularly bad at more senior levels.
Senior engineering contributors have a the potential to have huge impact on product development, engineering outcomes, all of which requires a bunch of trust on the part of the organization, and that kind of trust is often easier to build with someone who already has organizational experience

This isn’t to say that it’s impossible to hire Staff engineers, I’m just deeply dubious of the hiring process for these kinds of roles having both interviewed for these kinds of roles and also interviewed candidates for them. I’ve also watched more than one senior contributor not really get along well with a team or other leadership after being hired externally, and for reasons that end up making sense in retrospect. It’s really hard.

Staff Engineers Don’t Not Manage

Most companies have a clear distinction between the career trajectories of people involved in “management” and senior “individual contributor” roles (like Staff Engineers,) with managers involved in leadership for teams and humans, with ICs involved in technical aspects. This seems really clear on paper but incredibly messy in practice. The decisions that managers make about team organization and prioritization have necessary technical implications; while it’s difficult to organize larger scale technical initiatives without awareness of the people and teams. Sometimes Staff Engineers end up doing actual management on a temporary basis in order to fill gaps as organizations change or cover parental leave

It’s also the case that a huge part of the job for many Staff Engineer’s involves direct mentorship of junior engineers, which can involve leading specific projects, conversations about career trajectories and growth, as well as conversations about specific technical topics. This has a lot of overlap with management, and that’s fine. The major differences is that senior contributors share responsibility for the people they mentor with their actual managers, and tend to focus mentoring on smaller groups of contributors.

Staff Engineers aren’t (or shouldn’t be!) managers, even when they are involved in broader leadership work, even if the specific engineer is capable of doing management work: putting ICs in management roles, takes time away from their (likely more valuable) technical projects.

Staff Engineers Write Boring and Tedious But Easy Code

While this is perhaps not a universal view, I feel pretty safe in suggesting that Staff Engineers should be directly involved in development projects. While there are lots of ways to be involved in development: technical design, architecture, reviewing code and documents, project planning and development, and so fort, I think it’s really important that Staff Engineers be involved with code-writing, and similar activies. This makes it easy to stay grounded and relevant, and also makes it possible to do a better job at all of the other kinds of engineering work.

Having said that, it’s almost inevitable that the kinds of contribution to the code that you make as a Staff Engineer are not the same kinds of contributions that you make at other points in your career. Your attention is probably pulled in different directions. Where a junior engineer can spend most of their day focusing on a few projects and writing code, Staff Engineers:

consult with other teams.
mentor other engineers.
build long and medium term plans for teams and products.
breaking larger projects apart and designing APIs between components.

All of this “other engineering work” takes time, and the broader portfolio of concerns means that more junior engineers often have more time and attention to focus on specific programming tasks. The result is that the kind of code you end up writing tends to be different:

fixing problems and bugs in systems that require a lot of context. The bugs are often not very complicated themselves, but require understanding the implication of one component with regards to other components, which can make them difficult.
projects to enable future development work, including building infrastructure or designing an interface and connecting an existing implementation to that interface ahead of some larger effort. This kind of “refactor things to make it possible to write a new implementation.”
writing small isolated components to support broader initiatives, such as exposing existing information via new APIs, and building libraries to facilitate connections between different projects or components.
projects that support the work of the team as a whole: tools, build and deployment systems, code clean up, performance tuning, test infrastructure, and so forth.

These kinds of projects can amount to rather a lot of development work, but they definitely have their own flavor. As I approached Staff and certainly since, the kind of projects I had attention for definitely shifted. I actually like this kind of work rather a lot, so that’s been quite good for me, but the change is real.

There’s definitely a temptation to give Staff Engineers big projects that they can go off and work on alone, and I’ve seen lots of teams and engineers attempt this: sometimes these projects work out, though more often the successes feel like an exception. There’s no “right kind” of way to write software as a Staff Engineer, sometimes senior engineer’s get to work on bigger “core projects.” Having said that, if a Staff Engineer is working on the “other engineering” aspects of the job, there’s just limited time to do big development projects in a reasonable time frame.

New Beginnings: Deciduous Platform

Sun, 30 Aug 2020 00:00:00 +0000

I left my job at MongoDB (8.5 years!) at the beginning of the summer, and started a new job at the beginning of the month. I’ll be writing and posting more about my new gig, career paths in general, reflections on what I accomplished on my old team, the process of interviewing as a software engineer, as well as the profession and industry over time. For now, though, I want to write about one of the things I’ve been working on this summer: making a bunch of the open source libraries that I worked on more generally useable. I’ve been calling this the deciduous platform,¹ which now has its own github organization! So it must be real.

The main modification in these forks, aside from adding a few features that had been on my list for a while, has been to update the buildsystem to use go modules² and rewrite the history of the repository to remove all of the old vendoring. I expect to continue development on some aspects of these over time, though the truth is that these libraries were quite stable and were nearly in maintenance mode anyway.

Background

The team was responsible for a big monolith (or so) application: development had begun in 2013, which was early for Go, and while everything worked, it was a bit weird. My efforts when I joined in 2015 focused mostly on stabilization, architecture, and reliability. While the application worked, mostly, it was clear that it suffered from a few problem, which I believe were the result of originating early in the history of Go: First, because no one had tried to write big applications yet, the patterns weren’t well established, and so the team ended up writing code that worked but that was difficult to maintain, and ended up with bespoke solutions to a number of generic problems like running workloads in the background or managing Apia. Second, Go’s standard library tends to be really solid, but also tends towards being a little low level for most day-to-day tasks, so things like logging and process management end up requiring more code³ than is reasonable.

I taught myself to write Go by working on a logging library, and worked on a distributed queue library. One of the things that I realized early, was that breaking the application into “microservices,” would have been both difficult and offered minimal benefit,⁴ so I went with the approach of creating a well factored monolith, which included a lot of application specific work, but also building a small collection of libraries and internal services to provide useful abstractions and separations for application developers and projects.

This allowed for a certain level of focus, both for the team creating the infrastructure, but also for the application itself: the developers working on the application mostly focused on the kind of high level core business logic that you’d expect, while the infrastructure/platform team really focused on these libraries and various integration problems. The focus wasn’t just organizational: the codebases became easier to maintain and features became easier to develop.

This experience has lead me to think that architecture decisions may not be well captured by the monolith/microservice dichotomy, but rather there’s' this third option that centers on internal architecture, platforms, and the possibility for developer focus and velocity.

Platform Overview

While there are 13 or so repositories in the platform, really there are 4 major libraries: grip, a logging library; jasper, a process management framework; amboy, a (possibly distributed) worker queue; and gimlet, a collection of tools for building HTTP/REST services.

The tools all work pretty well together, and combine to provide an environment where you can focus on writing the business logic for your HTTP services and background tasks, with minimal boilerplate to get it all running. It’s pretty swell, and makes it possible to spin up (or spin out) well factored services with similar internal architectures, and robust internal infrastructure.

I wanted to write a bit about each of the major components, addressing why I think these libraries are compelling and the kinds of features that I’m excited to add in the future.

Grip

Grip is a structured-logging friendly library, and is broadly similar to other third-party logging systems. There are two main underlying interfaces, representing logging targets (Sender) and messages, as well as a higher level “journal” interface for use during programming. It’s pretty easy to write new message or bakcends, which means you can use grip to capture all kinds of arbitrary messages in consistent manners, and also send those messages wherever they’re needed.

Internally, it’s quite nice to be able to just send messages to specific log targets, using configuration within an application rather than needing to operationally manage log output. Operations folks shouldn’t be stuck dealing with just managing logs, after all, and it’s quite nice to just send data directly to Splunk or Sumologic. We also used the same grip fundamentals to send notifications and alerts to Slack channels, email lists, or even to create Jira Issues, minimizing the amount of clunky integration code.

There are some pretty cool projects in and around grip:

support for additional logging targets. The decudous version of grip adds twitter as an output format as well as creating desktop notifications (e.g. growl/libnotify,) but I think it would also be interesting to add fluent/logstash connections that don’t have to transit via standard error.'
While structured logging is great, I noticed that we ended up logging messages automatically in the background as a method of metrics collection. It would be cool to be able to add some kind of “intercepting sender” that handled some of these structured metrics, and was able to expose this data in a format that the conventional tools these days (prometheus, others,) can handle. Some of this code would clearly need to be in Grip, and other aspects clearly fall into other tools/libraries.

Amboy

Amboy is an interface for doing things with queues. The interfaces are simple, and you have:

a queue that has some way of storing and dispatching jobs.
implementations of jobs which are responsible for executing your business logic, and with a base implemention that you can easily compose, into your job types, all you need to implement, really is a Run() method.
a queue “group” which provides a higher level abstraction on top of queues to support segregating workflows/queues in a single system to improve quality of service. Group queues function like other queues but can be automatically managed by the processes.
a runner/pool implementation that provides the actual thread pool.

There’s a type registry for job implementations and versioning in the schema for jobs so that you can safely round-trip a job between machines and update the implementation safely without ensuring the queue is empty.

This turns out to be incredibly powerful for managing background and asynchronous work in applications. The package includes a number of in-memory queues for managing workloads in ephemeral utilities, as well as a distributed MongoDB backed-queue for running multiple copies of an application with a shared queue(s). There’s also a layer of management tools for introspecting, managing, the state of jobs.

While Amboy is quite stable, there is a small collection of work that I’m interested in:

a queue implementation that store jobs to a local Badger database on-disk to provide single-system restartabilty for jobs.
a queue implementation that stores jobs in a PostgreSQL, mirroring the MongoDB job functionality, to be able to meet job backends.
queue implementations that use messaging systems (Kafka, AMPQ) for backends. There exists an SQS implementation, but all of these systems have less strict semantics for process restarts than the database options, and database can easily handle on the order of a hundred of thousand of jobs an hour.
changes to the queue API to remove a few legacy methods that return channels instead of iterators.
improve the semantics for closing a queue.

While Amboy has provisions for building architectures with workers running on multiple processes, rather than having queues running multiple threads within the same process, it would be interesting to develop more fully-fledged examples of this.

Jasper

Jasper provides a high level set of tools for managing subprocesses in Go, adding a highly ergonomic API (in Go,) as well as exposing process management as a service to facilitate running processes on remote machines. Jasper also manages/tracks the state of running processes, and can reduce pressures on calling code to track the state of processes.

The package currently exposes Jasper services over REST, gRPC, and MongoDB’s wire protocol, and there is also code to support using SSH as a transport so that you don’t need to expose remote these services publically.

Jasper is, perhaps, the most stable of the libraries, but I am interested in thinking about a couple of extensions:

using jasper as PID 1 within a container to be able to orchestrate workloads running on containers, and contain (some) support for lower level container orchestration.
write configuration file-based tools for using jasper to orchestrate buildsystems and distributed test orchestration.

I’m also interested in cleaning up some of the MongoDB-specific code (i.e. the code that downloads MongoDB versions for use in test harnesses,) and perhaps reinvisioning that as client code that uses Jasper rather than as a part of Jasper.

Gimlet

I’ve written about gimlet here before when I started the project, and it remains a pretty useful and ergonomic way to define and regester HTTP APIs, in the past few years, its grown to add more authentication features, as well as a new “framework” for defining routes. This makes it possible to define routes by implementing an interface that:

makes it very easy to produce paginated routes, and provides some helpers for managing content
separates the parsing of inputs from executing the results, which can make route definitions easy to test without integration tests.
rehome functionality on top of chi router. The current implementation uses Negroni and gorilla mux (but neither are exposed in the interface), but I think it’d be nice to have this be optional, and chi looks pretty nice.

Other Great Tools

The following libraries are defiantly smaller, but I think they’re really cool:

birch is a builder for programatically building BSON documents, and MongoDB’s extended JSON format. It’s built upon an earlier version of the BSON library. While it’s unlikely to be as fast at scale, for many operations (like finding a key in a document), the interface is great for constructing payloads.
ftdc provides a way to generate (and read,) MongoDB’s diagnostic data format, which is a highly compressed timeseries data format. While this implementation could drift from the internal implementation over time, the format and tool remain useful for arbitrary timeseries data.
certdepot provides a way to manage a certificate authority with the certificates stored in a centralized store. I’d like to add other storage backends over time.

And more…

Notes

My old team built a continuous integration tool called evergreen which is itself a pun (using “green” to indicate passing builds, most CI systems are not ever-green.) Many of the tools and libraries that we built had got names with tree puns, and somehow “deciduous” seems like the right plan. ↩︎
For an arcane reason, all of these tools had to build with an old version of Go (1.10) that didn’t support modules, so we had an arcane and annoying vendoring solution that wasn’t compatible with modules. ↩︎
Go tends to be a pretty verbose language, and I think most of the time this creates clarity; however, for common tasks it has the feeling of offering a poor abstraction, or forcing you to write duplicated code. While I don’t believe that more-terse-code is better, I think there’s a point where the extra verbosity for route operations just creates the possibility for more errors. ↩︎
The team was small, and as an internal tools team, unlikely to grow to the size where microservices offered any kind of engineering efficiency (at some cost,) and there weren’t significant technical gains that we could take advantage of: the services of the application didn’t need to be globally distributed and the boundaries between components didn’t need to scale independently. ↩︎

Tycho Emacs Config Kit

Sat, 22 Aug 2020 00:00:00 +0000

So I made a thing, or at least, I took a thing I’ve built and made it presentable for other people. I’m talking, of course, about my Emacs configuration.

Check it out at github.com/tychoish/.emacs.d! The README is pretty complete, and giving it a whirl is simple, just do something like: :

mv ~/.emacs.d/ ~/emacs.d-archive
git clone --recurse-submodules git@github.com:tychoish/.emacs.d.git ~/.emacs.d/

If you’re using Emacs 27, you might also be able to clone it into ~/.config/emacs, for similar effect. It doesn’t matter much to me. Let me know what you think!

The tl;dr of “what’s special about this config,” is that, it has:

a good set of defaults out of the box.
a lot of great wiz-bang features enabled and configured.
very quick start times, thanks to lazy-loading of libraries.
great support for running as a daemon, and even running multiple daemons at once.

I’ve sporadically produced copies of my emacs config for other folks to use, but those were always ad hoc, and difficult to reproduce and maintain, and I’ve been working on and off to put more polish on things, so making it usable for other people seemed like a natural step at this point.

I hope other people find it useful, but also I think it’s a good exercise for me in keeping things well maintained, and it doesn’t bother me much one way or another.

Discussion

I think a lot about developer experience: I’ve spent my career, thus far, working on infrastructure products (databases, CI systems, build tools, release engineering,) that help improve developer experience and productivity, and even my day-to-day work tends to focus mostly on problems that impact the way that my teams write software: system architecture, service construction, observability, test infrastructure, and similar.

No matter how you slice it, editor experience is a huge factor in people’s experience, and can have a big impact on productivity, and there are so many different editors with wildly different strengths. Editors experience is also really hard: unlike other kinds of developer infrastructure (like buildsystems, or even programming languages) the field conceptualizes editors as personal: your choice in editor is seen to reflect on you (it probably doesn’t), and your ability to use your editor well is seen to be a reflection of your skills as a developer (it isn’t). It’s also the case that because editors are so personal, it’s very difficult to produce an editor with broad appeal, and the most successful editors tend to be very configurable and can easily lack defaults, which means that even great editors are terrible out of the box, which mostly affects would-be and new developers.

Nevertheless, time being able to have an editor that you’re comfortable with and that you can use effectively without friction does make it easier to build software, so even if folks often conceptualize of editors in the wrong way, improving the editing experience is important. I think that there are two areas for improvement:

editor configurations should--to some extent--be maintained at the project (or metaproject) level, rather than on the level individual engineer. The hard part here is how to balance individual preference with providing a consistent experience, particularly for new developers.¹
there should be more “starter kits” (like this one! or the many other starter kits, but also for (neo)vim, vscode, and others.) that help bootstrap the experience, while also figuring out ways to allow layering other project-based extensions on top of a base configuration.

Also, I want to give chemacs a shout out, for folks who want to try out other base configurations.

There are two kinds of new developers with different experiences but some overlap: folks who have development experience in general but no experience with a specific project, and folks who are new to all development. ↩︎

Pickle Quarantine

Sun, 16 Aug 2020 00:00:00 +0000

What a year, am I right?

I’ve avoided writing a more “journal”-type post because there’s both too much to say and somehow not enough all at the same time. This is a tough time to be a human, and I’ve been incredibly lucky in many regards: I can stay busy, I have a living situation that makes quarantining easy and tolerable, and while I wish the produce at my local grocery had better produce, it’s nice to live in a very walkable and accessible neighborhood. Even though “nowtimes” are fundamentally different from the “beforetimes,” the speed at which things become routine--if not normal--is almost disturbing. Delightfully, though, I’ve felt like I’m still getting things done. In no particular order:

As the title of the post suggests I have firmly entered the home pickling phase of quarantine, and have begun fermenting rather a lot of cauliflower, with additional vegetables to follow. Frankly I don’t know why I didn’t think of this earlier: I love pickles and it’s really hard to get compelling pickles for things that aren’t cucumbers. My first batch is cauliflower, and I’m also excited to do things like: baby okra, cabbage, greenbeans, and maybe some of my favorite root vegetables (shallots, raddishes, horseradish, jerusalem artichokes, etc.) Details forcoming, perhaps.
I’ve begun cooking more often. My go to, is to make a pot of chicken stock, and then figure out things to use the stock to cook. I’ve done some nice proto-beef stew things, and have made some pretty great lentil dishes. Nothing fancy or complex. I’m also delightfully, getting to a point where I can produce a meal that consists of more than one flavor profile, which is nice.
I’ve written enough common lisp that writing little bits of software in CL seem doable. It’s still a bit harder to get into bigger projects, and I have a lot to learn still, but I’ve crossed a line, and it’s nice to have an additional skill.
I replaced the video card in my home desktop, and I feel like I have a totally new computer! It’s amazing. I got this computer in 2013, and have mostly been using laptops for the past few years. Before quarantine, I was even planning to de-accession it, but I’ve been using it a lot more and the new video card really changes things. Since I don’t do a lot of graphics things and my computer use is not particularly performance intensive, I forget how much having an underpowered GPU can really impact perceptions: I was suffering through a lot of rendering-hangs and glitches that just don’t happen any more.
I’ve been writing blog posts more, and am working on a plan to have something that resembles a publication schedule. I wrote professionally for so long, that writing for fun never seemed like a viable way to spend time, but as time goes on, it’s clear that I really want to be able to experiment with ideas and reflect on things in a more regular way, and continuing to write posts here seems like a good way to do that. My lisp-related posts are now appearing on Planet Lisp, and I’m interested in expanding my reach… slightly.
I completed a sort of long term project to modernize the way that I use and configure emacs: I got all of the hardcoded weirdness undercontrol, I started using use-package, and I got lsp-mode to work, and I fixed a number of hilarious little bugs in how my config worked. The end result, is I get to use the editor I want in the way that I want, and it all starts up on my (old) computers in well under 2 seconds, not that I start up all that often. I expect I’ll write a bit more on this soon.
Since quarantine started I’ve gone through an entire complete box of Chemex filters (plus the tail end of a box, and I’m a few pots into the next box.) I started drinking coffee during the summer of 2015, and have primarily consumed coffee produced by other people (i.e. at work) in that time. While I’ve definitely gone through a few boxes of Chemex filters over the years, my new rate is noticeable. But quarantine means making your own coffee, and apparently I’ve made a lot of coffee. It feels like a milestone.

Lazy Lentils

Tue, 04 Aug 2020 00:00:00 +0000

Alternate title: yolo lentils

I’ve been cooking more in the last few months that I have, basically ever. I’m not a bad cook, and my instincts and raw skills are pretty decent, but I’m not super inspired, I guess. Left to my own devices I eat the same few things again and again. So the exercise of mostly cooking for myself has been a lot about figuring out things to cook that taste good (and different!) and that I can reproduce reliably. Sometimes I’m more successful than other times, and frankly a lot of what I’ve come up with isn’t super inspiring, but I recently stumbled upon something that was pretty great:

Ingredients:

1 Cup of Black Lentils
2 Cups of Chicken Stock
1 Onion chopped reasonably finely.
3 cloves of garlic sliced.
1 inch piece of fresh ginger microplaned or minced.
(some) Olive Oil
Spices: Sweet Curry Powder, Garam Masala, Tumeric
Salt and Pepper

Procedure:

Saute garlic onion, and ginger in olive oil until soft and becoming translucent, adding spices near the end. (5 minutes.)
Add stock and lentils, and cook until lentils soften. I did the whole thing in an instant pot for 8-10 minutes and let the pressure release naturally (without keep warm). I’m sure other kinds of pots and cooking methods work well too.

That’s it.

Modifications and notes:

I put the spices in at the end of the sauteing phase, and I might have put them in slightly earlier, but it’s fine.
I like that black lentils retain their form after cooking, but if you like more mushy texture other lentils would be fine.
If you don’t use a pressure cooker, it seems reasonable that more liquid would be useful.
I used chicken stock because it’s delicious and I had just made a batch of right before I made the lentils so it seemed reasonable. The stock flavor is secondary to the spices.
I didn’t measure the spices (which were not particularly fresh, I must confess,) but I listed them roughly in terms of amount.

Enjoy!

Running Emacs

Mon, 03 Aug 2020 00:00:00 +0000

OK, this is a weird post, but after reading a post about running emacs with systemd, ¹ I’ve realized that the my take on how I run and manage processes is a bit unique, and worth enumerating and describing. The short version is that I regularly run multiple instances of emacs, under systemd, in daemon mode, and it’s pretty swell. Here’s the rundown:

On Linux systems, I use a build of emacs with the lucid toolkit, rather than GTK, because of this bug with GTK, which I should write more about at some point. Basically, if the X session crashes with GTK emacs, even if you don’t have windows open, the daemon will crash, even if the dameon isn’t started from a GUI session. The lucid toolkit doesn’t have this problem.
I run the process under my user’s systemd instance, rather than under the PID 1 systemd instance. I like keeping things separate. Run the following command to ensure that your user’s systemd will start at boot rather than at login: :
```
sudo loginctl enable-ligner $(whoami)
```

I have a systemd service file named emacs@.service in my ~/.config/systemd/user/ directory that looks like this: :

[Unit]
Description=Emacs-tychoish: the extensible, self-documenting text editor.

[Service]
Type=forking
ExecStart=/usr/bin/emacs --daemon=%i --chdir %h
ExecStop=/usr/bin/emacsclient --server-file=hud --eval "(progn (setq kill-emacs-hook 'nil) (kill-emacs))"
Restart=always
TimeoutStartSec=0

[Install]
WantedBy=default.target

I then start emacs dameons: :

systemctl --user start emacs@work
systemctl --user start emacs@personal
systemctl --user start emacs@chat

To enable them so that they start following boot: :

systemctl --user enable emacs@work
systemctl --user enable emacs@personal
systemctl --user enable emacs@chat

Though to be honest, I use different names for daemons.

I have some amount of daemon specific code, which might be useful: :

(setq server-use-tcp t)

(if (equal (daemonp) nil)
(setq tychoish-emacs-identifier "solo")
  (setq tychoish-emacs-identifier (daemonp)))

;; this makes erc configs work less nosily. There's probably no harm in
;; turning down the logging
(if (equal (daemonp) "chat")
(setq gnutls-log-level 0)
  (setq gnutls-log-level 1))

(let ((csname (if (eq (daemonp) nil)
     "generic"
     (daemonp))))
  (setq recentf-save-file (concat user-emacs-directory system-name "-" csname "-recentf"))
  (setq session-save-file (concat user-emacs-directory system-name "-" csname "-session"))
  (setq bookmark-default-file (concat user-emacs-directory system-name "-" csname "-bookmarks"))
  (setq helm-c-adaptive-history-file (concat user-emacs-directory system-name "-" csname "--helm-c-adaptive-history"))
  (setq desktop-base-file-name (concat system-name "-" csname "-desktop-file"))
  (setq desktop-base-lock-name (concat system-name "-" csname "-desktop-lock")))

Basically this just sets up some session-specific information to be saved to different files, to avoid colliding per-instance.

Additionally, I use the tychoish-emacs-identifier from above to provide some contextual information as to what emacs daemon/window I’m currently in: :
```
(setq frame-title-format '(:eval (concat tychoish-emacs-identifier ":" (buffer-name))))

(spaceline-emacs-theme 'daemon 'word-count)
(spaceline-define-segment daemon tychoish-emacs-identifier)
```
Also, on the topic of configuration, I do have a switch statement that loads different mu4e configurations in different daemons.

To start emacs sessions, I use operations in the following forms: :

# create a new emacs frame. Filename optional.
emacsclient --server-file=<name> --create-frame --no-wait <filename>

# open a file in an existing (last-focus) frame/window. Filename required.
emacsclient --server-file=<name> --no-wait <filename>

# open a terminal emacs mode. Filename optional .
emacsclient --server-file=<name> --tty --no-wait <filename>

That’s a lot to type, so I use aliases in my shell profile: :

alias e='emacsclient --server-file=personal --no-wait'
alias ew='emacsclient --server-file=personal --create-frame --no-wait'
alias et='emacsclient --server-file=personal --tty'

I create a set of aliases for each daemon prefixing e/ew/et with the first letter of the daemon name.

And that’s about it. When I’ve used OS X, I’ve managed something similar using launchd but the configuration files are a bit less elegant. On OS X, I tend to install emacs with cocoa toolkit, using homebrew.

Using multiple daemons is cool, though not required, for a number of reasons:

you can have good separation between personal things and professional/work things, which is always nice, but particularly gratifying during the pandemic when it’s easy to work forever.
Managing multiple separation of email. While mu4e has profiles and contexts, and that’s great, I like a firmer boundary, and being able maintain separate email databases.
Running emacs lisp applications that do a lot of networking, or do other blocking operations. The main case where this matters in my experience is running big erc instance, or something else that isn’t easily broken into async/subprocesses.

And commenting! ↩︎

Emacs and LSP Mode

Sat, 25 Jul 2020 00:00:00 +0000

I’ve been using emacs for a long time, and it’s mostly pretty unexceptional. A few months ago, I started using LSP mode, and it’s been pretty great… mostly. This post reflects on some of the dustier corners, and some of the broader implications.

History and Conflict

Language Server is a cool idea, that (as far as I know) grew out of VS Code, and provides a way to give a text editor “advanced, IDE-like” features without having to implement those features per-language in the text editor. Instead, editor developers implement generic integration for the protocol, and language developers produce a single process that can process source code and supports the editor features. Sounds great!

There are challenges:

The LSP protocol uses JSON for encoding data between the server and emacs. This is a reasonable choice, and makes development easy, but it’s not a perfect fit. Primarily, in emacs, until version 27, which hasn’t reached an official release yet (but is quite usable on Linux), parsed JSON in lisp rather than C, which made everything rather slow, but emacs 27 helps a lot. Second,
There is some overhead for the lsp, particularly as you attempt to scale very large numbers of files or very large files. Emacs is really good with large files, as an artifact of having had to support editing larger files on computers 20 or 30 years ago, and lsp feels sluggish in these situations.

I think some of this overhead is probably depends on the implementation of the language server’s themselves, and some percentage is probably protocol itself: http’s statelessness is great in big distributed web services, and for the local use cases where responsiveness matters, it’s less good. Similarly JSON’s transparency and ubiquity is great, but for streaming streaming-type use cases, perhaps less so.

I’ve not poked around the protocol, or any implementations, extensively, so I’m very prepared to be wrong about these points.

There’s also some tension about moving features, like support for adding editor support for a language, out of lisp and into an external process. While I’m broadly sympathetic, and think there are a bunch of high-quality language integrations in emacs lisp, and for basic things, the emacs lisp ones might be faster. Having said that, there’s a lot of interface inconsistency between different support for languages in emacs, and LSP support feels spiffy my comparisons. Also at least for python (jedi,) go (gocode), and C/C++ (irony,) lisp (slime,) the non-lsp services were already using external processes for some portion of their functionality.

It’s also the case that if people spend their emacs-lisp-writing time doing things other than writing support for various development environments it’s all for the better anyway.

Suggestions / Notes

things will break or get wedged, call lsp-restart-workspace to restart the server (I think.) It helps.
use lsp-file-watch-ignored to avoid having the language servers process emacs process files from vendored code, intermediate/generated code, or similar, in cases where you might have a large number of files that aren’t actually material to your development work
I believe some (many? all?) servers are sourced from the PATH (e.g. clang/llvm, gopls), which means you are responsible for updating them and their dependencies.
I’ve found it useful to update lsp-mode itself more often than I typically update other emacs packages. I also try and keep the language servers as up to date as possible in coordination with lsp-mode updates.
If Emacs 27 can work on your system, it’s noticeably faster. Do that.
If you have an existing setup for a language environment, the LSP features end up layering over existing functionality, and that can have unexpected results.

My Configuration

None of this is particularly exciting, but if you use use-package, then the following might be an interesting starting point. I stole a lot of this from other places, so shrug.

I must say that I don’t really use dap or treemacs, because I tend to keep things pretty barebones.

(use-package helm-lsp
  :ensure t
  :after (lsp-mode)
  :commands (helm-lsp-workspace-symbol)
  :init (define-key lsp-mode-map [remap xref-find-apropos] #'helm-lsp-workspace-symbol))

(use-package lsp-mode
  :diminish (lsp-mode . "lsp")
  :bind (:map lsp-mode-map
("C-c C-d" . lsp-describe-thing-at-point))
  :hook ((python-mode . #'lsp-deferred)
(js-mode . #'lsp-deferred)
(go-mode-hook . #'lsp-deferred))
  :init
  (setq lsp-auto-guess-root t       ; Detect project root
lsp-log-io nil
lsp-enable-indentation t
lsp-enable-imenu t
lsp-keymap-prefix "C-l"
lsp-file-watch-threshold 500
lsp-prefer-flymake nil)      ; Use lsp-ui and flycheck

  (defun lsp-on-save-operation ()
(when (or (boundp 'lsp-mode)
     (bound-p 'lsp-deferred))
  (lsp-organize-imports)
  (lsp-format-buffer))))

(use-package lsp-clients
  :ensure nil
  :after (lsp-mode)
  :init (setq lsp-clients-python-library-directories '("/usr/local/" "/usr/")))

(use-package lsp-ui
  :ensure t
  :after (lsp-mode)
  :commands lsp-ui-doc-hide
  :bind (:map lsp-ui-mode-map
     ([remap xref-find-definitions] . lsp-ui-peek-find-definitions)
     ([remap xref-find-references] . lsp-ui-peek-find-references)
     ("C-c u" . lsp-ui-imenu))
  :init (setq lsp-ui-doc-enable t
     lsp-ui-doc-use-webkit nil
     lsp-ui-doc-header nil
     lsp-ui-doc-delay 0.2
     lsp-ui-doc-include-signature t
     lsp-ui-doc-alignment 'at-point
     lsp-ui-doc-use-childframe nil
     lsp-ui-doc-border (face-foreground 'default)
     lsp-ui-peek-enable t
     lsp-ui-peek-show-directory t
     lsp-ui-sideline-update-mode 'line
     lsp-ui-sideline-enable t
     lsp-ui-sideline-show-code-actions t
     lsp-ui-sideline-show-hover nil
     lsp-ui-sideline-ignore-duplicate t)
  :config
  (add-to-list 'lsp-ui-doc-frame-parameters '(right-fringe . 8))

  ;; `C-g'to close doc
  (advice-add #'keyboard-quit :before #'lsp-ui-doc-hide)

  ;; Reset `lsp-ui-doc-background' after loading theme
  (add-hook 'after-load-theme-hook
   (lambda ()
     (setq lsp-ui-doc-border (face-foreground 'default))
     (set-face-background 'lsp-ui-doc-background
              (face-background 'tooltip))))

  ;; WORKAROUND Hide mode-line of the lsp-ui-imenu buffer
  ;; @see https://github.com/emacs-lsp/lsp-ui/issues/243
  (defadvice lsp-ui-imenu (after hide-lsp-ui-imenu-mode-line activate)
(setq mode-line-format nil)))

;; Debug
(use-package dap-mode
  :diminish dap-mode
  :ensure t
  :after (lsp-mode)
  :functions dap-hydra/nil
  :bind (:map lsp-mode-map
     ("<f5>" . dap-debug)
     ("M-<f5>" . dap-hydra))
  :hook ((dap-mode . dap-ui-mode)
(dap-session-created . (lambda (&_rest) (dap-hydra)))
(dap-terminated . (lambda (&_rest) (dap-hydra/nil)))))

(use-package lsp-treemacs
  :after (lsp-mode treemacs)
  :ensure t
  :commands lsp-treemacs-errors-list
  :bind (:map lsp-mode-map
     ("M-9" . lsp-treemacs-errors-list)))

(use-package treemacs
  :ensure t
  :commands (treemacs)
  :after (lsp-mode))

Common Lisp Grip, Project Updates, and Progress

Sun, 12 Jul 2020 00:00:00 +0000

Last week, I did a release, I guess, of cl-grip which is a logging library that I wrote after reflecting on common lisp logging earlier. I wanted to write up some notes about it that aren’t covered in the read me, and also talk a little bit4 about what else I’m working on.

cl-grip

This is a really fun and useful project and it was really the right size for a project for me to really get into, and practice a bunch of different areas (packages! threads! testing!) and I think it’s useful to boot. The read me is pretty comprehensive, but I thought I’d collect some additional color here:

Really at it’s core cl-grip isn’t a logging library, it’s just a collection of interfaces that make it easy to write logging and messaging tools, which is a really cool basis for an idea, (I’ve been working on and with a similar system in Go for years.)

As result, there’s interfaces and plumbing for doing most logging related things, but no actual implementations. I was very excited to leave out the “log rotation handling feature,” which feels like an anachronism at this point, though it’d be easy enough to add that kind of handler in if needed. Although I’m going to let it stew for a little while, I’m excited to expand upon it in the future:

additional message types, including capturing stack frames for debugging, or system information for monitoring.
being able to connect and send messages directly to likely targets, including systemd’s journal and splunk collectors.
a collection of more absurd output targets to cover “alerting” type workloads, like desktop notifications, SUMP, and Slack targets.

I’m also excited to see if other people are interested in using it. I’ve submitted it to Quicklisp and Ultralisp, so give it a whirl!

See the cl-grip repo on github.

Eggqulibrium

At the behest of a friend I’ve been working on an “egg equilibrium” solver, the idea being to provide a tool that can given a bunch of recipes that use partial eggs (yolks and whites) can provide optimal solutions that use a fixed set of eggs.

So far I’ve implemented some prototypes that given a number of egg parts, attempt collects recipes until there are no partial eggs in use, so that there are no leftovers. I’ve also implemented the “if I have these partial eggs, what can I make to use them all.” I’ve also implemented a rudimentary CLI interface (that was a trip!) and a really simple interface to recipe data (both parsing from a CSV format and an in memory format that makes solving the equilibrium problem easier.)

I’m using it as an opportunity to learn different things, and find a way to learn more about things I’ve not yet touched in lisp (or anywhere really,) so I’m thinking about:

building a web-based interface using some combination of caveman, parenscript, and related tools. This could include features like “user submitted databases,” as well as links to the sources the recpies, in addition to the basic “web forms, APIs, and table rendering.”
storing the data in a database (probably SQLite, mostly) both to support persistence and other more advanced features, but also because I’ve not done database things from Lisp at all.

See the eggquilibrium repo on github it’s still pretty rough, but perhaps it’ll be interesting!'

Other Projects

Writing more! I’m trying to be less obsessive about blogging, as I think it’s useful (and perhaps interesting for you all too.) I’ve been writing a bunch and not posting very much of it. My goal is to mix sort of grandiose musing on technology and engineering, with discussions of Lisp, Emacs, and programming projects.
Working on producing texinfo output from cl-docutils! I’ve been toying around with the idea of writing a publication system targeted at producing books--long-form non-fiction, collections of essays, and fiction--rather than the blogs or technical resources that most such tools are focused on. This is sort of part 0 of this process.
Hacking on some Common Lisp projects, I’m particularly interested in the Nyxt and StumpWM.

Common Lisp and Logging

Sat, 04 Jul 2020 00:00:00 +0000

I’ve made the decision to make all of personal project code that I write to do in Common Lisp. See this post for some of the background for this decision.

It didn’t take me long to say “I think I need a logging package,” and I quickly found this wonderful comparsion of CL logging libraries, and only a little longer to be somewhat disappointed.

In general, my requirements for a logger are:

straightforward API for logging.
levels for filtering messages by importance
library in common use and commonly available.
easy to configure output targets (e.g. system’s journal, external services, etc).
support for structured logging.

I think my rationale is pretty clear: loggers should be easy to use because the more information that can flow through the logger, the better. Assigning a level to all log messages is great for filtering practically, and it’s ubiquitous enough that it’s really part of having a good API. While I’m not opposed to writing my own logging system,¹ but I think I’d rather not in this case: there’s too much that’s gained by using the conventional choice.

Configurable outputs and structured logging are stretch goals, but frankly are the most powerful features you can add to a logger. Lots of logging work is spent crafting well formed logging strings, when really, you just want some kind of arbitrary map and makes it easier to make use of logging at scale, which is to say, when you’re running higher workloads and multiple coppies of an application.

Ecosystem

I’ve dug in a bit to a couple of loggers, sort of using the framework above to evaluate the state of the existing tools. Here are some notes:

log4cl

My analysis of the CL logging packages is basically that log4cl is the most mature and actively maintained tool, but beyond the basic fundamentals, it’s “selling” features are… interesting.² The big selling features:

integration with the developers IDE (slime,) which makes it possible to use the logging system like a debugger, almost. This is actually a phenomenal feature, particularly for development and debugging. The downside is that it wouldn’t be totally unreasonable to use it production, and that’s sort of terrifying.
it attempts to capture a lot of information about logging call sites so you can better map back from log messages to the state of the system when the call was made. Again, this makes it a debugging tool, and that’s awesome, but it’s overhead, and frankly I’ve never found it difficult to grep through code.
lots of attention to log rotation and log file management. There’s not a lot of utility in writing log data to files directly. In most cases you want to write to standard out: the program is being used interactively, and users may want to see the log of what happens. In cases where you’re running in a daemon mode, any more you’re not, systemd or similar just captures your output. Even then, you’re probably in a situation where you want to send the log output to some other process (e.g. an external service, or some kind of local socket for fluentd/etc.)
hierarchical organization of log messages is just less useful than annotation with metadata, in practice, and using hierarchical methods to filter logs into different streams or reduce logging volumes just obscures things and makes it harder to understand what’s happening in a system.

Having said that, the API surface area is big, and it’s a bit confusing to just start using the logger.

a-cl-logger

The acl package is pretty straightforward, and has a lot of features that I think are consistent with my interests and desires:

support for JSON output,
internal support for additional output formats (e.g. logstash,)
more simple API

It comes with a couple of pretty strong draw backs:

there are limited testing.
it’s SBCL only, because it relies on SBCL fundamentals in collecting extra context about log messages. There’s a pending pull request to add ECL compile support, but it looks like it wouldn’t be quite that simple.
the overhead of collecting contextual information comes at an overhead, and I think logging should err on the side of higher performance, and making expensive things optional, just because it’s hard to opt into/out of logging later.

Conclusion

So where does that leave me?

I’m not really sure.

I created a scratch project to write a simple logging project, but I’m definitely not prioritizing working on that over other projects. In the mean time I’ll probably end up just not logging very much, or maybe giving log4cl a spin.

Notes

When I started writing Go I did this, I wrote a logging tool, for a bunch of reasons. While I think it was the right decision at the time, I’m not sure that it holds up. Using novel infrastructure in projects makes integration a bit more complicated and creates overhead for would be contributors. ↩︎
To be honest, I think that log4cl is a fine package, and really a product of an earlier era, and it makes the standard assumptions about the way that logs were used, that makes sense given a very different view of how services should be operated. ↩︎

A Common Failure

Fri, 03 Jul 2020 00:00:00 +0000

I’ve been intermittently working on a common lisp library to produce a binary encoding of arbitrary objects, and I think I’m going to be abandoning the project. This is an explanation of that decision and an reflection on my experience.

Why Common Lisp?

First, some background. I’ve always thought that Common Lisp is a language with a bunch of cool features and selling points, but unsurprisingly, I’ve never really had the experience of writing more than some one-off bits of code in CL, which isn’t surprising. This project was a good experience for really digging into writing and managing a conceptually larger project which was a good kick in the pants to learn more.

The things I like:

the implementations of the core runtime are really robust and high quality, and make it possible to imagine running your code in a bunch of different contexts. Even though it’s a language with relatively few users, it feels robust in a way. The most common implementations also have ways of producing fully self contained static binaries (like Go, say), which makes the thought of distributing software seem reasonable.
quicklisp, a package/library management tool is new (in the last decade or so,) has really raised the level of the ecosystem. It’s not as complete as I’d expect in many ways, but quicklisp changed CL from something quaint to something that you could actually imagine using.
the object system is really nice. There isn’t quite compile time-type checking on the values of slots (attributes) of objects, though you can opt in. My general feeling is that I can pretty easily get the feeling of writing statically typed code with all of the freedom of writing dynamic code.
multiple dispatch, and the conceptual approach to genericism, is amazing and really simplifies flow control in a lot of cases. You implement the methods you need, for the appropriate types/objects and then just write the logic you need, and the function call machinery just does the right thing. There’s surprisingly little conditional logic, as a result.

Things I don’t like:

there are all sorts of things that don’t quite have existing libraries, and so I find myself wanting to do things that require more effort than necessary. This project to write a binary encoding tool would have been a component in service of a much larger project. It’d be nice if you could skip some of the lower level components, or didn’t have your design choices so constrained by gaps in infrastructure.
at the same time, the library ecosystem is pretty fractured and there are common tools around which there aren’t really consensus. Why are there so many half-finished YAML and JSON libraries? There are a bunch of HTTP server (!) implementations, but really you need 2 and not 5?
looping/iteration isn’t intuitive and difficult to get common patterns to work. The answer, in most cases is to use (map) with lambdas rather than loops, in most cases, but there’s this pitfall where you try and use a (loop) and really, that’s rarely the right answer.
implicit returns seem like an over sight, hilariously, Rust also makes this error. Implicit returns also make knowing what type a function or method returns quite hard to reason about.

Writing an Encoder

So the project I wrote was an attempt to write really object oriented code as a way to writing an object encoder to a JSON-like format. Simple enough, I had a good mental model of the format, and my general approach to doing any kind of binary format processing is to just write a crap ton of unit tests and work somewhat iteratively.

I had a lot of fun with the project, and it gave me a bunch of key experiences which make me feel comfortable saying that I’m able to write common lisp even if it’s not a language that I feel maximally comfortable in (yet?). The experiences that really helped included:

producing enough code to really have to think about how packaging and code organization worked. I’d written a function here and there, before, but never something where I needed to really understand and use the library/module/packaging (e.g. systems and libraries.) infrastructure.
writing and running lots of tests. I don’t always follow test-driven development closely, but writing lots of tests is part of my process, and being able to watch the layers of this code come together was a lot of fun and very instructive.
this project for me, was mostly about API design and it was nice to have a project that didn’t require much design in terms of the actual functionality, as object encoding is pretty straight forward.

From an educational perspective all of my goals were achieved.

Failure Mode

The problem is that it didn’t work out, in the final analysis. While the library I constructed was able to encode and decode objects and was internally correct, I never got it to produce encoding that other implementations of the same specification could reliably read, and the ability to read data encoded by other libraries only worked in trivial cases. In the end:

this was mostly a project designed to help me gain competence in a programming language I don’t really know, and in that I was successful.
adding this encoding format isn’t particularly useful to any project I’m thinking of working on in the short term, and doesn’t directly enable me to do anything in particular.
the architecture of the library would not be particularly performant in practice, as the encoding process didn’t deploy a buffer pool of any kind, and it would have been harder than not to back fill that in, and I wasn’t particularly interested in that.
it didn’t work, and the effort to debug the issue would be more substantive than I’m really interested in doing at this point, particularly given the limited utility.

So here we are. Onto a different project!

There's No Full Stack

Sun, 26 Apr 2020 00:00:00 +0000

Software engineers use terms like “backend” and “frontend” to describe areas of expertice and focus, and the thought is that these terms map roughly onto “underlying systems and business logic” and “user interfaces.” The thought, is that these are different kinds of work and no person can really specialize on “everything.”

But it’s all about perspective. Software is build in layers: there are frontends and backends at almost every level, so the classification easily breaks down if you look at it too hard. It’s also the case that that logical features, from the perspective of the product and user, require the efforts of both disciplines. Often development organizations struggle to hand projects off between groups of front-end and back-end teams.¹

Backend/Frontend is also a poor way to organize work, as often it forces a needless boundary between people and teams wokring on related projects. Backend work (ususally) has to be completed first, and if that slips (or estimation is off) then the front end work has to happen in a crunch. Even if timing goes well, it’s difficult to maintain engineering continuity through the handoff and context is often lost in the process.

In response to splitting projects and teams into front and backend, engineers have developed this idea of “full stack” engineering. This typically means “integrated front end and backend development.” A noble approach: keep the same engineer on the project from start to finish, and avoid an awkward handoff or resetting context halfway through a project. Historic concerns about “front end and backend being in different languages” are reduced both by the advent of back-end javascript, and a realization that programmers often work in multiple languages.

While full stack sounds great, it’s a total lie. First, engineers by and large cannot maintain context on all aspects of a system, so boundaries end up appearing in different places. A full stack engineer might end up writing front end and the APIs on the backed that the front end depends on, but not the application logic that supports the feature. Or an engineer might focus only a very specific set of features, but not be able to branch out very broadly. Second, specialization is important for allowing engineers to focus and be productive, and while context switching projects between engineers, having engineers that must context switch regularly between different disciplines is bad for those engineers. In short you can’t just declare that engineers will be able to do it all.'

Some, particularly larger, teams and prodcuts can get around the issue entirely by dividing ownership and specialization along functional boundaries rather than by engineering discipline, but there can be real technical limitations, and getting a team to move to this kind of ownership model is super difficult. Therefore, I’d propose a different organization or a way of dividing projects and engineering that avoids both “frontend/backend” as well as the idea of “full stack”:

feature or product engineers, that focus on core functionality delivered to users. This includes UI, supporting backend APIs, and core functionality. The users of these teams are the users of the product. These jobs have the best parts of “full stack” type orientation, but draw an effective “lower” boundary of responsibility and allow feature-based specialization.
infrastructure or product platform engineers, that focus on deployment, operations and supporting internal APIs. These teams and engineers should see their users as feature and product engineers. These engineers should fall somewhere between “backend engineers,” and the “devops” and “sre” -type roles of the last decade, and cover the area “above” systems (e.g. not inclusive of machine management and access provisioning,) and below features.

This framework helps teams scale up as needs and requirements change: Feature teams can be divided and parallelized and focus in functionality slices, while, infrastructure teams divide easily into specialties (e.g. networking, storage, databases, internal libraries, queues, etc.) and along service boundaries. Teams are in a better position to handle continuity of projects, and engineers can maintain context and operate using more agile methods. I suspect that, if we look carefully, many organizations and teams have this kind of de facto organization, even if they use different kind of terminology.

Thoughts?

In truth this problem of coordination between frontend and backend teams is really that it forces a waterfall-like coordination between teams, which is always awkward. The problem isn’t that backend engineers can’t write frontend code, but that having different teams requires a handoff that is difficult to manage correctly, and around that handoff processes and management happens. ↩︎

Experimental Sweater Pattern

Tue, 24 Dec 2019 00:00:00 +0000

I wrote this post nearly 5 years ago, and have been sitting on the draft for a long time: not for any reason, I think it’s actually pretty good post. For non-knitters, this is kind of a “ask a great cook for their comfort food recpie,” but in a narrative form.

In any case, I haven’t really been knitting very much recently, and while I enjoy writing knitting patterns there’s a lot of work in writing a well formed knitting pattern that I’m poorly positioned for ring now (test knitting! good photography! talking with knitters!) But, perhaps someone will find it useful… Enjoy!

Part of my recent return to knitting has been about taking a much more simple approach to yarn. I think yarn is cool and working with good yarn is awesome but at a certain point, I think yarn distracts from the things that I like most about knitting: the consistency, the dependability, the rhythm of the activity, and coordination of parallel activities.

Novel yarns and yarn variety actually makes the process of knitting less enjoyable for me. It also doesn’t really jive with my taste in clothing: I like plain things that fit well without a lot of adornment. While I enjoy knitting patterned sweaters for the rhythm, I don’t really wear them much. I also, live and spend my time in a climate where a I’m almost always wearing a light sweater (during the cold months) and inside during the rest of the months.

The result of this is that I’ve mostly been working on sock knitting. I like wearing wool socks, and after a period of not wearing them for a few reasons, I didn’t actually have that many wool socks. Which has lead me to get acclimated to knitting fingering weight yarn with size 0 needles.

So I want to make a sweater in this mold: fingering weight, very plain lines, probably knit in the round using the Elizabeth Zimmerman system. Starting from the bottom, I’ve been leaning away from ribbings at the bottom, and have tended to like hems though they sometimes flare. Ususally, I just cast on provisionally and add the hem (or whatever) at the end anyway. There’s time yet to decide.

More importantly, I’m quite interested in having a rolled collar for the neck, but I tend to think that rolled ends mostly have a flare look anyway. I can defer this decision for a while.

For shaping and even most of the styling I intend to copy the Chrome Cobra Zip Up, which is, by far, my favorite article of clothing.

I think really subtle increases (so that the sweater tapers to the waist) is a good feature and might choose to do some of those, particularly if the model sweater has them. The model sweater has a really long back, and I think I might moderate this slightly.

The shoulders are an open question. If this goes well, I think I’d like to knit my way though most of the standard EZ shoulder constructions: I think I’ve knit all of the options at least once, but I’ve not done all of them in plain knitting, and most of the sweaters are a bit odd in one way or another. There’s a long project. I want to start with, and hopefully master, the set-in-sleeve.

For those of you playing along at home, set-in-sleeves are probably what you think of as “normal” sleeves, the garment fits in the shoulders, and the sleeves angle gently down from the shoulders. Most shirts have this shaping but the shapes aren’t terribly natural for knitting.

To knit set in sleeves in the round, you join the sleeves to the body, setting some stitches aside where the pieces meet and then decreasing body snitches into the sleeves as you knit until the body is just as wide as the shoulders. Then decrease the sleeve stitches into the body until you have about 3 inches of sleeve stitches left. Finally, knit knit short rows across the front and back (or just the front) stitches, decreasing the remaining stitches in the short rows ending with a 3 needle bind-off at the appropriate moment. To get a good crew neck, begin shaping the front of the neck every row 1.5 inches before you start the shoulder short-rows, and shape the back of the neck every other row when you start the shoulder short rows.

The shaping and body of the knitting is pretty straightforward from design perspective. The hard part from the perspective of the success of the sweater is the hems and/or ribbing, and figuring out the right thing to for each hem. It’s always something.

What is it That You Do?

Tue, 24 Dec 2019 00:00:00 +0000

The longer that I have this job, the more difficult it is to explain what I do. I say, “I’m a programmer,” and you’d think that I write code all day, but that doesn’t map onto what my days look like, and the longer it seems the less code I actually end up writing. I think the complexity of this seemingly simple question grows from the fact that building software involves a lot more than writing code, particularly as projects become more complex.

I’d venture to say that most code is written and maintained by one person, and typically used by a very small number of pepole (often on behalf of many more people,) though this is difficult to quantify. Single maintainer software is still software, and there are lots of interesting problems, but as much as anything else I’m interested in the problems adjacent to multi-author code-bases and multi-operator software development.¹

Fundamentally, I’m interested in the following questions:

How can (sometimes larger) groups of people collaborate to build something that’s bigger than the scope of any of their work?
How can we build software in a way that lets individual developers focus most of the time on the features and concerns that are the most important to them and their users.²

The software development process, regardless of the scope of the problem, has a number of different aspects:

Operations: How does is this software execute and how do we know that its successful when it runs?
Behavior: What does it do, and how do we ensure it has the correct behavior?
Interface: How will users interact with the process, and how do we ensure a consistent experience across versions and users' environment?
Product: Who are the users? What features do they want? Which features are the most important?

Sometimes we can address these questions by writing code, but often there’s a lot of talking to users, other developers, and other people who work in software development organizations (e.g. product managers, support, etc.) not to mention writing a lot of English (documentation, specs, and the like.)

I still don’t think that I’ve successfully answered the framing question, except to paint a large picture of what kinds of work goes into making software, and described some of my specific domain interests. This ends up boiling down to:

I write a lot of documents describing new features and improvements to our software. [product]
I think a lot about how our product works as it grows (scaling), and what kinds of changes we can make now to make that process more smooth. [operations]
How can I help the more junior members of my team focus on the aspects of their jobs that they enjoy the most, and help illustrate broader contexts to them. [mentoring]
How can we take the problems we’re solving today and build the solution that balances the immediate requirements with longer term maintainability and reuse. [operations/infrastructure]

The actual “what” I’m spending my time boils down to reading a bunch of code, meeting with my teamates, meeting with users (who are also coworkers.) And sometimes writing code. If I’m lucky.

I think the single-author and/or single-operator class is super interesting and valuable, particularly because it includes a lot of software outside of the conventional disciplinary boundaries of software and includes things like macros, spreadsheets, small scale database, and IT/operations (“scripting”) work. ↩︎
It’s very easy to spend most of your time as a developer writing infrastructure code of some sort, to address either internal concerns (logging, data management and modeling, integrating with services) or project/process automation (build, test, operations) concerns. Infrastructure isn’t bad, but it isn’t the same as working on product features. ↩︎

The Case for Better Build Systems

Sat, 20 Jul 2019 00:00:00 +0000

A lot of my work, these days, focuses on figuring out how to improve how people develop software in ways that reduces the amount of time developers have to spend doing work outside of development and that improves the quality of their work. This post, has been sitting in my drafts folder for the last year, and does a good job of explaining how I locate my workand* makes a case for high quality generic build system tooling that I continue to feel is compelling.*

Incidentally, it turns out that I wrote an introductory post about buildsystems 6 years ago. Go past me.

Canonically, build systems described the steps required to produce artifacts, as system (graph) of dependencies¹ and these dependencies are between source files (code) and artifacts (programs and packages) with intermediate artifacts all in terms of the files they are or create. Though different development environments, programming languages, and kinds of software have different.

While the canonical “build systems are about producing files,” the truth is that the challenge of contemporary _software development isn’t really just about producing files. Everything from test automation to deployment is something that we can think about as a kind of build system problem.

Let’s unwind for a moment. The work of “making software,” breaks down into a collection of--reasonably disparate--tasks, which include:

collecting requirements (figuring out what people want,)
project planning (figuring out how to break larger collections of functionality into more reasonable units.)
writing new code in existing code bases.
exploring unfamiliar code and making changes.
writing tests for code you’ve recently written, or areas of the code base that have recently chaining.
rewriting existing code with functionally equivalent code (refactoring,)
fixing bugs discovered by users.
fixing bugs discovered by an automated test suite.
releasing software (deploying code.)

Within these tasks developers do a lot of little experiments and tests. Make a change, see what it’s impact is by doing something like compiling the code, running the program or running a test program. The goal, therefore, of the project of developer productivity projects is to automate these processes and shorten the time it takes to do any of these tasks. In particular the feedback loop between “making a change” and seeing if that change had an impact. The more complex the system that you’re developing, with regards to distinct components, dependencies, target platforms, compilation model, and integration’s, them the more time you spend in any one of these loops and the less productive you can be.

Build systems are uniquely positioned to suport the development process: they’re typically purpose built per-project (sharing common infrastructure,) most projects already have one, and they provide an ideal environment to provide the kind of incremental development of additional functionality and tooling. The model of build systems: the description of processes in terms of dependency graphs and the optimization for local environments means.

The problem, I think, is that build systems tend to be pretty terrible, or at least many suffer a number of classic flaws:

implicit assumptions about the build or development environment which make it difficult to start using.
unexpressed dependencies on services or systems that the build requires to be running in a certain configuration.
improperly configured dependency graphs which end up requiring repeated work.
incomplete expression of dependencies which require users to manually complete operations in particular orders.
poorly configured defaults which make for overly complex invocations for common tasks.
operations distributed among a collection of tools with little integration so that compilation, test automation, release automation, and other functions.

By improving the quality, correctness, and usability of build systems, we:

improve the experience for developers every day,
make it really easy to optimize basically every aspect of the development process,
reduce the friction for including new developers in a project’s development process.

I’m not saying “we need to spend more time writing build automation tools” (like make, ninja, cmake, and friends,) or that the existing ones are bad and hard to use (they, by and large are,) but that they’re the first and best hook we have into developer workflows. A high quality, trustable, tested, and easy to use build system for a project make development easier, continuous integration easy and maintainable, and ultimately improve the ability of developers to spend more of their time focusing on important problems.

ideally build systems describe directed acylcic graph, though many projects have buildsystems with cyclic dependency graphs that they ignore in some way. ↩︎

Observation at Scale

Fri, 19 Jul 2019 00:00:00 +0000

I wrote this thing about monitoring in software, and monitoring in web applications (and similar) about a year ago, and I sort of forgot about it, but as I was cleaning up recently I found this, and think that I mostly still agree with the point. Enjoy!

It is almost always the case that writing software that does what you want it to do is the easy part and everything else is the hard part.

As your software does more a number of common features emerge:

other people are responsible for operating your software.
multiple instances of the program are running at once, on different computers.
you may not be able to connect to all of the running instances of the program when something goes wrong.
people will observe behaviors that you don’t expect and that you won’t be able to understand by observing the program’s inputs or outputs''

There are many things you can do to take a good proof of concept program and turn it into a production-ready program, but I think logging and introspection abilities are among the most powerful: they give you the most bang for your buck, as it were. It’s also true that observability (monitoring) is a hot area of software development that’s seeing a lot of development and thought at the moment.

While your application can have its own internal reporting system, its almost always easier to collect data in logs first rather than

Aggregate Logs

Conventionally operators and developers interact with logs using standard unix stream processing tools: tail, less, and grep and sometimes wc, awk, and sed. This is great when you have one (or a small number) process running on one machine. When applications get bigger, stream processing begins to break down.

Mostly you can’t stream process because of volume there’s too much data, it’s hard to justify spending disk space on all of your application servers on logs, and there’s too much of it to look at and do useful things. It’s also true that once you have multiple machines, its really helpful to be able to look at all of the logs in a single place.

At the lowest level the syslog protocol and associated infrastructure solves this problem by providing a common way for services to send log data via a network (UDP, etc.) It works but you still only have stream processing tools, which may be fine, depending on your use case and users.

Additionally there are services and applications that solve this problem: splunk (commercial/enterprise software ) sumologic (commercial/cloud software) and the ELK stack (an amalgamation of open source tools.) that provide really powerful ways to do log search, reporting, and even build visualizations. There are probably others as well.

Use them.

Structure Logs

The most common interview question for systems administrators that my colleagues give is a “log sawing” question. This seems pretty standard, and is a pretty common exercise for parsing information out of well known streams of log data. Like “find a running average request time,” or figure out the request rate.

The hard part is that most logs, in this example are unstructured in the sense that they are just line-wise printed strings, and so the exercise is in figuring out the structure of the messages, parsing data from the string, and then tracking data over the course of the logs. Common exercise, definitely a thing that you have to do, and also totally infuriating and basically impossible to generalize.

If you’re writing software, don’t make your users do this kind of thing. Capture events (log messages) in your program and output them with the information already parsed. The easiest way is to make your log messages mapping types, and then write them out in JASON, but there are other options.

In short, construct your log messages so that they’re easy to consume by other tools: strongly (and stably) type your messages, provide easy way to group and filter similar messages. Report operations in reasonable units (e.g. seconds rather than nanoseconds) to avoid complex calculations during processing, and think about how a given data point would beinteresting to track over time.

Annotate Events

Building on the power of structured logs, it’s often useful to be able to determine the flow of traffic or operations through the system to make it possible to understand the drivers of different kinds of load, and the impact of certain kinds of traffic on overall performance. Because a single operation may impact multiple areas of the system, annotating messages appropriately makes it possible to draw more concrete conclusions based on the data you collect.

For example when a client makes a user request for data, your system probably has a request-started and request-ended event. In addition this operation may retrieve data, do some application-level manipulation, modify other data, and then return it to the user. If there’s any logging between the start and end of a request, then it’s useful to tie these specific events together, and annotations can help.

Unlike other observability strategies, there’s not a single software feature that you can use to annotate messages once you have structured capabilities, although the ability of your logging systems to have some kind of middleware to inject annotations is quite useful.

Collect Metrics

In addition to events produced by your system, it may be useful to have a background data collection thread to report on your application and system’s resource utilization. Things like, runtime resource utilization, garbage collector stats, and system IO/CPU/Memory use can all be useful.

There are ways to collect this data via other means, and there are a host of observability tools that support this kind of metrics collection. But using multiple providers complicates actually using this data, and makes it harder to understand what’s going in the course of running a system. If your application is already reporting other stats, consider bundling these metrics in your existing approach.

By making your application responsible for system metrics you immediately increase the collaboration between the people working on development and operations, if such a divide exists.

Conclusion

In short:

collect more data,
increase the fidelity and richness of the data you collect,
aggregate potentially related data in the same systems to maximize value,
annotate messages to add value, and provide increasingly high level details.

Citation Practices in Informal Writing

Thu, 18 Jul 2019 00:00:00 +0000

I’ve been thinking recently about the way that my interest in writing and publishing blog posts has waned, and while there are a lot of factors that have contributed to this, I think there’s a big part of me that questions what the purpose of writing is or should be, and because I mostly write about the things I’m learning or thinking about, my posts end up being off-the-cuff explanations of things I’ve learned or musings on a theoretical point which aren’t particularly well referenced, and while they’re fun to write and useful for my own process they’re not particularly useful to anyone else. Realizing this puts me at something of a crossroads with my own writing, and has me thinking a lot about the practice of citation.¹

Mechanically, citation anchors text in relationship to other work,² but it also allows discussion to happen in and between texts. Also, the convention for citation in the context of informal writing is a link or an informal reference, so it’s difficult to track over time, and hard to be systemtic in the way that one text interacts with its sources.

Blogs bring out confessional writing with ambling³ structure and the freedom to say just about anything, which I have found liberating and generally instructive, but it’s also limiting. For writing that comes out of personal experience, it’s difficult to extrapolate and contextualize your argument, or even to form an argument, particularly in the context of a blog where you’re writing a larger number of shorter pieces. It’s also probably true that by framing discussions in personal experience its hard for people with different experiences to relate to the content, and more importantly the concepts within.

I’m not arguing against journaling: journals are greatgreat, but sometimes, I think journals might be best unpublished. I’m also not arguing against the personal essay as a form: there are many topics that are well served by that genre of writing. I do want to think about what else is possible⁴ and how to write things that are stronger, more grounded, and easier to relate to and interact with. I think more citations and references are the key, but I’m left with two problems:

Style. There aren’t great conventions for referencing things in informal writing. Throwing a link in the right context works, and is clear, but it might not be enough as it’s hard to know what’s a citation-typed-reference and other kinds of links. Also links don’t hold up well over time. The more formal approaches are rooted in out of date technologies and tactics. Citations often reference page numbers, footnotes don’t often make sense in informal situations,⁵ and bibliography conventions are mostly non-existant.
Tooling. I’m pretty sure that well cited texts are well-cited, because their authors have great memories for things they’ve read, but because researchers often have tooling that supports managing a database of references, notes and bibliographic information. If you have a record of the resources you’ve read (or otherwise consumed), it becomes easier to pull out citations as you write and edit.⁶

Neither of these are insurmountable, but I think would require a good deal of work both on figuring out better citation formats and patterns, as well as developing better tooling. I don’t have answers yet, but I do want to think more about it, and probably play with writing some tools.

My initial intent was to sort of discuss the personal conflict, and reflect on the corpus of this site and consider my own growth as a writer, which might have been a fine way to tell this story, but it felt much more self indulgent, and I think probably makes my point by example better than I am here. ↩︎
Admittedly this should be cited. ↩︎
Limited, of course, by size. ↩︎
The problem is that I think there aren’t a lot of great examples or models to follow. I’ve been thinking about other kinds of writing (e.g. journalism, academic writing, and fiction,) for potential models. The academic writing and journalism are clearly the starting points for this post. ↩︎
My attempts to the contrary aside. Having said that I expect that ↩︎
Zotero is probably the most popular one. There are tools that allow you to maintain BibTeX files, with similar effects. The space is probably underdeveloped, and most tooling is targeted at researchers in specific fields. It’s unfortunately a difficult space to develop a compelling tool in because the technology is easy (so it’s easy to overengineer,) and there are just enough users (and different kinds of users) to make the interface/interaction design problems non trivial. ↩︎

Things I Learned About Queues

Tue, 11 Sep 2018 00:00:00 +0000

I think the queue is a really powerful metaphor for organizing and orchestrating the internal architecture of an application. Once you have a queue, and tasks that are running in that queue, making that system run well requires some attention. This post is very much a sequel to the application framework post.

Factors of your queue implementation and system may impact the applicability of any of these suggestions for your particular application. Additionally, there is lots of work on queue theory so there are formally described properties of queues, and this is really just a collection of informal knowledge that I’ve collected on this subject. I hope you find it useful!

As the operator of a queue there are two properties: latency, or time to completion, for work in the queue and throughput, or total amount of work completed. These properties are generally trade-offs with each other, and often work to improve throughput will impact latency, and vice versa. It turns out, however, that the theoretical limits of your system’s capacity for either latency or throughput are below the actual requirements of your application, so you can generally just focus on improving one area or the other without really feeling like you’re trading latency for throughput.

All tasks in the queue should, generally, of similar size in terms of execution time and resource usage. When there are tasks that run slowly or take a long time and tasks that run quickly, you can easily end up in situations where long running tasks group together. Indeed, this isn’t just a possibility, but a near certainty. If you can’t break work into similar sized units, then you main recourse is to either separate the work into different queues and proportion resources as needed to ensure that both queues are making progress. You generally want to run longer tasks before shorter tasks, but the impact on overall performance depends on other characteristics and the way that your application expects certain kinds of latency and throughput.

Always monitor task runtime (by type,) as well as overall queue depth, and if possible currently running operations. When something goes wrong, or there’s an external event that impacts queue performance, you’ll need these data to understand the state of your world and debug the underlying issue. Don’t wait for something to go wrong to set this up.

Idempotentcy, or the ability of a task to run more than once without chaining the final outcome is a great property in any distributed system, but the more idempotent your operations are the less you have to worry about edge cases where you might run them more than once, particularly around process restarts and deployments. While you generally only want to run things once for efficiency sake, it’s important to be able to know that you can run things more than once without causing a crisis.

While it’s easy to think about the time a task spends waiting in a queue when tasks are ordered in the queue in a first-in-first-out model, other ordering mechanisms can easily lead to items getting stuck in the queue. Consider the following behaviors:

if there are dependencies between tasks, or cases where doing work leads to the creation of more tasks, always run these tasks earlier before other tasks.
consider boosting the relative priority of tasks that have been waiting longer relative to other tasks in the queue. If tasks have a priority, and new tasks come in that have higher priority than older tasks, then some lower priority tasks may never get done. While priority of tasks is important, if its important that all tasks get done, balance wait time with priority.
alternatively, consider elimiting tasks that have been waiting in the queue for longer than a specified period. These “TTL” for queue items can avoid wasting resources doing work that is not useful.
separate desperate priority or types of work into seperate queues to reduce latency. Having additional queues often incurs some per-queue/per-worker resource overhead. When worker infrastructure canbe shared between queues, and both queues are not consistently running at capacity (e.g. have backlogs).
quantify the job dispatching overhead. While generally breaking apart larger tasks into smaller execution units improves efficiency, if the overhead of creating, dispatching, and running jobs is a significant portion of a job’s runtime, then your system is probably spending too many resources on overhead and you can increase throughput by increasing the overall task size.

There’s more, but this definitely covers the basics.

Non-Trad Software Engineer

Mon, 30 Jul 2018 00:00:00 +0000

It happened gradually, and it wasn’t entirely an intentional thing, but at some point I became a software engineer. While a lot of people become software engineers, many of them have formal backgrounds in engineering, or have taken classes or done programs to support this retooling (e.g. bootcamps or programming institutes.)

I skipped that part.

I wrote scripts from time to time for myself, because there were things I wanted to automate. Then I was working as a technical writer and had to read code that other people had written for my job. Somewhere in there I was responsible for managing the publication workflow, and write a couple of build systems.

And then it happened.

I don’t think it’s the kind of thing that is right for everyone, but I was your typical, nerdy/bookish kid who wasn’t great in math class, and I suspect that making software is the kind of thing that a lot of people could do. I don’t think that my experience is particularly replicable, but I have learned a number of useful (and important) things, and I realize as I’ve started writing more about what I’m working on now, I realize that I’ve missed some of the fundamentals¹

Formal education in programming, from what I’ve been able to gather strikes me as really weird: there are sort of two main ways of teaching people about software and computer science: Option one is that you start with a very theoretical background that focuses on data structures, the performance of algorithms, or the internals of how core technologies function (operating systems, compilers, databases, etc.) Option two, is that you spend a lot of time learning about (a) programming language and about how to solve problems using programming.

The first is difficult, because the theory² is not particularly applicable except invery rare cases and only at the highest level which is easy to back-fill as needed. The second is also challenging, as idioms change between languages and most generic programming tasks are easily delegated to libraries. The crucial skill for programming is the ability to learn new languages and solve problems in the context of existing systems, and developing a curriculum to build those skills is hard.

The topics that I’d like to write about include:

Queue behavior, particularly in the context of distributed systems.
Observability/Monitoring and Logging, particularly for reasonable operations at scale.
build systems and build automation.
unit-testing, test automation, and continuous integration.
interface design for users and other programmers.
maintaining and improving legacy systems.

These are, of course, primarily focused on the project of making software rather than computer science or computing in the abstract. I’m particularly interested (practically) in figuring out what kinds of experiences and patterns are important for new programmers to learn, regardless of background.³ I hope you all find it interesting as well!

This is, at least in part, because I mostly didn’t blog very much during this process. Time being finite and all. ↩︎
In practice, theoretical insights come up pretty infrequently and are mostly useful for providing shorthand for characterizing a problem in more abstract terms. Most of the time, you’re better off intuiting things anyway because programming is predominantly a pragmatic exercise. For the exceptions, there are a lot of nerds around (both at most companies and on the internet) who can figure out what the proper name is for a phenomena and then you can look on wikipedia. ↩︎
A significant portion of my day-to-day work recently has involved mentoring new programmers. Some have traditional backgrounds or formal technical education and many don’t. While everyone has something to learn, I often find that because my own background is so atypical it can be hard for me to outline the things that I think are important, and to identify the high level concepts that are important from more specific sets of experiences. ↩︎

In Favor of an Application Infrastructure Framework

Fri, 27 Jul 2018 00:00:00 +0000

The byproduct of a lot of my work on Evergreen over the past few years has been that I’ve amassed a small collection of reusable components in the form of libraries that address important but not particularly core functionality. While I think the actual features and scale that we’ve achieved for “real” features, the infrastructure that we built has been particularly exciting.

It turns out that I’ve written about a number of these components already here, even. Though I think, my initial posts were about these components in their more proof-of-concept stage, now (finally!) we’re using them all in production so their a bit more hardened.

The first grip is a logging framework. Initially, I thought a high-level logging framework with plug-able backends was going to be really compelling. While configurable back-ends has been good for using grip as the primary toolkit for writing messaging and user-facing alerting, the most compelling feature has been structured logging.

Most of the logging that we do, now, (thanks to grip,) has been to pass structures (e.g. maps) to the logger with key/value data. In combination with log aggregation services/tools (like ELK, splunk, or sumologic,) we can basically take care of nearly all of our application observablity (monitoring) use cases in one stop. It includes easy to use system and golang runtime metrics collection, all using an easy push-based collection, and can also power alert escalation. After having maintained an application using this kind of event driven structured logging system, I have a hard time thinking about running applications without it.

Next we have amboy which is a queue-system. Like grip, all of the components are plug-able, so it support in-memory (ephemeral) queues, distributed queues, dependency graph systems and priority queue implementations as well as a number of different execution models. The most powerful thing that amboy affords us is a single and clear abstraction for defining “background” execution and workloads.

In go it’s easy to spin up a go routine to do some work in the background, it’s super easy to implement worker pools to parallelize the processing of simple tasks. The problem is that as systems grow, it becomes pretty hard to track this complexity in your own code, and we discovered that our application was essentially bifurcated between offline (e.g. background) and online (e.g. request-driven) work. To address all of this problem, we defined all of the background work as small, independent units of work, which can be easily tested, and as a result there is essentially no-adhoc concurrency in the application except what runs in the queues.

The end result of having a unified way to characterize background work is that scaling the application because much less complicated. We can build new queue implementations, without needing to think about the business logic of the background work itself, and we add capacity by increasing the resources of worker machines without needing to think about the architecture of the system. Delightfully, the queue metaphor is independent of external services, so we can run the queue in memory backed by a heap or hash map with executors running in dedicated go-routines if we want, and also scale it out to use databases or dedicated queue services with additional process-based workers, as needed.

The last component, gimlet, addresses building HTTP interfaces, and provides tools for registering routes, writing responses, managing middleware and authentication, an defining routes in a way that’s easy to test. Gimlet is just a wrapper around some established tools like negroni, gorilla/mux, all built on established standard-library foundations. Gimlet has allowed us to unify a bunch of different approaches to these problems, and has lowered the barrier to entry for most of our interfaces.

There are other infrastructural problems still on the table: tools for building inter-system communication and RPC when you can’t communicate via a queue or a shared database (I’ve been thinking a lot about gRPC and protocol buffers for this,) and also about object-mapping and database access patterns, which I don’t really have an answer for.¹

Nevertheless, with the observability, background tasks, and HTTP interface problems well understood at supported, it definitely frees developers to spend more of their time focused core problems of importance to users and the goals of the project. Which is a great place to be.

I built a database migration tool called anser which is mostly focused on integrating migration workflows into production systems so that migrations are part of the core code and can run without affecting production traffic, and while these tools have been useful, I haven’t seen a clear path between this project and meaningfully simplifying the way we manage access to data. ↩︎

Why Write

Thu, 26 Jul 2018 00:00:00 +0000

I’ve had a blog¹ for more than 15 years, and I’ve found this experience to be generally quite rewarding. I’ve learned a lot about writing, and enjoyed the opportunity to explore a number of different topics² in great detail. While I haven’t blogged as much in recent years, I’ve been thinking in the past few weeks about getting back into writing more regularly, which has lead me to reflect on my writing in the past and my goals for this in the future.

First, the blog as a genre has changed fundamentally in the last 17 or 18 years. In 2000 or 2001, blogs were independent things that grew out of communities (e.g. MetaFilter, or web-diary/journaling) and were maintained by independent writers or small groups. Then the tooling got better, the community got better and eventually started to segment based on topic, and finally the press³ gained competence in the form.

Publishing a blog today, is a vastly different proposition today even in the recent past.

True to my form, this leaves me with a collection of divergent thoughts:

maybe the “write everything in one blog even though the topics are not really of interest to any specific group” approach that I’ve always taken. More distinct blogs means more writing (maybe a good thing,)
having a writing practice is good for focusing thoughts, but also for sharing and distributing understating, and I think that sharing understanding is a really important part of learning and growing, and I miss having a structure for these kinds of notes.
perhaps, it would make sense to outsource/hire a freelancer to take care of some editing and marketing-adjacent work, which is more required if you want to engage with users more consistently but that I find distracting and outside of my ability to focus on properly. The problem then is figuring out how to fund that work in a longer-term/sustainable way.
In the past RSS has been a (the?) leading way to distribute content to serious readers, but that isn’t true now and likly hasn’t been true for years. So while I feel able to write a lot of things, I don’t know what the best way to engage with regular readers is
I used to think that I wanted to organize group blogs, and while I think that blog-discussions are fun, and I think there is merit to combined efforts, I’m less interested in doing the organizing myself.
There was a period where I wasn’t blogging much because my day job was very writing focused, and I needed side projects that didn’t involve the English language, and I spent a long time focusing on learning programming, which took a lot of time. Now that work mostly involves programming, and only a little writing, and I’ve had some time to recover as a writer, it feels like I have some space.
I’m not really sure how to host a blog in 2018. The old set up and server I have is more than functional, but there are a lot of services, tools, and patterns that I’m not familiar with and I have some learning to do, even though I probably mostly just want to write things.

In one form or another, though the archives are all here. ↩︎
I’ve written blogs about Philosophy, Hand Knitting, Technology, Documentation, Programming, Science Fiction, Folk Music and Dance, and Economics. ↩︎
Both old media institutions (news papers, television companies, book and magazine publishers) and new institutions that grew out of blogging itself (e.g. HuffPo, Gawker, etc.) ↩︎

Evergreen Intro

Wed, 25 Jul 2018 00:00:00 +0000

Almost two years ago, I switched teams at work to join the team behind evergreen which is a homegrown continuous integration tool that we use organization wide to host and support development efforts and operational automation. It’s been great.

From the high level, Evergreen takes changes that developers make to source code repositories and runs a set of tasks for each of those changes on a wide variety of systems, and is a key part of the system that allows us to verify that the software we write works on computers other than the ones that we interact with directly. There are a number of CI systems in the world, but Evergreen has a number of interesting features:

it runs tasks in parallel, fanning out tasks to a large pool of machines to shorten the “wall clock” time for task execution.
tasks execute on ephemeral systems managed by Evergreen in response to demands of incoming work.
the service maintains a queue of work and handles task dispatching and results collection.

This focus on larger scale task parallelism and managing host pools gives Evergreen the ability to address larger scale continuous integration workflows with a lower maintenance overhead. This is totally my jam: we get to both affect the development workflow and engineering policies for basically everyone and improving operational efficiency is a leading goal.

My previous gig was more operational, on a sibling team, so it’s been really powerful to be able to address problems relating to application scale and drive the architecture from the other side. I wrote a blog post for a work-adjacent outlet about the features and improvements, but this is my blog, and I think it’d be fun to have some space to explore “what I’ve been working on,” rather than focusing on Evergren as a product.

My first order of business, after becoming familiar with the code base, was to work on logging. When I started learning Go, I wrote a logging library (I even bloged about it), and using this library has allowed us to “get serious about logging.” While it was a long play, we now have highly structured logging which has allowed the entire logging system to become a centerpiece in our observably story, and we’ve been able to use centralized log aggregation services (and even shop around!) As our deployment grows, centralized logging is the thing that has kept everything together.

Recently, I’ve been focusing on how the application handles “offline” or background work. Historically the application has had a number of loosely coupled “cron-job” like operations that all happened on single machine at a regular interval. I’m focusing on how to move these systems into more tightly coupled, event-driven operations that can be distributed to a larger cluster of machines. Amboy is a big part of this, but there have been other changes related to this project.

On the horizon, I’m also starting to think about how to reduce the cost of exposing data and operations to clients and users in a way that’s lightweight and flexible, and relatively inexpensive for developer time. Right now there’s a lot of technical debt, a myriad of different ways to describe interfaces, and inconsistent client coverage. Nothing insurmountable, but definitely the next frontier of growing pains.

The theme here is “how do we take an application that works and does something really cool,” and turn it into a robust piece of software that can both scale as needs grown, but also provide a platform for developing new features with increasing levels of confidence, stability, and speed.

The conventional wisdom is that it’s easy to build features fast-and-loose without a bunch of infrastructure, and that as you scale the speed of feature development slows down. I’m pretty convinced that this is untrue and am excited to explore the ways that improved common infrastructure can reduce the impact of this ossification and lead to more nimble and expansive feature development.

We’ll see how it goes! And I hope to be able to find the time to write about it more here.

Combating Legacy Code

Tue, 24 Jul 2018 00:00:00 +0000

I wrote some notes about to write a post about a software project I worked on a year and a half ago, that I think is pretty cool, but I was on writing hiatus. Even better the specific code in question is now no longer in use. But I think it serves as a useful parable, but I will attempt to reflect.

Go’s logging¹ support in standard library works, and it successfully achieves its goals on its own terms. The problem is that it’s incredibly simple and lacks a number of features that are standard in most logging systems.² So as a result, I’m not surprised that most applications of consequence either use a couple of more fully featured logging packages or end up writing a large number of logging wrappers.

The fact that my project at work was using a special logging library is not particularly surprising, particularly because the project is old for a Go project. The logging library in question is a log4j-inspired package, that had been developed by a different group internally, but was no longer being used by that group. It worked, but there were a host of problems.³

I’d also written a logging package myself which was a definite improvement on the state of the art. I had two chief problems:

how to convince teammates to make the change,
how to make the change without disrupting ongoing work or the functioning of the system which had to be always deploy-able.

Here’s what I did…

First, I learned as much as I could about the existing system, it’s history and how we used it. I read a lot of code, documentation (such as it was,) and also related bug reports, feature requests, and history.

Second, I implemented wrappers for my system that (mostly) cloned the interfaces for the existing library in my own package. It’s called slogger, and it’s still there, though I hope to delete it soon. I wanted to make it possible to make the switch⁴ in the project initialization without needing to change every last logging statement.⁵

Then, we actually made the change so that logging used the new code internally but wrapped by the old interfaces. I think there were a couple of very obvious bugs early on, but frankly none of them are so memorable that I could describe them any more.

Finally, we went through and updated all of the logging statements. It was a big change, and impacted all of the code, but it happened quite late in the process and there were no bugs, because it was the least interesting or radical part of the project.

And then we had a new logger. It’s been great. With the new tool we’ve been able to easily add support for more structured approaches to logging and collecting log output in a variety of third party services.

In summary:

replacing legacy subsystems can be a good way to improve the functionality of your project.
change is hard, but there are ways to make changes easier and less disruptive. They often involve doing even more work.*
write code to facilitate transitions, and then delete it later.
the larger a change is, the less risky it should be. While there are lots of small-and-low risk changes you can make, the inverse should be true as rarely as possible.

This is to say, application logging facility. ↩︎
This includes filtering by log level, different formatting options, (semi) structured logging, conditional logging, buffering, and other options. ↩︎
Hilariously something in the way we were using the logger was tripping up the race detector. While the logger did a decent job of providing the file name and line number of the logging statement, it was pretty focused on printing content to a file/standard output. ↩︎
Potentially this should have been behind a feature flag, though I think I didn’t actually use a feature flag. ↩︎
The short version here is, “interfaces are great.” ↩︎

Consciousness Rising

Thu, 24 Nov 2016 00:00:00 +0000

The subtitle of this post should be “or, how the internet learned about intersectionality,” but while I love a good pretentious academic title, I don’t think that’s particularly representative of my intent here.

Sometime in the last 5 or 10 years, the popular discourse on justice on the internet learned about intersectionality. Which is great. Intersectionality, generally is the notion that a single identity isn’t sufficient to explain an individuals social experience particularly vis a vis privilege. Cool.

This is really crucial and really important for understanding how the world works, but for totally understandable and plain ways. People have a lot of different identities which lead to many different experiences, perspectives, and understandings. All of these identities, experiences, perspectives, and understandings interact with each other in a big complex system

Therefore our analysis of our experiences, thought, understandings, and identities, must explore identities (ET AL) not only on their own terms, but in conversation with each other and with other aspects of experience.

Intersectionality is incredibly important. It’s also incredibly useful as a critical tool because it makes it possible for our thought to reflect actual lived experiences and the way that various aspects of experience interact to create culture and society.¹

While intersectionality is an interesting and important concept that could certainly support an entire blog post, I’m more interested, the genealogy of this concept in the popular critical discourse.

I know that I read a lot about intersectionality in college (in 2004-2007), I know that the papers I read were at least 10 years old, and I know that intersectionality wasn’t an available concept to political conversations on the internet at the time in the way that it is now.²

Concepts take a long time, centuries sometimes, to filter into general awareness, so the delay itself isn’t particularly notable. Even the specific route isn’t that interesting in and for itself. Rather, I’m interested in how a concept proliferates and what is required for a concept to become available to a more popular discourse.

If interesectionality was an available concept in the academic literature, what changes and evolutions in thought--both about intersectionality, but in the context--needed to happen for that concept to become available more broadly.

I think it’s particularly exciting to trace the recent intelectual history of a specific concept in discourse, because it might give us insight into the next concepts that will help inform our discourse and things we can do to facilitate this process in the future for new concepts and perspectives.

As we understand the history of this proliferation, we can also understand its failures and inefficiencies and attempt to deploy new strategies that resolve those shortcomings.

A lot of arguments in favor of intersectional analysis and perspectives are political, and raise the very real critique that analysis that is not intersectional tends to recapitulate normative cultural assumptions. I’d argue, additionally, that intersectionality is really the only way to pull apart experiences and thoughts and understand fundamentally how culture works. It’s not just good politics, but required methodology for learning about our world and our lives. ↩︎
I admit that this post is based on the conceit that there was a point when the popular discourse (on the internet) was unaware of intersectionality followed linearly by another point where the concept of intersectionality was available generally. This isn’t how the dissemination of concepts into discourses work, and I’m aware that I’ve oversimplified the idea somewhat. This is more about the process of popularization. ↩︎

Cache Maintence

Fri, 11 Nov 2016 00:00:00 +0000

Twice this fall I’ve worked on code that takes a group of files and ensures that the total size of the files are less than a given size. The operation is pretty simple: identify all the files and their size (recursively, or not but accounting for the size of directories,) sort them, and and delete files from the front or back of the populated list. When you’ve reached the desired size.

If you have a cache and you’re constantly adding content to it, eventually you will either need an infinite amount of storage or you’ll have to delete something.

But what to delete? And how?

Presumably you use some items in the cache more often than others, and some files that change very often while others change very rarely, and in many cases, use and change frequency are orthogonal.

For the cases that I’ve worked on, the first case, frequency of use, is the property that we’re interested in. If we haven’t used a file in a while relative to the other files, the chances are its safe to delete.

The problem is that access time (atime) is that while most file systems have a concept of atime, most of them don’t update it. Which makes sense: if every time you read a file you have to update the metadata, then every read operation becomes a write operations, and everything becomes slow.

Relative access time or, relatime, helps some. Here atime is updated, but only if you’re writing to the file or if it’s been more than 24 hours since your last update. The problem, of course, is that if cache are write-once-read-many and operates with a time granularity of less than a day, then relatime is often just creation time. That’s no good.

The approach I’ve been taking is to use the last modification time, (mtime), and to intentionally update mtime (e.g. using touch or a similar operation,) after cache access. It’s slightly less elegant than it could be, but it works really well and requires very little overhead.

Armed with these decisions all you need is a thing that crawls a file system, collects objects and stores their size and time, so we know how large the cache is, and can maintain an ordered list of file objects by mtime. The ordered lists of files should be a heap, but the truth is that you build and sort the structure once, and then just remove the “lowest” (oldest) items until the cache is the right size and then throwing it all away, so you’re not really doing many heap-ish operations.

Therefore, I present lru. Earlier this summer I wrote a less generic implementation of the same principal, and was elbows deep into another project when I realized I needed another cache pruning tool. Sensing a trend, I decided to put a little more time into the project and built it out as a library that other people can use, though frankly I’m mostly concerned about my future self.

The package has two types, a Cache type that incorporates the core functionality and FileObject which represents items in the cache.

Operation is simple. You can construct and add items to the cache manually, or you can use DirectoryContents or TreeContents which build caches from a starting file system point. DirectoryContents looks at the contents of a single directory (skipping sub-directories optionally) and returns a Cache object with those contents. If you do not skip directories, each directory has, in the cache the total size of its contents.

TreeContents recurses through the tree and ignores directories, and returns a Cache object with all of those elements. TreeContents does not clean up empty directories.

Once you have a Cache object, use its Prune method with the maximum size of the cache (in bytes), any objects to exclude, and an optional dry-run flag, to prune the cache down until it’s less than or equal to the max size.

Done.

I’m not planning any substantive changes to the library at this time as it meets most of my needs but there are some obvious features:

a daemon mode where the cache object can “watch” a file system (using ionotify or similar) and add items to or update existing items in the cache. Potentially using fsnotify.
an option to delete empty directories encountered during pruning.
options to use other time data from the file system when possible, potentially using the times library.

With luck, I can go a little while longer without doing this again. With a little more luck, you’ll find lru useful.

Deleuze and Grove

Thu, 10 Nov 2016 00:00:00 +0000

I’ve been reading, two books non-fiction intermittently in the last little bit: Andy Grove’s High Output Management and Deleuze and Guatteri’s What is Philosophy?. Not only is reading non-fiction somewhat novel for me, but I’m sorting delighting in the juxtaposition. And I’m finding both books pretty compelling.

These are fundamentally materialist works. Grove’s writing from his experience as a manager, but it’s a book about organizing that focuses on personal and organizational effectiveness, with a lot of corporate high-tech company examples. But the fact that it’s a high-tech company that works on actually producing things, means that he’s thinking a lot about production and material constraints. It’s particularly interesting because the discussion technology and management often lead to popular writing that’s handwavey and abstract: this is not what Grove’s book is in the slightest.

Deleuze is more complex, and Guatteri definitely tempers the materialism, though less in the case of What is Philosophy than the earlier books. Having said that, I think What is Philosophy is really an attempt to both justify philosophy in and for itself, but also to discuss the project of knowledge (concept) creation in material, mechanistic terms.

To be honest this is the thing that I find the most compelling about Deleuze in general: he’s undeniably materialist in his outlook and approach, but but his work often--thanks to Guatteri, I think--focuses on issues central to non-materialist thought: interiority, subjectivity, experience, and identity. Without loosing the need to explore systems, mechanisms, and interfaces between and among related components and concepts.

I talked with a coworker about the fact that I’ve been reading both of these pieces together, and he said something to the effect of “yeah, Grove rambles a bunch but has a lot of good points, which is basically the same as Deleuze.” Fair. I’d even go a bit further and say that these are both books, that are despite their specialized topics and focus, are really deep down books for everyone, and guides for being in the world.

Read them both.

Isolation and Ideology Change

Thu, 10 Nov 2016 00:00:00 +0000

Following the 2016 election my father, who is a much more active participant in Facebook than I, said something to the effect of “don’t mourn; organize. I had a long winded post on the topic of ‘don’t celebrate; organize’, but the bottom line is the same: organize.”

I’d append to this just to make clear that I’m of the opinion that self care, survival and the care for and survival of our communities is crucial. Which sometimes means celebration and sometimes means mourning and sometimes means a quiet night at home with the and friends.

At the 2016 New England Sacred Harp Convention a friend of mine gave a lesion for those members of the community who were unable to attend because of profound illness which was delivered in conjunction with a lession in memorial for members of the community who had died in the last year. These lessons are a common and enduring tradition of Sacred Harp conventions.

The lesson focused on isolation, and the ways that illness, care-giving (and indeed dying, death, and grief) are isolating. But it went on to discuss the ways that we combat isolation, through connections to people and communities, and by the project of meaning making.

Connection and meaning making are related, of course, and are central to why I sing. I mean I also enjoy the music, but it’s the connection with other singers, and the ways that our practices in and around singing are about making meaning.

I heard this almost 6 weeks ago, but I keep coming back to this in a number of different contexts. There’s a lot in the world that either directly isolates, or provokes feelings of isolation.

Bottom line, the way that we can fight isolation is by forming connections and by working to create meaning in our lives.

I was talking on Wednesday with a couple of friends, one of who was most distraught at the seeming impossibility of progress. “What can I do? There are all these people, and I’m not sure anything I can do will have any effect.” I think this distress is incredibly common and reasonable, given the size of the task and the amount of time any person has in the world.

The task of effecting change is huge on its own, but the project is compounded by its scale: there are a lot of people in the world and a lot of different views. It’s difficult to even know where to begin.

I think fundamentally this kind of distress is about the isolation created by the experience of difference, by the size of the task.

There are tools that we can use for managing and fighting our own isolation: building connections to each other, creating meaning in our lives and in our social spheres.

This is also, interestingly, these are the same methods that we use to organize, to build consciousness, and to change ideologies.

On Wednesday, I said, that (for the most part) people are just people: the way that thought changes is through meaningfulrelationships, conversation, and through additional opportunities to make meaning and to form connections in a larger context.

Seek out people and experiences that are different. Stay safe. Listen. Learn. Talk. Teach. Share your experiences with people who are like you. Work hard. Take breaks. Remember that people are, for the most part, just people, and we’re all alone in this together. All of us.

Shimgo Hugo

Sun, 06 Nov 2016 00:00:00 +0000

In an effort to relaunch tychoish with a more contemporary theme and a publishing tool that (hopefully) will support a more regular posting schedule, I also wrote a nifty go library for dealing with reStructuredText, which may be useful and I think illustrates something about build systems.

In my (apparently still) usual style, there’s some narrative lead in that that takes a bit to get through.

Over the past couple of weeks, I redesigned and redeployed my blog. The system it replaced was somewhat cobbled together, was missing a number of features (e.g. archives, rss feeds, social features, etc) and to add insult to injury it was pretty publishing was pretty slow, and it was difficult to manage a pipeline of posts.

In short, I didn’t post much, though I’ve written things from time to time that I haven’t done a great job of actually posting them, and it was hard to actually get people to read them, which was further demotivating. I’ve been reading a lot of interesting things, and I’m not writing that much for work any more, and I’ve been doing enough things recently that I want to write about them. See this twitter strand I had a bit ago on the topic.

So I started playing around again. Powering this blog is hard, because I have a lot of content¹ and I very much want to use restructuredText. ² There’s this thing called hugo which seems to be pretty popular. I’ve been using static site generators for years, and prefer the approach. It’s also helpful that I worked with Steve (hugo’s original author) during its initial development, and either by coincidence, or as a result our conversations and a couple of very small early contributions a number of things I cared about were included in its design:

support for multiple text markup features (including reStructuredText,) (I cobbled together rst support. )
customizeable page metadata formats. (I think I pushed for support of alternate front-matter formats, specifically YAML, and might have made a few prototype commits on this project)
the ability to schedule posts in the future, (I think we talked about this.)

I think I also winged a bunch in those days about performance. I’ve written about this here before, but one of the classic problems with static site generators is that no one expects sites with one or two thousand posts/content atoms, and so they’re developed against relatively small corpus' and then have performance that doesn’t really scale.

Hugo is fast, but mostly because go is fast, which I think is, in most cases, good enough, but not in my case, and particularly not with the rst implementation as it stood. After all this preamble, we’ve gotten to the interesting part: a tool I’m calling shimgo.

The initial support for rst in hugo is straight forward. Every time hugo encounters an rst file, it calls the shell rst2html utility that is installed when you install docutils, passing it the content of the file on standard input, and parsing from the output, the content we need. It’s not pretty, it'’s not smart, but it works.

Slowly: to publish all of tychoish it took about 3 minutes.

I attempted an rst-to-markdown translation of my exiting content and then ran that through the markdown parsers in hugo, just to get comparative timings: 3ish seconds.

reStructuredText is a bit slower to parse than markdown, on account of it’s comparative strictness and the fact that the toolchain is in python and not go, but this difference seemed absurd.

There’s a go-rst project to write a pure-go implementation of reStructuredText, but I’ve kept my eye on that project for a couple of years, and it’s a lot of work that is pretty far off. While I do want to do more to support this project, I wanted to get a new blog up and running in a few weeks, not years.

Based on the differences in timing, and some intuition from years of writing build systems, I made a wager with myself: while the python rst implementation is likely really slow, it’s not that slow, and I was loosing a lot of time to process creation, teardown, and context switching: processing a single file is pretty quick, but the overhead gets to be too much at scale.

I built a little prototype where I ran a very small HTTP service that took rst as a POST request and returned processed HTML. Now there was one process running, and instead of calling fork/exec a bunch, we just had a little but of (local) network overhead.

Faster: 20 second.

I decided I could deal with it.

What remains is making it production worthy or hugo. While it was good enough for me, I very much don’t want to get into the position of needing to maintain a single-feature fork of a software project in active development, and frankly the existing rst support has a difficult to express external dependency. Adding a HTTP service would be a hard sell.

This brings us to shimgo: the idea is to package everything needed to implement the above solution in an external go package, and package it behind a functional interface, so that hugo maintainers don’t need to know anything about its working.

Isn’t abstraction wonderful?

So here we are. I’m still working on getting this patch mainlined, and there is some polish for shimgo itself (mostly the README file and some documentation), but it works, and if you’re doing anything with reStructuredText in go, then you ought to give shimgo a try.

While I think it would be reasonable to start afresh, I think the whole point of having archives is that you mostly just leave them around. ↩︎
It’s not the most popular markup language, but I’ve used it more than any other text markup, and I find the fact that other langauges (e.g. markdown) vary a lot between implementations to be distressing. Admitedly the fact that there aren’t other implementations of rst is also distressing, but one the balance is somewhat less distressing. ↩︎

Going Forward

Sun, 24 Jul 2016 00:00:00 +0000

I wrote a post about moving on from being a technical writer, and I’ve definitely written some since then about programming and various side projects, but I haven’t really done the kind of public reflection on this topic that I’ve done historically about, many other things.

When I switched to a programming team, I knew some things about computers, and I was a decent Python programmer. The goal, then was to teach myself a second programming language (Go,) and learn how to make “real” software with other people, or on teams with other people. Both of those projects are going well: I think I’ve become pretty solid as a Go programmer, although, it’s hard to say what “real” software is, or if I’m good at making it, but all indications are positive.

This weekend, for various reasons, I’ve been reviving a project that I did some work on this fall and winter, that I’ve abandoned for about 6 months. It’s been both troubling (there are parts that are truly terrible,) and kind of rewarding to see how much I’ve grown as a programmer just from looking at the code.

Queue then, I guess, the self reflective interlude.

My reason for wanting to learn--really learn--a second programming language, was to make sure that all the things I knew about system design, algorithms, and data structures was generalizable, and not rooted in the semantics of a specific language or even implementation of that language. I was also interested in learning more about the process of learning new programming languages so that I had some experience with the learning process, which may come in handy in the future.

Learning Go, I think helped me achieve or realize these goals. While I haven’t really set out to learn a third language yet, it feels tractable. I’ve also noticed some changes and differences in some other aspects of my interests.

I used to be really interested in programming qua programming, and I thought a lot about programming languages. While I still can evaluate programming languages, and have my own share of opinions about “the way things work,” I’m less concerned with the specific syntax or implementation. I think a lot about build tools, platform support, deployment models, and distributing methods and stories, rather than what it can do or how you have to write it. Or, how you make it ship it and run it.

I’ve also gotten less interested in UNIX-esque systems administration and operations, which is historically a thing I’ve been quite interested in. These days, I find myself thinking more about the following kinds of problems:

build systems, the tools building software from source files, (and sometimes testing it!) and the ways to do this super efficiently and sensibly. Build systems are quite hard because in a lot of ways they’re the point through which your software (as software) interacts with all of the platforms it runs on. Efficient build systems have a huge impact on developer productivity, which is a big interest.
developer productivity, this is a big catch all category, but it’s almost always true that people are more expensive than computers, so working on tools and features (like better build systems, or automating various aspects of the development process,)
continuous integration and deployment, again connected to developer productivity, but taking the “automate building and testing,” story to its logical conclusion. CD environments mean you deploy changes much more often, but you also require and force yourself to trust the automated systems and make sure that project leadership and management is just as automated as the development experience.
internal infrastructure, as in “internal services and tools that all applications need,” like logging, queuing systems, abstractions for persistence, deployment systems, testing, and exposed interfaces (e.g. RPC systems, REST/HTTP, or command line option option parsing). Having good tools for these generic aspects of the application make writing actual features for users easier. I’m also increasingly convinced that the way to improve applications and systems is to improve these lower level components and their interfaces.

Free Software and open source are still important, as is UNIX, but these kinds of developer productivity and automation issues are a level above that. I’ve changed in the last 5 years, software has changed in the last five years, the way we run software on systems has changed in the last 5 years. I’m super excited to see what kinds of things I can do in this space, and where I end up in 5 years.

I’m also interested in thinking about ways to write about this. I’d written drafts of a number of posts that were about learning how to program, about systems administration, and now that I’m finding and making more time for writing, one of the things I don’t really know about is what kind of writing on these topics I’m interested in doing, or how to do it in a way that anyone would be interested in reading.

We shall see. Regardless, I hope that I’m back, now.

Works In Progress

Sun, 08 Nov 2015 00:00:00 +0000

I’ve posed about some of these projects before, and I used to regularly post little overviews of the things that I’m working on. But I’ve not done a lot of this recently. Time to do some back fill.

I think it’s probably good to take a step back from time to time and inventory and evaluate the priorities of various projects. Not to mention the fact that I usually say “I’m not really doing much these days,” when this isn’t really true. Here goes:

Mango

This is a project that is private, at the moment, because its mostly useful as an experimental piece of testing infrastructure for work. The idea is to use the same underlying infrastructure to start, stop, and configure processes, but provide REST and command line interfaces for all of these operations.

We have a lot of distinct software that does this internally and it’s always fragile and limited. While grand discussions of code reuse are sort of silly, in this case, it’s a bit annoying that we’ve reinvented this wheel a dozen times… And have to make different changes to a dozen tools as configurations change.

This was also my first project written in Go, which means its been a great learning experience and the place where a number of other Go packages that have become useful in their own right.

Future work:

Write all the tests.
Make the REST interface feature compatible with one of the legacy tools it aims to supplant.
Make a new REST interface that’s more sensible and might be easier to use in more circumstances.
Figure out better ways to block for the appearance of synchronous operations, despite the fact that internally the operations are non-blocking.

Gimlet

Gimlet Blog Post

Gimlet Github

This is really just some convenience work to make it easy to build REST interfaces in Go, without needing to suffer through tools that are designed to support complete “full-stack” web applications. It’s built on the same Negroni/Gorilla Mux stack that I think everyone uses, and it’s very net/http compliant, but with an API that makes it easy (even fun,) to provide high quality JSON+HTTP interfaces.

It struck me, when working on part of Mango, that this chunk of the code didn’t have anything to do with the actual core application and was all about getting a REST-like application to happen. So I split that out, for everyone’s pleasure/suffering.

Future work:

Documentation.
More tests.
Exposed API stabilization and versioning.
Develop story for authentication, sessions and SSL termination.

Grip

Grip Blog Post

Grip Github

Grip is a logging package for Go that attempts to resolve my constant feelings of “I miss x feature of the Python logging package.” It’s not feature comparable with Python logging (but that’s ok,) and since I was working on writing a logging package, I got to add some nifty features.

Future Work:

More documentation.
Better examples, and potentially support for “print this message x% of the time.”
Support for logging to conventional syslog sources in addition to systemd’s logging.

Archer/Dropkick

I’ve wanted to work on a tool to unify a number of personal operations and scripts in a single system and tool. The problem that I’m trying to solve is that I have a number of different computers that I use with some frequency, including laptops, desktops, and a number of servers, and test systems, and I want to be able to describe their configuration, and synchronize files and git data between machines with ease.

My first approach was getting a bunch of random system setup scripts out of a makefile and into a configuration file that a Go program knew how to read and process, and then to expand from there.

I haven’t gotten to the git repository management stuff, because I was working on the Gitgone project.

Future Work:

add better support for creating and managing containers and images using systemd-nspawn and docker.
support for setting up git repositories
support for syncing automatically (i.e. dropox-like functional it -> dropkick).
report status of repositories via a REST API
triggering syncs on remote systems.

Gitgone

Gitgone Github

The idea here is to provide a consistent and full featured way to access and interact with git repositories from Go without needing to wrap the git command yourself (or worse, learn the ins and outs of the git2go). This is largely modeled off of a similar project I did as part of libgiza that does the same sort of thing for Python repositories.

The cool thing about this project is its build abstractly so that you can use one interface and switch between a libgit2 implementation and one that wraps the git command itself.

Future Work:

complete implementation using libgit2
write more extensive tests.
add support for creating repository tags.
provide access to the log.

Novel Project

I’ve been, sporadically, working on notes for writing a new novel. It’s not going anywhere fast, and I’m not even to the point. where I’m outling plot.

I’m trying to tell a story about urban development and how smaller local communities/groups participate in larger communities/groups. How does urban development in place a, impact nation building more globally, and what does this all look like to people as they get to work in the morning, and have to build neighborhood institutions like gyms and restaurants and grocery stores.

But there’s a lot of work to do, and while thinking about the project is fun, there’s a lot of work, and I feel like I’m not ready to commit to a writing project of this scope and, I’m not sure how publishable this project will be (and furthermore, even if its' publishable, will I be willing to do all of that work.)

Software projects are much harder to justify and prioritize than writing projects.

Get a Grip

Sun, 01 Nov 2015 00:00:00 +0000

I made another Go(lang) thing. Grip is a set of logging tools modeled on Go’s standard logging system, with some additional (related) features, including:

level-based logging, with the ability to set a minimum threshold to exclude log messages based on priority (i.e. debugging.)
Error capture/logging, to log Go error objects.
Error aggregation, in continue-on-error situations, where you want to perform a bunch of operations and then return any errors if any of them returned an error but don’t want to return an error after the first operation fails.
Logging to the systemd journal with fallback to standard library logging to standard output.

There are helper functions for logging using different kinds of default string formatting, as well as functions that take error objects, and a “lazy” logging method that take a simple interface for building log messages at log-time rather than at operation time.

None of these features are terribly exciting, and the systemd library wraps the systemd library from CoreOS. I’m a big fan of log levels and priority filtering, so it’s nice to have a tool for that.

In the future, I’d like to add more generic syslog support if that’s useful, and potentially tools for better categorical logging. There’s also a good deal of repeated code and it might be nice to us this as an excuse to write a code-generator using go tool.

Pull requests and feedback are, of course, welcome.

Said on the Train

Fri, 04 Sep 2015 00:00:00 +0000

I finished, on the train this week, reading Freud and the Non-European by Edward Said (on the recommendation of zmagg and it was, one of the better reading experiences I’ve had in a while.

Said is brilliant, and clear and says really complex important hard things in a really clear and approachable style. He’s also frustratingly correct, which isn’t really a problem, but as an engaged and independent reader, I occasionally realize that the internal monologue of my response is an unintelligent “yep yep” chorus, and I feel like I’ve fallen down on the job of being a good reader.

I might have a bit of a complex.

The thing is, that he actually is very right, and does an amazing job of meeting Freud in his historical context, respecting in that context for the audacity of his mission and the power of his insights to encourage us to think about culture, its impact on human motivation, and how personal and cultural histories combine to produce identity, and inspire behavior. Or, more simply, that self-hood and experience are a product of history and context.

Without, of course, in anyway excusing the flaws in Freud’s methods, biases, basis in fact (or lack there of), or utility (or lack there of) in the care of the mentally ill.

Moreso, Said uses Frued, and his ideas about Jewish identity, and himself as an example of late a certain phenotype of 19th century Jewishness, to help contextualize (roughly) contemporary thinking about jewish identity and Israeli culture and statehood.

It’s roughly brilliant.

I’ve long struggled with any kind of theory that engages seriously with Freud or his intellectual successors: there’s so much crap around Freud, and it sort of feels like good energy after bad to try and justify or resuscitate the tradition. And hurts when Freudian are used to support what are otherwise really interesting intellectual projects.

If nothing else Said gives a good example of a successful intellectual interaction with Freud can occur, and what kinds of parameters and context promote that kind of successful and productive interaction.

Maybe someday, I’ll learn how to be a quarter the reader that Said was. If I’m lucky.

In the mean time, I’m just going to keep reading things on the train.

Have a Gimlet: A Go JSON/HTTP API Toolkit

Sun, 26 Jul 2015 00:00:00 +0000

Look folks, I made a thing!

It’s called gimlet, and it’s a Go(lang) tool for making JSON/HTTP APIs (i.e. REST with JSON). Give it a whirl!

It’s actually even less a tool, and more of a toolkit or just “a place to put all of the annoying infrastructure that you’ll inevitably need when you want to build an JSON/HTTP interface, but that have nothing to do what whatever your API/application does: routing, serializing and serializing JSON.

Nothing hard, nothing terribly interesting, and certainly not anything you couldn’t do another way, but, it’s almost certainly true that this layer of application infrastructure is totally orthogonal to whatever you application is actually doing, so you should focus on that, and probaly use something like Gimliet.

Background

I’m using the term HTTP/JSON APIs for services where you send and recive JSON data over HTTP. Sometimes people call these REST APIs, and that’s not inaccurate, but I think REST is a bit more complicated, and not exactly the core paradigm that I’m pursuing with Gimlet.

Sending and reviving JSON over HTTP makes a lot of sense: there are great tools for parsing JSON and HTTP is a decent high level protocol for interprocess communication between simple data applications. Look up “microservices” at your leisure.

Go is a great language for this it has a lot of tooling that anticipates these kinds of applications, and the deployment model is really friendly to operations teams and systems. Also the static

typing and reasonable separation of private and public interfaces is particularly lovely.

So it should be no surprise that there are a lot tools for building stweb applications, frameworks even. They’re great, things like gorilla and negroni are great and provide a very useful set of tools for building Go web apps. Indeed even Gimlet uses components of each of these tools.

The issue, and reason for Gimlet, is that all of these tools assume that you’re building a web application, with web pages, static resources, form handling, session state handling, and other things that are totally irrelevant to writing JSON/HTTP interfaces.

So then, Gimlet is a tool to build these kinds of APIs: simple, uses Negroni and Gorilla’s mux, and does pretty much everything you need except actually write your code.

Example

Set up the app with some basic configuration: :

import "github.com/tychoish/gimlet"

app := gimlet.NewApp()
app.SetPort(9001)
app.SetDefaultVersion(1)

This sets which port the HTTP server is going to listen for requests and configures the default version of the API. You do want all of your endpoints prefixed with “/v<number>” right? The default version of the API is also avalible without the prefix, or if the version of the route is 0. If you don’t set it to 0.

Then register some routes: :

app.AddRoute("/<path>").Version(<int>).Get().Handler(http.HandlerFunc)
app.AddRoute("/<path>").Version(<int>).Post().Handler(http.HandlerFunc)

app.AddRoute returns an API route object with a set of chainable methods for defining the routes. If you add multiple HTTP methods (GET POST and the like,) then Gimlet automatically defines multiple routes with the same handler for each method.

For handlers, I typically just write functions that take arguments from the top level context (database connections, application configuration, etc) and returnhttp.HandlerFunc objects. For example: :

func helloWorld(config *Configuration) http.HandlerFunc {
     return func(w http.ResponseWriter, r *http.Request) {
          input := make(map[string]interface{})
          response := make(map[string]interface{})            

          err := gimlet.GetJSON(input)

          // do stuff here

          gimlet.WriteJSON(w, response)
     }
}

Gimlet has the following functions that parse JSON out of the body of a request, or add JSON output to the body of a response, they are:

WriteJSONResponse(w http.ResponseWrite, code int, data interface{})
GetJSON(r *http.Request, data interface)

Which read or write data into the interface{} object (typically a struct.) The following three provide consistent response writers for common exit codes:

WriteJSON(w http.ResponseWriter, data interface{}) // 200
WriteErrorJSON(w http.ResponseWriter, data interface{}) // 400
WriteInternalErrorJSON(w http.ResponseWriter, data interface{}) // 500

Finally, when you’ve written your app, kick it all off, with the following: :

err := app.Run()
if err != nil {
   fmt.Println(err)
   os.Exit(1)
}

And that’s it. Enjoy, tell me in the comments or on the issues feed if you find something broken or confusing. Contribution welcome, of course.

A New Era

Thu, 26 Feb 2015 00:00:00 +0000

I’m writing the draft of this from an airplane bound for Ireland for a week of singing. I travel often: taking weekend jaunts to go to folk festivals, singing conventions, Morris dancing tours, and so forth, I don’t really vacation often. I find travel and managing the logistics of being in unfamiliar places stressful, and my idea of a good time has a lot to do with sipping a cup of coffee¹ writing something, and reading a book.

Even though I love spending time at home, it’s still important to (sometimes) leave, try new things, and exist somewhere differently for a little while to reset and reflect a bit. With luck, this vacation thing will become something I feel comfortable doing, at least occasionally.

While it certainly wasn’t part of the initial plan, it turns out that this trip is pretty well timed both as denotes a relatively significant change in my life, and I think is a fitting celebration of a period of large changes in my life.

I am no longer a technical writer.

When I return to work, I’m joining the core engineering team to work on build infrastructure and systems projects. I’ll be working on automating our release process, maintaining continuous integration systems, along with an eclectic set of other projects (some of which may involve some of technical writing. There are somethings that you can never escape.)

The truth though, is that this change has been a long time coming: it feels pretty natural. I’ve been working on the documentation build system for a while, and that’s increasingly been the most fun part of my job, so it’ll be good to spend more time doing that kind of work and learn from folks who know more about this kind of thing. Also, build infrastructure and packaging is incredibly important to how people use software and how engineers work, which have been consistent interests of mine for years.

Also, through the last release process, I’ve also found that I’m burning out on writing documentation. I can do it, and I’m not bad at it. After writing, editing and shepherding, more than a million words of documentation (over several thousand pages,) I sometimes feel like I’ve seen it all. I’m interested in seeing the new ideas and perspectives that will prosper in my absence. I’m also eager to see how the foundation I’ve built stands up without me around. It was time.

The decision to change jobs happened rather suddenly. While I’ve built a narrative (see above), in reality, something clicked and I realized it was time. Ten days later there was a plan. I said to myself, “Shit I thought I was done with major life changes for a while.”

In the last year, I’ve bought an apartment and moved to Brooklyn and reorganized my local family grouping:² The good news is that I find my self in a good state, and ‘there’s nothing left to change.³

It’s been a rocky year, nothing is going to change that. Even if in retrospect I find myself satisfied with my actions and decisions, and even if I come out the other end better for my struggle, this is a year I wouldn’t care to repeat.

And even so, I’m excited about the future, about continuing to do interesting work professionally, about enjoying my city and local geography, to surround myself with top notch humans, and to make cool things.

These are early days, and there’s work to do.

In the mean time, I’ll be over here, enjoying something different for a little while.

Astute readers of the blog will note that I am historically a tea drinker. I changed to coffee in late June 2014: I discovered that I didn’t mind the taste as much as I thought, and I like a slightly more potent caffeine delivery system. ↩︎
Break ups suck, each in its own special fucked up way. ↩︎
I’ve started singing more tenor, I guess, so maybe there’s more to change after all. ↩︎

Style Chameleon

Wed, 25 Feb 2015 00:00:00 +0000

When I started my current job there were three major problems with the documentation:

There was too much duplicated content, so it was difficult to know where to point people.
Given that there were always multiple versions of the product in use, it was hard to figure out which paragraph refereed to which version, particularly as the product changed.
Each page felt like it was written by someone else (it was!) and the reading experience could be quite jarring.

The first two were huge tasks, but the solutions were pretty straight forward: build system for documentation that was structured such that it could theoretically hold all the information (replace lots of information repositories with a single information repository) and then use maintenance branches in a version control system to snapshot and fork off old branches as they’re released.

Done.

The last problem is harder. Much harder.

I did a pretty good job, at first, of just writing everything myself, which meant that the first drafts all sounded like they were written by one person: because they were. This doesn’t scale. The next step was to edit the hell out of all contributions that weren’t by me.

This also doesn’t scale.

There are some canonical solutions: write a long style guide and try to get people to comply with it; use templates and standard formats for documents so that everything uses common structure and forms. It’s still hard to enforce, but it’s something.

I put a lot of time into content reuse systems that had additional structure. It helps, and reduces some editorial overhead, but has the same weak points as the conventional solution.

At some point, you need actual humans to edit and make sure things are clear and consistent across the entire corpus. If you don’t have the resources for good editors, then either writers have to spend a significant amount of time editing, which is a huge time suck (and slows progress,) or you start to get drift.

I’m not sure that we have a good answer yet. In the mean time

Appendices

Style in Group Processes

As you innovate and improve it’s really hard to resist the impulse to go back to older pieces to revise them. You should make passes through all your content on some sort of schedule, but you have to give yourself permission and allowance to go back and fix style later.
Common style is less about “being right,” and more about figuring out the kind of communication that’s appropriate for the target audience(s) and using effective structure to support them. In short: compromise.
Style is about the entire reading expenses, not just the syntax, or the typesetting: it is both of these things as well as others. For some kinds of texts, the trick is to often to write as few words as possible, and to make it so that people can scan through documents as quickly as possible while only reading the sections that are relevant to them.

Personal Observations

It’s actually interesting to write blogs again, as I’m (re?)discovering a style of writing that I have been very comfortable with in the past but haven’t really exercised recently. It’s also interesting to see how my own writing and writing process has changed as a result of writing so much technical material.

Knitting and Programming

Tue, 24 Feb 2015 00:00:00 +0000

For 2013 and some of 2012 and 2014, I mostly stopped knitting and spent the time teaching myself how to really make software. It wasn’t exactly that intentional in the moment, but time is finite and both knitting and coding require similar kinds of attention and thought. In retrospect this is what happened.

I’ve started knitting again, not as much or in the same ways that I used to, but there’s a bit more space in my brain for additional projects. I’ve thought often about writing about similarities between knitting and programming.

The two activities have much in common. Code and knitting are both highly structured activities, with a reasonably finite syntax of possible component operations. Most programming languages have a few dozen keywords, fundamental types, and built in operations that give rise to all programs (more or less,) and knitting similarly provides a small number of operations (knit, purl, increase, decrease, slip) that you can combine and alter to produce pretty much any kind of fabric.

Knitting patterns are procedural (like programs, generally) and have a standard syntax that shares a number of common elements with contemporary programming languages, including loops, conditionals, map (functional) operations, as well as some non-trivial concurrency.

While each knitting stitch depends on the stitches that happen before and after it, they are all reasonably atomic within the knitted object. At the same time, many knitting projects have lots other objects that happen at the same time, along different dimensions: shaping, stitch patterns, short rows, sleeves/gloves as well as parallel objects like socks. Knitters routinely track multiple things happening at the same time, with some success: a task that many programmers struggle with.

The connection between knitting and programming isn’t terribly surprising or novel: indeed there’s a long history between the development of textile production and technology more broadly, but hand knitting is sort of a technological oddity, so the connections don’t have the same history that you’d have in weaving (say.)

I’ve struggled to write this post for a while and have always had a hard time figuring out what the conclusion is or should be. I think I’ve felt like there should be some potential for synthesis, some way to take the things I’ve learned about software and use that to write programs to support knitting or design work, or vice versa. Increasingly, I think that this is the anti-conclusion.

Sure you can represent knitting patterns using formal languages (such as programming languages,) and it wouldn’t be that hard to write software that converted a pattern, or validated the math, but the math is really the easy part of knitting.

The hard part about knitting things is not figuring out how many stitches you need to knit an object that’s the right size, but rather figuring out what the right size is. It’s easy to know how many stitches you need, and hard to decide how big or small a thing should be. It’s also true that actually knitting things takes the largest amount of time; figuring out the math and making the design decisions are comparatively quick for most projects.

Computers wouldn’t be able to tell how long we should knit the cuffs of our socks or the sleeves of our sweaters, no matter how much code we write.

In any case, I’ll be over here, knitting away at my sweater.

Brunch Plans

Sat, 03 Jan 2015 00:00:00 +0000

There’s been a lot of change in my life in the last six months or a year, and I’ve mostly avoided reflecting about it in writing. I’ve tried writing about it and tried fictionalizing it: I’ve concluded that I’m just not comfortable with my own interpretation of either the last six months or my current state in a way that I could write from. Nevertheless, given the recent calendar roll over, there’s a lot of reflection in the air.

The longer I don’t write, the larger the burden of not writing becomes. Surprisingly, I’ve not been alarmed by the feeling that I am not trustworthy observer of my own life: I’m confident that I am rapidly reaching a place where the present and the future will be easy to write from, particularly on the smaller scale. There’s just this hole in the larger narrative of my life, that I feel obligated to feel.

But maybe it’s alright if I don’t. Or if I don’t just yet.

It also feels fine to start writing about little things that have changed, things that I’m enjoying in my life that are new and different. This will be the first post of this kind, but hopefully not the last.

Today I had four people at my house for brunch.

I made collard greens cooked with a whole onion, an entire head of garlic, and one beef sausage for flavor. I’d never made collards before (it was great). For next time, I definitely need a larger pan, and need more onion and garlic, perhaps adding mushrooms instead of the sausage, which was nice but not noteworthy. Will make again.

I also made oatmeal, of the steal cut variety cooked using the overnight method where you boil the oats in milk and let it simmer for two minutes, and then let it cool and sit over night, and then boil again in the morning and simmer for 10-12 minutes. My modification was to put a couple of tea spoons of olive oil in with the oats at night. I served with blackberry jam and greek yogurt.

And, of course, several pots of coffee. (Chemex, for the win.)

Simple, not a lot of work. Excessively delicious. Nice to have people in my living room, chatting, eating, and enjoying being with people, and a good way to spend time on a weekend.

I used to think of weekend mornings as an opportunity to write or work on my own projects that required a lot of attention and focus. I’m not sure that I really like this way of ritualizing creativity. I also now live in the same neighborhood as a large number of my friends with half a dozen brunch places (or more!) within a mile. Brunch became a comfortable baseline ritual, and something that I’m much more interested in carving out space for.

While I certainly enjoy going somewhere for brunch, there’s something great about having people over and feeling awesome and comfortable about having friends over.

I don’t think I’m a different person now than I was before. I don’t think its fair or correct to narrate the changes in my life as a progress narrative. Lives change, settings change, people change: things are different, and perhaps that’s enough.

Deep Dark Piles

Tue, 30 Dec 2014 00:00:00 +0000

For the last 9 years or so, I’ve kept a copy of a novel that I wrote during my junior year of high school. It’s so crappy: I think I have a Word 95 file of it somewhere, but it mainly exists in this binder, which turns out to be just the right height to use as a foot rest, to keep my feet planted on the floor while I sit up at my desk.

It’s so bad folks. So bad. A few years back I reread the prologue in order to rip off a few things for a different, only slightly less terrible story. The plotting isn’t particularly inventive, the characters are juvenile, and the settings are bland.

I realized the other day that it’s like all of those shell scripts that I wrote when I was learning to program that are hundreds of lines long, that intermingle argument parsing with actual logic, with horrible unabstracted data structures and access patterns, and implementations that are neither clear nor efficient. Those same scripts that I run a few dozen times a day because they just work, and if I don’t think about it too hard, it doesn’t matter much.

There’s also the two hundred thousand words of total crap that I wrote for my last job. And another couple of un-salvageable fiction projects that I didn’t even bother to print. Even with my current job, I sometimes come across awful paragraphs that I know that I must have written a few years ago, but completely fail to understand what I was thinking.

We all, probably have our own examples of the detritus of our past efforts. I spent a long time feeling like these concrete examples of my ineptitude were a burden, a body of work that I had to revise before I could start working on new projects.

Untrue.

I learned a lot from writing these crappy pieces of code, the failed stories, and terrible pieces of work product. Furthermore all of the work I’ve done in these areas since producing the terrible dark piles, has been better, and continues to get better.

Embedded in the idea that these dark piles are a burden is the notion that the production of text, of code is difficult, challenging, and the majority of the work in creation. But this isn’t true. The work is in figuring out the interfaces, the implementation, figuring out the structure and flow of ideas through a system of functions or paragraphs. Typing it all out is the easy and fun part. It is, of course, important to revise and polish work, but sometimes it’s enough to learn from your mistakes, to remind ourselves of our progress, and move on.

Every time I sit down to write, I put my feet on this binder and remember how far I’ve come.

More than anything, I am thankful.

As You Mean to Go On

Mon, 24 Nov 2014 00:00:00 +0000

I’ve not posted for a long time. A lot of things have changed since I last wrote, and this post is probably not the best place to recount all of them. Indeed many things haven’t changed, but the highlights..

I’m still living in New York, but I bought a coop in Brooklyn and moved, which has been great. I’m surprised at how quickly I have felt at home and rooted. The sequence of changes in my life that brought me there are simple, really, but I’ve struggled to make sense of things even so. The fact that I am aware of development is both a great comfort, but it has been hard to write about my life with confidence.

I replaced my tea habit with a coffee habit. I attend yoga classes regularly. I sing Sacred Harp (and sometimes other shaped notes). I Morris dance with the Bowery Boys, the Men’s team in New York City, and continue to dance with Braintrust Morris, an kind of butch Morris team with a spiritual center in the Midwest.

I’m still working on the same kinds of developer documentation, build systems, and software development projects for the same company as before. I still think open source software is important. I am really interested in helping people develop technological literacy and understanding. I want to work on improving infrastructure for developers and development. I use Linux extensively, write a lot of Python, tinker with Go and Common Lisp, and live and breathe in Emacs.

Life is good. Life is difficult. Life is.

I look forward to writing about it here.

So the blog looks different. I changed some things:

I switched from using ikiwiki to using Sphinx and an extension called ablog. I wanted to use a system that I was familiar with and could hack on (that’s Sphinx, which is the core of the tool chain I use at work.) Also, I wanted to use reStructuredText, which I prefer.

Ikiwiki is great software, and I quite enjoy it’s architecture and use. The problem is that it’s not in particularly active development, the code base is in Perl (which I don’t know, and except for ikiwiki, have no reason to learn,) and the project has probably peaked in terms of its adoption curve. To boot, my goals and user story aren’t totally inline it’s goals and user stories.

Sphinx isn’t totally right for a blogging engine either, but it’s solid. I actively develop and maintain tools around Sphinx. Indeed, I was able to make some small changes in ablog that shaved about a third of the time off of the total build time. Not bad.

It took a lot of time to convert all the posts to the new format, but now that everything is in order and the tools are usable it is time to start writing again.

It’s good to be home.

Writing More Again

Sun, 18 May 2014 00:00:00 +0000

I haven’t really written very much in the past six months or more, and while I think I’ve done cool things and learned about cool things,

Let me take that back. I haven’t written anything in a sustained sort of way that wasn’t for work. Hell, even what I’ve been doing for work has been smaller and more tactical. Unfortunately you don’t write books, or book-like-objects as small tactical approaches.

So I’ve been trying to figure out how to get back to that. It’s about rhythm more than anything. If you approach time management for log-form rationally, there’s never enough time, so you have to make time, and trick yourself into writing and figure out how to make a little progress on a regular

As someone who definitely tends towards binge writing (and who doesn’t) remembering to write a little bit on a regular basis is hard and nearly counter intuitive. So as I’ve been attempting to restart the writing habit, I’ve been thinking about what I’ve done in the past that’s worked to keep up the momentum and work on projects. I’ve uncovered:

Reading

Even more important, I think, than reestablishing a habig of writing regularly, is restablishing the practice of reading regularly. I often get hung up on the fact that I don’t think I’m a very fast or very through reader. But reading is quite inspiring, and I often find that the more I read, the more I want to write.

So I’ve been reading the Vorkosigan novels, and it’s been great.

Tracking Word Counts

I like having some record of my progress, and I’ve taken different approaches to tracking progress, mostly using word counts, over time.

When you’re writing longer pieces, particularly in editors that don’t reflect word counts or page numbers in the interface, it’s easy to loose context for how much progress you’re making.

Way back when, I used to record the current page and word count for all of my projects in a note boot, and this developed into a rather incurable tick to mash a few keys down every few sentences to check the word count. Then, three years ago, I wrote a script to check the word count of all my projects on a regular interval.

I stopped really using it about a year ago, because the script handled different branches (in git) of the same project really poorly, and there wasn’t a good way to hack that in.

So I recently rewrote this program, and I rather like it. I’ll post more about it soon.

Physical Activity and Care

To do writing one definitely needs to spend a serious amount of time in front of a computer typing. Unless you really like notebooks and pens, there’s no other way to get things written.

But you can’t take this to the extreme: if all you do is sit in front of your computer and stare at cursor waiting for inspiration to strike. As people we need different kinds of focus, and different kinds of experiences, on the small scale, to keep the generative impulses functional.¹

Reading voraciously, as above, is a key part of being able to write effectively, but other kinds of activity are also useful. When I’m not writing, my first instinct is to try and find more time to write, but sometimes--often--what I need more is more time for things that aren’t writing: exercise, reading, and so forth.

I’ve started to do yoga, as I’ve written before, a few times a week. I need to also remember to set aside time for other non-writing activities: more exercise, walks, dinners with friends, dates with my email backlog, etc.

Onward and Upward!

I like the idea of generative impulses rather than creative muscles. Writing isn’t always creative, but it is, like any other kind of production, work, and always requires energy and effort. ↩︎

Fabric Complexity

Sun, 11 May 2014 00:00:00 +0000

I’ve been a big fan of fabric which is a Python tool for deployment orchestration: deploying a new release of an application, uploading files, deploying new configurations to a group of hosts in a cluster. Before fabric, the options were either to write fragile shell scripts that often didn’t do a good job of handling multiple hosts, or use more heavy weight configuration management tools, which had a lot of overhead and bother.

Fabric is great. Make a python module called “fabfile,” write idiomatic Python, stick anywhere, and then you can call functions in that module using the fab command. And it includes a lot of helpers for interacting with groups of hosts and dealing with remote and local resources that is designed for takes into consideration common contemporary application development.

For a while, I basically thought that fabric, or tools like it would completly obviate the need for shell scripting

I think there are two major problems with using fabric as a framework for general purpose scripting:

Global state can get messy.

This isn’t only an issue with Fabric, but I think the fab solution isn’t great here. Fabric has this envy object that you can import and then use to track the settings or set custom “global” data values. It’s better than having actual global data, maybe, but you end up having to use it too much, particularly it for communication between different related operations and you end up using tasks and global state to run-time configuration.
There’s no real overriding abstraction to guide code organization.

In a lot of ways fab scripts are just a way of writing a large number of scripts within a single code base and a command line interface for specifying different entry points. It would be nice to be able to express relationships between tasks, or centralize the registry of tasks in some of non-adhoc.

I actually don’t think that these are problems with fabric, itself, but I think it’s probably the case that we need a few different kinds of tools and patterns to help support the authorship of useful and maintainable shell scripts and build systems.

a good, usable, non-procedural way manage argument parsing.

In python, I like argprase and think it works really well, so I’m not saying we need some set of tools to actually parse the arguments that people input on the command line. But I think when you’re writing a script, argparse is too low level, and pretty soon there are a few hundred lines of declarative code, and you still have to connect this up with the actual code, which leads to maintenance headaches and all sorts of dumb errors.
a good framework for expressing multi-stage tasks and their dependencies and relationships.

This may basically be waf, but I think we can do something similar. I made an attempt at this with buildcloth but I think I ended up with something that’s too much of a (meta) build automation tool, and not enough of a general purpose paradigm for organizing code to perform related tasks in a clean, modular, and well organized sort of way.

These aren’t fabric specific problems, or Python-specific problems, of course. But they are particularly relevant to Python moderatly sized Python code bases in general and most fabric implementations that I’veb seen. Maybe Python and fabric make writing useful code easy enough that we start to see complexity problems earlier than we would otherwise. Not sure.

As you may have been able to guess at this point, I am somewhat burdened by a large and somewhat overgrown fabric-based project, that needs refactoring. Thankfully, matters are not critical, but I do want to figure out the right way to solve this kind of problem before I start hacking.

Best Practice Practices

Sat, 10 May 2014 00:00:00 +0000

At work people often ask for more “best practices” guides. In some ways this is sign of success: they’re no longer begging for fundamental reference material and descriptions of basic use. Nevertheless I almost always wince:

1. “Best practices” carries an implicit sense of guarantee along the lines of “if you adhere to the best practices, then you won’t run into problems,” which is sort of difficult to assert with confidence, and is really a product design issue, not a documentation issue.

It is really hard to make one-size-fits-all recommendations.

3. Best practice-guides don’t address actual needs of users, or actual solutions to real problems, because they generalize problems and solutions beyond the point of re-usability.

This isn’t to say that it’s difficult or impossible to give recommendations, and indeed documentation is the best way that the purveyors of software have to to shape practices. However, the documentation should guide users towards better practices everywhere and isolating recommendations into certain types of documents is probably counter productive.

Onward and Upward!

Common Lisp Scripts

Sat, 19 Apr 2014 00:00:00 +0000

As part of my project to learn common lisp, or at least write more common lisp as part of my day to day work and life, I’ve

This is a total rip off of this blog post, with a few minor changes:

I hacked some makefile goodness so that it will automatically create binaries for all .lisp files, and means that you can drop a script in the directory and not have to edit the makefile to get the magic to happen.
I switched to using buildapp to rather than cl-launch. Buildapp feels a bit more maintained, and I wanted practice using it. Otherwise, I don’t think it matters.

The makefile:

TARGETS := $(subst .lisp,,$(wildcard *.lisp))

all:$(TARGETS)

%:%.lisp
    @echo [build]: creating $@
    @buildapp --load $< --entry script:run --output $@
    @echo [build]: created $@

clean:
    @rm -f $(TARGETS)

Hello world:

(cl:defpackage #:script
(:use #:cl)
(:export #:run))

(cl:in-package :script)

(defun run (argv)
   (format t "hello world~%"))

As an aside, I also updated the buildapp aur package.

Pros:

Common lisp scripts.
The ability to integrate writing lisp into your existing Linux/Unix workflow and processes.
Not having to think about packaging or build architecture for trivial operations.

Cons:

~40+ meg executable.
Only one source file per script.

Common Lisp Progress

Sun, 06 Apr 2014 00:00:00 +0000

The backstory: I’m trying to learn Common Lisp. It’s sort of an arcane programming language with a few aspects that I rather like, and I’m viewing this as an exercise to generalize my programming experience/knowledge.

I’ve written some common lisp over the years, mostly because I use stumpwm, but I’ve been struggling to find a good project to start on my own or hack on an existing project.

A few weekends ago, I started hacking on coleslaw, which is a static site generator written in common lisp. The reasons are simple:

I know something about the static site generator domain, so I’m only trying to learn one thing.
Coleslaw is typical in many respects of site generators, but the architecture makes sense, and it’s reasonably simple and hasn’t been overly optimized.

There are three features I’m interested in adding:

I want make it so that Coleslaw builds content incrementally: there’s no reason that programs like this should have to rebuild all content on every build. There are two missing concepts “which pages changed,” and “if one page changes, what other pages must change.”
I want to make the entire build process more configurable. Thise includes expanding the options that are available in the site configuration, and allowing users to define and edit themes within their projects.
It feels wrong to leave the program signally threaded, particularly when build systems are inherently parallel, and Common Lisp real threads.

I want to make this blog more blog-like. The current site works fine, but I’m growing restless, and I want to explore more.

So I began hacking and managed to learn a lot about the codebase, and am getting into the swing of this project. I played with a naive concurrency approach but it didn’t stick. For about a day of work, I:

Made the docutils package usable from coleslaw and fixed some issues there. See pull request.
Added a lot of configuration and made more things customizable. Pull requests.

There’s a lot of little things that I like or feel like I will shortly like, but the sticking points at the moment are:
A lot of things have started to make sense. let forms feel very natural. The object system is pretty great. I feel comfortable with the organization of code within a function. I don’t mind the parentheses.
I remain pretty confused by the variable binding system outside of the basic let forms. I’m never sure about setf and setq (my policy thus far has just been to setf and troubleshoot if it doesn’t do what I want.) If Python is all about the power of the namespace, it feels like CL is all about the power of scopes and I haven’t yet learned all of them.
Oddly, conditionals feel really hard and cumbersome. The if form is more limited and not particularly clear, cond is pretty useful but often feels like a sledge hammer. unless, and when are clear but not perfect.

I’ve not found the idiomatic way to perform “set a variable to the return of a function if it’s current value is 0/null/false, otherwise pass” which I find myself reaching for pretty frequently (and awkwardly.)

More to come, I’m sure!

Learning Problems

Sun, 02 Mar 2014 00:00:00 +0000

Learning how to make computer software is hard. Not fundamentally hard: lots of people can do it, and even more people do things that are functionally equivalent to programming though they wouldn’t think of it as such. But teaching people how to write good computer software is a challenge, and one that I’m generally interested in exploring more.

For a long time, I’ve been interested in this problem from the outside: I didn’t really know how to program in any meaningful sort of way and I was interested in deconstructing the process of making software. Then something clicked and years of tinkering with systems administration and reading about programming languages and practices clicked and while I think I have a lot left to learn, I’ve started thinking about the problem from the other side.

We accumulate many skills and kinds of knowledge in an incremental sort of way: you study and practice and little by little our brains (and bodies) form new connections and we “learn.” Other kinds of learning follow a more “step-based” approach: we practice and study for a long period of time without much discernible change in understanding or skill until at some point we experience some sort of larger improvement in ability.

At least for me (and perhaps you as well,) things like dance, knitting, writing, and most structured/classroom-based topics tend to be incremental, mostly. Other things, like programming (at least initially) and singing/music tend to be step-based.

This isn’t to say that step-based areas of focus don’t require regular ongoing practice, just that the observable markers of progress may lag inconsistently behind effort and pedagogy.

When I was doing more non-professional writing, I was fond of the school of writing advice that said “the way to learn how to write (fiction) is to have a good story to tell;” when I think about learning to program I think the first step has to be a need to automate something on a computer.

I’ve even written up something on the topic

Hackers describe this as the “scratch your own itch” method (from CatB and elsewhere.)

Neither the idea of step-based versus incremental learning nor the notion of using a personal need to drive learning are new, but I think they illuminate eachother well.

Since I stopped being a student somewhat abruptly in 2007, I’ve become increasingly glad both that my education and personal development has continued and I’ve had the opportunity to explore things in ways that didn’t make sense in a structured context (e.g. “I want to learn about how databases work without formal CS/systems training,” or “I want to learn how to sing and withouta lot of music theory.")

For most of the past year I’ve been pretty heads-down on the “learning to program” project, and I’ve had a number of interesting problems that I’ve used to help explore the topic:

generalized `Sphinx <http://sphinx-doc.org>`_ publishing toolkit.

This is both a work project and a personal project.

Sphinx is a great tool for producing text, and I’m quite fond of it. At the same time, I’m not a fan of its architecture (and have a number of approaches to optimize the build process,) and there are a number of tasks: dependency resolution, version management, theme management, and deployment that any reasonably complex Sphinx-based project needs to address.

While this project requires ongoing development and improvement, it’s basically feature complete, and it’s given
I’ve had a couple of personal side projects that I’ve used to explore different kinds of programming problems, with greater and lesser success.

Buildcloth, which is pretty cool and needs more work but I fear may be too complicated for the use-case.

csc, which isn’t fully off the ground and may not provide a significant improvement upon Sphinx for most cases.

dtf, which is a decent idea, but I’ve not had time to really implement and exercise the program and I fear that the core code quality isn’t great.
More recently I’ve begun working on a dependency analytics package with a co-worker/friend to help him and his teammates understand and untangle a larger C++ project.

It’s been nice to be able to actively work on a project with another developer, and to be able to focus on performance and architecture issues while someone else focuses on feature prototyping and use-cases.

In a lot of ways this is a good “capstone” project for me because I’ve gotten to use and apply many of the things I’ve learned from writing concurrent/parallel Python, as well as moderate sized Python programs comes together well here.

Most of the projects that had been open and on my plate for the last few months have mostly wrapped up. There’s more work to be done on them, I could do a lot more work, and I think I will, but none of them are lacking a feature that I really need in order to accomplish something that I want to do.

I’m interested in learning more: about writing (documentation, science fiction, etc.), about software development and computing.

Documentation Inheritance

Sun, 23 Feb 2014 00:00:00 +0000

I’m interested in using metaphors and methods from programming and engineering to make documentation better. There are some obvious elements that are ripe for stealing in terms of process (scrum, iteration, etc.) as well as tooling (issue tracking, version control.) As I’ve continued to explore the connections and metaphors have become less obvious, but remain very helpful.

Recently I’ve been thinking about and using the idea of inheritance to help address content duplication issues. The new approach to tutorial content is one of these applications. Actually, this is a bit of retroactive intellectualizing: the need for reuse came first, and relating this back to inheritance is an idea that I want to explore.

I’m thinking about inheritance in the same way that we talk about the inheritance of classes in object oriented programs.

In the past, I’ve talked about the failure of the promise of single sourcing to solve the complexity problems in documentation as being related to the failure of object oriented programming styles to resolve code duplication and promote widespread code reuse. Or, if not related, symptoms of related causes.

For code, I think the issue is that while there are a couple of applications for inheritance (i.e. representing trees, some basic templating,) it’s not a universally applicable metaphor. The mantra composition over inheritance draws attention to the fact that “has a” relationships are more prevalent and useful than “is a” relationships.

Tactically, speaking, using inheritance rather than simple inlining or inclusion is quite helpful for thinking about content reuse in documentation. Inlining is technically easy to implement, but doesn’t actually help facilitate content reuse because it’s hard to write content for inclusion that’s sufficiently “context free,” whereas using inheritance makes it possible to reuse structures and portions of content without requiring writers to write context-free content.

Inheritance isn’t perfect of course: if you have to overload all or most inherited elements you end up with a confusing mush that’s hard to untangle, but it’s a decent starting point.

Onward and upward!

Rambling about a Bug

Sat, 22 Feb 2014 00:00:00 +0000

I spent some time yesterday evening dealing with a bug in some code I wrote/maintain, and I thought it would be a good exercise to just talk about the issue, and approaches problem solving. This is both an effort to demystify the act of programming and debugging and a brainstorming exercise. I explained a bunch of the background in an external page, to make the content a bit more accessible.

The System

To build a large complex documentation site we use Sphinx as the core “compiler” to translate source into outputs like HTML and PDFs, but this is really just the core operation, there’s a lot of other work: generating content, analyzing content to make sure that everything that needs to be recompiled is, and migrating content to staging and production environments.

Essentially the build happens in three-parts:

In the first step, we generate a bunch of content from structured sources. Then we have to assemble a dependency graph of all the content and update the time stamp (i.e. mime) all files that depend on files that have changed so that Sphinx will actually rebuild the files that need it.

Most of this work happens in some sort of parallel execution model. I’ll rant about concurrency in Python in a bit. Hold your questions.
In the second step Sphinx runs. For testing purposes, this just builds one output (i.e. HTML), but our production builds need to build multiple output formats. When there are multiple builds, this runs each build in its own thread. Builds execute as sub-processes. Sphinx’s processing has a two-phase approach: there’s a read phase and a write phase. The read phase happens in one linear execution, but writing files can happen in a processing pool (if sphinx needs to write large numbers of multiple files.)
Finally we do a bunch of post-processing on the builds to move the files into the right places. This happens (typically) in a process pool that runs within the thread where the build executed.

See my overview of concurrency, and rant on Python concurrency for some background.

The Problem

There’s a deadlock. Sometimes, one of the Sphinx will just die, and stop doing anything, produces no messages and waits forever. (This is in step 2 above.) Everything else works fine.

Great.

The problem, or part of the problem, is that all of the parts of the system work fine in isolation. The code that starts the external process gets called extensively and works fine. The thread/process management code is also well exercised. In essence the question is:

“What could make code that works fine most of the time, fail oddly only sometimes.”

As I think about it, I’ve observed intermittent deadlocks from time to time, but they’ve gotten much worse recently:

We weren’t doing automated building for a few weeks, for an unrelated reason and the problem shows up more frequently in automated builds.
We moved the execution of the Sphinx process to thread pools from process pools: they were already running in yielding operations, no need to double the overhead.
A few more operations moved to threads as well, again, mostly operations with lots of yielding, to reduce the overhead of processes and in one situation when a task was incompatible with processes (i.e. not pickelable.)

Attempts

Make sure that the output of the sphinx build isn’t filling the output buffering that subprocess is doing. Write output to a temporary file rather than using the buffer. (see tempfile)
Use Popen.poll() to figure out when an operation is done rather than using a blocking call which might prevent the thread from ever waking up.

Current state.

The bug is difficult to reproduce, but the above issues seems to reduce the incidence of the bug by a significant margin. My current theory is there were two deadlocks: one in sphinx itself that was pretty uncommon, and one with how we were calling sphinx that was more common and exacerbated by switching to the pool that ran sphinx from a process pool to a thread pool.

So the bug has probably been around for a while, but was much more rare given the different execution model. And regardless, was easily confused with another bug. Delightful.

Having resolved the bigger issue, it’s no longer a real problem for our automated testing, so I consider it a (minor) success. Longer term, I think we need to streamline the execution and process management, but this is not crucial now s we can continue to increment toward that solution.

Python Concurrency Rant

Fri, 21 Feb 2014 00:00:00 +0000

What and Why

Concurrency is the term we use to think about operations that can happen at the same time. A computer program is just a list of operations that have some sort of ordering. If you run those operation in the order the programmer wrote them in, and the programmer wrote the right code, everything should work fine: If operation B depends on the outcome of operation A, then you just have to make sure that operation A happens before B.

There are two kinds of problems that this kind of model doesn’t address well:

Programs that respond to user or environmental interactions.

Many programs don’t actually have a linear procedure, and depend on user input (e.g. a word processor or text editor must wait for you to type text).

Let’s call these “event driven” programs.
Programs may consist of operations that don’t depend on each other.

Consider a program that has operations A, B, C, and D. Sometimes, you may have to run D after C and C after B, and B after A; but sometimes B, C, and D are totally (or mostly) independent of each other and can run in any order.

Let’s call these “potentially parallel workloads.”

While both of these kinds of models help us to think about ways that software can reflect and respond to operations happening at the same time, concurrency is more subtle than “parallelism.” Concurrency is about modeling the dependencies and relationships between operations, parallelism is really just an implementation detail.

Parallelism requires concurrency; but you can execute concurrent designs in parallel or not as needed or desired.

There are caveats both during development and at runtime:

Concurrency makes some aspects of programming harder because if the order and timing of operations changes, the possibilities for conflicts and errors grows. It also makes it harder to follow the (possible) chain of operations.
At runtime, if you use parallelism to execute a concurrent program, there’s an amount of overhead spent managing the more complex execution model.

Reference: <http://blog.golang.org/concurrency-is-not-parallelism>

Implementation Details (Python)

A single Python process does not support parallel execution: only one operation can execute at a time. Sort of. The rule is more complex:

Python operations in Python code.

This covers all Python code you write and depend on, and much of the standard library.
(some) Python operations implemented in C.

This covers some operations like computing hashes and reading and writing to files or network connections.
Operations that run external processes. (e.g. “run this shell command”)

The second two operations can run in parallel with each other and with execution of Python code (i.e. these operations are “yielding”). These yielding operations typically account for the operations that take the most amount of time. The downside is that yielding operations account for a small percentage of the number of operations in a Python program.

Python provides two (native) parallelism metaphors: threads and processes.

Threads are lightweight, have low start-up costs, and have access to the shared state of the master process. The downsides is that only yielding operations can actually run in parallel. Otherwise only one Python operation can run at once. Except that operations will interleave at some level, which means you can get some kinds of concurrency bugs (deadlocks/races) even though there’s limited parallel operation.
Processes are less lightweight, have slightly higher start-up costs, but can all execute Python code at the same time. They also don’t have access to shared state, which means there are more costs associated with copying memory to-and-from the process. While there are more limitations on what can run in processes, because there’s isolation and no shared state, its more safe.

The best part is that the interfaces for working with threads and processes are the same, which makes testing easier.

The Rant

The problem isn’t that Python doesn’t have concurrency tools, it’s that no one started writing Python with the idea that parallelism and concurrency would be a defining element of most systems that people would need or want to write.

The result is that while it’s theoretically possible to modify Python itself to be more concurrent, one of the two things happen:

Everything breaks. There’s a lot of Python that depends on the current behavior and concurrency semantics. The Python Standard Library is big. The ecosystem of software written in Python is even bigger and everything would break.
In order to prevent everything from breaking, to make the changes required to support more intrinsic parallelism you actually end up slowing-down the arguably more common non-parallel operation.

The work on this is ongoing, of course, and eventually I suspect there will be some solution, but the change is unlikely to be revolutionary. In the mean time, it’s awkward and sometimes awful:

You can write concurrent code, which is nice, but there is some awkwardness around these expression: calling lots of functions inside of [multiprocessing.Pool.apply_async()]{.title-ref} (or something similar) is pain; callbacks and passing passing function pointers around is awkward and prone to error.
Because so little of the Python tooling expects thing to be running in parallel, there are huge warts: error handling blows; the documentation doesn’t really cover what yields or doesn’t, and what can or will block.
In some situations, you can get pretty good parallel performance. This feels great, but often doesn’t feel predictable or reproducible.

What would make this better?

There should be standard ways to express concurrency that feels less like a hack. This is a syntax/library deficiency.
Errors in processes should bubble up more forcefully.
Documentation of Python APIs should affirmatively describe the concurrency semantics of all operations.

Write Sane Software

Fri, 21 Feb 2014 00:00:00 +0000

a checklist

There’s a difference between software that works, software that’s brilliant, software that’s maintainable, and software that’s good. This is post that begins to enumerate the kinds of things that you can do as you write software to help make it sane and possibly good.

This isn’t about computer science, or really even about engineering principals. We all have a sense of what makes a physical object (furniture, buildings, electronics) feel like they are well made. This is about making software have the same feel, and what you can think about when you’re writing code to help give the finished product that feel.

I’ll expand on these items later, but as an outline.

Have logging everywhere.
Solid test code that mirrors app functionality.

If you feel like you can safely make a change to the code and be confident that your tests will catch regressions, then you’re good, otherwise; write more tests.

Never more hierarchy then you need.
Make the building infrastructure really robust.

5. Have internal abstractions for internal configuration and persistence.

Configuration and data persistence should be encapsulated by some internal interface and the application logic shouldn’t depend on the implementations of how the application is configured or how you persist the data.

Consider concurrency when possible.
Don’t confuse exceptions and conditions.
Break long sequences of sections into groups of logical operations.
Avoid unnecessarily tangled execution paths.

Please discuss!

Structure, Documentation, and Poetry

Mon, 17 Feb 2014 00:00:00 +0000

At work I’ve introduced a new method for storing the content for procedural documents (i.e. tutorials/how-to guides/etc,) that is more structured. Rather than just writing a tutorial in a conventional way, we capture the steps in a specific structure that we use to generate the content in a specific way. The structure lets us do cool things with the presentation that would be hard to do well otherwise and help us focus theses kinds of documents on the core procedural or sequence-based nature of these documents.

Doing documentation well is really hard, but probably not for the reasons you think: Writing clear text about complex ideas isn’t easy, but it is straight forward. Figuring out what needs to be documented is harder, but it boils down to a business problem and if you talk to the right people it’s not a significant challenge. Ensuring that the documentation remains correct over time as the product develops, particularly with regards to duplicated content: really hard. Managing complexity of large texts that are always growing and changing, particularly with regards to correctness and content duplication: really fucking hard.

The problem is that the solutions to the hard problems are at odds with each other: you help address the complexity problems by decoupling the organization of the source material from the presentation, which never works well enough. You solve the duplication problem using a “single sourcing strategy” where you store common bits of information in one place and then inject that text as needed, which increases complexity.

There is no winning.

The new structured approach to procedures may not be winning, exactly, but it works pretty well. Essentially we put a huge core of our content into a YAML document and then render it out, but we get some nice benefits:

For each step in a sequence we can inherit from any other step in the project and override any component of that step. This facilitates reuse, and forces us to think about potential reuse throughout the process.
The compiler and parsers tell us when we get something wrong.
The output is very regular, so we can be confident that all of the tutorials look the same and have the same structure.
Minimal loss of editing clarity: YAML is great to edit, the structure is like a JSON document, but commentsare allowed and the syntax does smart things with newlines and requires less escaping.

Basically, at its core, we’re restricting the available structural possibles for some documents and using (simple) software to enforce those requirements. The results are quite good, but in some ways it makes the writing part a bit more difficult. There’s less room for imprecision, and there are some weak rhetorical formulations that become more important to avoid. For example: complex conditional structures don’t always work well and positional references (i.e. “as above”) are almost always unhelpful.

In some ways the effect is kind of like writing in a specific poetic form. Less metric, but the same kind of toying with squeezing an idea into a very specific form. The coolest side effect is that given the constraints that the new system imposes, the quality of everyone’s output has improved.

I can live with that!

Practicing

Sat, 08 Feb 2014 00:00:00 +0000

In response to my Knitting Practices post, on Facebook my father commented “The word “practice” is apt. Is there an influence from yoga?”

The answer is obviously “yes,” though the route is somewhat indirect and travels through a story about programming. Stick with it for a little while.

Generators and Python Memory Efficiency

I was talking with a dancing friend about memory efficiency in Python programs, particularly with regards to loops and the range() function. Say you want to do something 1000 times, in Python the easiest way to do this is:

for it in range(1000):
   do_thing(it)

This says, “make a list of numbers from 1 to 1000,” and then call the do_thing() operation on that number (i.e. assigned to the variable it). range(1000) evaluates to a list, which Python stores in memory and the code above loops over.

The way to get this effect in other languages (JavaScript below) is to do something like this:

for (var it=0; it<= 1000; it++) {
   do_thing(it)
}

The for loop has three statements: an action to perform running the content of the loop (create a variable), a condition that will terminate the loop by returning false (variable is less than or equal to 1000), and an operation to run after each loop. The effect of the JavaScript is the same as the Python, except that the Python has to build this (potentially large) list for grins.

There’s a couple of quick answers to this specific question:

(In Python 2) use xrange() which is a special iterable type (i.e. it works with loops), that doesn’t need to build a list in memory, it just spits out values incrementally as needed.

Let’s imagine that it does this by having a function that returns values starting at a certain point (i.e. 0) pausing after each value, and then returning the next value the next time it runs. Thes ea re called “generators” in Python.
Wait for the Python 3 switch to complete: In Python 3, range() is a generator of sorts, and it’s efficient in this way.
Not care about memory use so much. In most cases this will not be the bottle neck in your application. Really long lists of integers may take up megabytes of memory, but that’s not a huge deal.

Generators are great, and they’re worth using in most cases, but no one will laugh at you if you don’t use them in this situation.

The coolest thing, really is that you can really easily write generator functions. Here’s a silly example: :

def animals("input_animal"=None):
   for animal in [ 'cat', 'dog', 'cow']:
      if input_animal != animal:
          yield animal

This function returns an animal, as long as the input_animal isn’t in the list.

I explained how generators work to my friend and he said something like “Nifty, I guess they’re not really part of my Python practice yet.”

The phrase sort of stuck with me.

Programming is a Practice

There’s a bunch of theory to programming that’s grounded in how computers work, and a lot of things that are actually useful for programmers to know but in truth programming is a practice. Being able to look at a design or a program and understand how to write code to achieve some goal or connect two pieces of functionality, is really about the practice

Beyond understanding what makes good software good and how the machines work, most of programming is knowing the tools really wall and being able to identify the right tool for the job. Everything else is just typing and flow.

While the fundamental knowledge and knowledge of the tools is always important, at some point programming becomes more about being to figure out what the right solution is for any given obstacle or situation.

Knitting as Practice

Knitting is kind of the same. There are some fundamental skills (stitches, operations) and there are some basic fundamental modalities/patterns¹ that you use and combine to make some common objects.

Knitting itself is always about repetition, stitch after stitch, but it’s also about repetition on a higher level. Knitting the second sock, the second sleeve. Making pair of sock after pair of sock, or sweater after sweater.

At some point being a knitter stops being about the skills and modalities/patterns and starts being about actually making things and figuring out how to apply what you know about knitting to a new situation: the thing you want to make, the yarn your using, and the tools you’re using.

It’s all practice. All the way down.

In programming terms, these would be patterns. I’m using the term modality/pattern to disambiguate. I’m thinking about things like “how to shape a neck opening on a sweater,” or “turn a heel on a sock.” Not the pattern for an entire object, but the method for knitting a particular object. ↩︎

The First Ankle Sock

Mon, 20 Jan 2014 00:00:00 +0000

I recently finished my first knitting project after my [[return to knitting|/knitting/a-knitting-practice/]]: a pair of ankle socks in a medium (for socks) weight yarn. The socks are plain and I’m pleased with the result, but there were some unexpected parts of the project, and some useful lessons:

These Aren’t The Socks You Meant To Make

For the most part I’m interested in knitting pretty plain socks in plain (solid) yarn: probably even undated yarn that I can soak in tea or henna post-knitting. So the fact that this is one of those patterned, superwash sock yarn is something of a misnomer.

I also intended to get 7 inch cuffs out of the yarn (in a 50 gram ball and I even knit most of a sock: but I ran out of yarn. My assumption that I could knit a 64 stitch sock with sport weight yarn was a touch over the mark, but it was close.

So having failed that project I took a break from the yarn and ripped it all back, and attempted to do a 60 stitch sock (right choice!) and knit a 2 inch cuff.

I could have gotten away with a slightly longer cuff, but I’ve been wanting to have a couple of short pairs of socks to wear while dancing, and to be honest the yarn looked better on the ball than knitted up.

The first moral: Given enough time, I figured it out.

The second moral: It’s ok to rip something out if it doesn’t work out. It’s also ok to not rip it out immediately after you realize it doesn’t work out.

I Kitchnered Toes All By Myself

I’ve never really been successful at kitchnering anything. I find every instruction to be incomprehensible, and every attempt to kitchner something has ended in disaster. My solution for the past couple of years has been to send the occasional sock off to my mother for her to do the honors.

The thing is, I understand the concept of what’s happening, which makes the fact that I couldn’t do it so frustrating. While I was knitting the first sock, I decided that I could probably think my way through the problem, and after a bit of fussing, I was ale to actually complete the sock toe.

I had to turn the sock inside out, and the ends weren’t perfect, but it was close.

The Wearing Report

I wore these socks to a big Balkan Dance festival, and they worked! I thought the cuffs were a bit too short, but the sizing was correct in every other regard. Next pair will have slightly longer cuffs.

Huzzah!

The Perfect Rib

Sat, 18 Jan 2014 00:00:00 +0000

I knit socks on four (5 inch) double pointed needles, these days from the cuff down, and I enjoy socks with some amount of ribbing at the cuff.

Seems reasonable. I also tend to have two other restrictions:

Each needle should have the same number of stitches.
In the ribbing, each needle should begin with a knit stitch and end with a purl stitch.

This somewhat constrains the possibilities. Also given a preference for rubbings that are biased towards knit stitches, I tend to veto options that might otherwise be workable. Here’s a quick cheat sheet of compliant patterns. Patterns are for one needle, repeat for each

12 stitches (48):
- Knit 2, purl 2 till end.
- Knit 3, purl 1 till end.
13 stitches (52): Knit 3, purl 2 twice; then knit 2, purl 1.
14 stitches (56): Knit 3, purl 2 twice; then knit 3, purl 1.
15 stitches (60):
- Knit 3, purl 2 till end.
- Knit 2, purl 1 till end.
16 stitches (64):
- Knit 2, purl 2 till end.
- Knit 3, purl 1 till end.
17 stitches (68): Knit 2, purl 2 twice; then knit 2, purl 1 three times.
18 stitches (72):
- Knit 2, purl 1 till end.
- Knit 3, purl 2 three times; then knit 2, purl 1.
19 stitches (76): Knit 3, purl 2 three times; then knit 3, purl 1.
20 stitches (80):
- Knit 2, purl 2 till end.
- Knit 3, purl 1 till end.
- Knit 2, purl 1 till end.
- Knit 3, purl 2 three times; then knit 2, purl 1.
21 stitches (84):
- Knit 2, purl 1 till end.
- Knit 3, purl 2 three times; then knit 2, purl 2.
22 stitches (88):
- Knit 2, purl 1 six times, then knit 2, purl 2.
- Knit 3, purl 3; then, knit 2, purl 2 till end..
23 stitches (92):
- Knit 2, purl 2 five times; then knit 2, purl 1
- Knit 3, purl 1 five times; then knit 2, purl 1
- Knit 3, purl 2 four times; then knit 2, purl 1
24 stitches (96):
- Knit 2, purl 2 till end.
- Knit 3, purl 1 till end.
- Knit 2, purl 1 till end.
- Knit 3, purl 2 three times; then knit 2, purl 1.
25 stitches (100):
- Knit 3, purl 2 till end.

A Knitting Practice

Thu, 09 Jan 2014 00:00:00 +0000

I’ve not done much knitting for the last year or so. I have a sweater in progress (a cardigan,) and I have an in progress scarf thing, and that’s about it.

Or was.

I enjoy knitting, and find it both relaxing (the rhythm) and stimulating (the meditative aspects, the project planning and design). And I’m pretty good at it. I’ve been knitting (mostly on) for the last 10 years (or so,) and am very technically competent (I think.)

At the same time I lost a lot of interest in knitting. There were and are other things in my life: learning how to make software became (and is) more challenging than making sweaters and shawls, and then there’s singing, a full time (and then some) job, and what not.

I’ve also burnt out on knitting culture. For a long time, part of knitting involved, blogging about knitting, reading knitting blogs, taking pictures of projects and yarn, shopping for yarn, hanging out on ravelry, working in yarn shops, and while knitting itself is a big part of knitting culture I’m realizing a few things:

I don’t really need more sweaters.

I have a huge pile of sweaters that I never wear in my apartment (where storage is a premium,) and have a huge stack of sweaters in my parents house. While a few exceptions, most of the sweaters that I have (the ones here) aren’t particularly wearable in my day to day life. (They’re odd, or a bit loud, or more commonly too damn warm.)

Indeed, I bought this Merino cycling zip-up sweater (from Chrome) and it’s basically my new uniform. This has proven a couple of things, first that I can wear wool against my skin without discomfort and second that the more plain and fitted a sweater is the more likely I am to wear it.
The thing that I like the most about knitting is being able to sit down, and let my hands work and let my mind go thinking about the knitting or about something that I’m writing, or nothing at all.

While I’m good at designing things, and good at doing complicated sweaters, I find that I tend to avoid projects where I end up focusing too much on what needs to happen next, or worry about running out of yarn, or need to follow a pattern.
Knitting is a personal rather than a social act.

I can knit with other people, but it’s not the kind of thing that I want to do when having a conversation, or actively doing something else. I can concentrate on things that are happening around me, but I can’t really read and I can’t really talk much.

The solution seems pretty clear: don’t knit things you don’t want to do, and knit in a way that makes sense for your interests. I’ll continue to work on the projects that I have in progress, and I’ll probably still make sweaters, but I think I’ll be oing a lot more plain knitting I think a lot more sock knitting.

Sock Knitting

Historically I’ve not knit very many socks, I’ve made a half dozen pairs, some more successful than others, most wearable. I can knit a sock completely off book, and am have a basic sock pattern memorized and don’t even really need to do much test knitting.

I finished the pair of socks that I started knitting (probably six months in the running) worsted socks with a wierd ribbing pattern in an evening and then and cast on for a couple of new pairs. Given that I’ve discovered how awesome wool socks are with a new pair of boots that I have (and that I can wear them on my skin without wanting to die.) I’m even more inspired to knit socks.

The new socks are:

a pair of sport weight socks using some striping German sock superwash yarn. 64 stitches, planed 7 inch cuff, 2.5mm (US 1.5) needles, cuff down, 2x2 ribbing.
a pair of fingering weight socks (undyed LB 1878) 80 stitches, planed 7 inch cuff, 2.5mm (US 1.5) needles, cuff down, 2x2 ribbing.

The details probably won’t change much. My hope is to be able to figure out a handknit sock pattern that’s suitable for year round wear.

Knitting Blogging Here

I’ll probably continue to write about knitting here, I like writing, and I’m thinking about it, but I’ll probably address this space more like I would a software or philosophy blog, rather than the more typical knitting blog format. I hope to see you around!

2013 Reflections

Wed, 01 Jan 2014 00:00:00 +0000

I have a hard time passing up a significant calendar event as an opportunity to reflect and synthesize past experiences. Having said that, while I had a pretty good year, it was not a year of big changes. There were a lot of pretty interesting and fun smaller changes and developments in my world and my projects in the last year. In no particular order:

This was the year that I went from being “someone who tinkers on code” to being someone who can write code when the situation calls for it. This felt like an revolutionary change as it happened, but I recognize now that it has been a long time coming and in retrospect
This was the year where I finally slayed my copy quality woes. I’ve long struggled with producing clean copy (English text) in a reasonable amount of time (without spelling/grammar errors.) A couple years ago, I got to a point where I could write clean copy but it felt like I had to spend hours being neurotic. Now, I’ve trained myself to catch things in much more quickly. I’m not perfect, but it’s no longer something that I feel like I struggle with constantly.
I reevaluated my relationship with my hobbies and avocational pursuits. I decided to spend more time learning about technology and computer science, I decided to spend less time writing science fiction and less time knitting. I decided to prioritize my social/folk times to go to more singings both in town and out of town, and focus my dancing around spending time with my friends rather than dancing for the sake of dancing.
This was the year that I decided to stay in New York City. I’ve been here for almost three years now, and I like living here. While I can envision living somewhere else in the right situation, I think I’ve decided to stay for the long haul. This has meant doing things like furniture shopping (though not buying,) and saving for the eventual purchase of an apartment.
I worked too much. But I think I accomplished cool things. In most ways, I think we’re doing some really innovative things with documentation and documentation processes (and writing strong documentation for a cool product.) There have been struggles, as always, but it’s been good. And a lot of work. I don’t think that I would go back and change anything, but I do look forward to trying to establish a little more sanity here.

There are other things, of course, but that covers it. It seems hard to avoid making resolutions, or at least articulating goals at his juncture:
Keep doing the cool things that I’m doing: learning, coding, doing cool things at work.
Develop a reading practice. I love reading things, but I’m not very good at reading regularly and keeping up with my reading habit. I’d like to read novels as well as stay generally on top of Asimov’s (and potentially the New York Review of Science Fiction.)
Develop a more extensive knitting practice. Full post on the subject forth coming, but I want to knit because I enjoy the activity of knitting and the cadence of making things like socks.

I want to stop feeling like my knitting needs to have an increasing complexity to make it interesting, or that every thing I knit needs to be unique in some specific way.
Become proficient in a programming language that isn’t Python.
Start or restart a non-day-job related writing project, and implement a project plan that centers on regular ongoing progress rather than binges.

Coming Soon: Thoughts on implementing goals.

Happy New Year!

Static Site Compiler Design Notes

Sun, 15 Dec 2013 00:00:00 +0000

For a year or more I’ve been playing with the idea of writing my own static site generator. I’ve been producing websites using static site generators for the last 4 or 5 years. For an overwhelming majority of cases, static generation is the right modality; however, there are some significant ways in which the tool chain is not mature.

I’ve been working on an initial pass at a better kind of static site generator, which is a ways away from being “production quality,” but the initial idea is in place, so I feel comfortable discussing some of the details.

Most static site compilers retain simple designs where the process of building the site is non-incremental: all pages are always rebuilt even if the content of the page has not changed. Furthermore, there’s no parallelism, so pages must be built one at a time.

Typically the program will need to process all pages twice, because the rendering phase may require links between content, which complicates the requirements slightly.

The simple design is a huge win for three reasons: it’s easy to understand how these build systems work. There are never any false negatives where pages aren’t rebuilt that should be. Best of all, for small and moderately sized sites, serial/non-incremental approaches are faster.

The downside is pretty big: the amount of time to build the site grows linearly with the number of pages, and there’s no real way to improve performance, particularly as sites become larger.

The design of the “improved” solution is pretty obvious. Iterate through all pages to build and “ingest” their content and metadata. You can do this in parallel, and if possible it makes sense to as much here as you can: build pages that don’t depend on other pages if possible. Also tag pages that include aggregated content here for latter processing.

As you collect the results from these operations, you have enough information to generate the pages that include aggregated information, and you can run and do any remaining rendering.

The only other efficiency is hashing the input page upon initial read as well as the rendered page when you’re done so that you can completely skip pages that don’t need rendering.

For most builds, you might only need to render changed pages; unfortunately, there’s still a fixed cost per-page, which can be amortized somewhat.

For smaller collections of pages, doing everything in a row is faster, and requires less code to get to a proof of concept. As a result, most static site generator implementations follow the serial/non-incremental pattern. The problem is, of course, that this doesn’t hold up as sites grow.

The current state of the art isn’t great:

My current blogging solution, ikiwiki does incremental rebuilds but doesn’t rebuild in parallel.
Sphinx does incremental rebuilds, though it’s spotty in cases, and for thewrite phase only works in parallel (latest version only). The tinkerer blog tool is built on top of Sphinx.

I think it’s possible to do most of the work of the build in parallel, and I’m interested in writing a site compiler that uses a concurent design, targeted not at raw speed but maximum efficiency. My hope is that such a site generator could make these systems more plausible for general content management for organizations and groups that need flexible systems but don’t need the overhead or complexity of database-driven applications.

I’m using the python job runner as a starting point, building using reStructuredText pages in using the general format of Jekyll pages. Supporting markdown doesn’t seem beyond the realm of possibility. The initial implementation is, of course, in Python on account of my personal comfort and affection for reStructuredText, though I think it would be useful to attempt to port the system to Go at some point, no plans for that at the moment.

Onward and Upward!

Buildcloth v0.3 Planning

Sun, 24 Nov 2013 00:00:00 +0000

I spent a lot of time at the end of the summer working on finishing out the basic buildcloth functionality, and haven’t really gotten the chance to use it properly. There were some flaws:

a dumb oversight means that the hash-based dependency checking doesn’t work.
Buildcloth is a bit complicated and designed for a general purpose. In practical terms, I made buildcloth to perform a task that I’ve been able to accomplish with 10% or less of the code.
There’s no good separation between “the management of a build system” and “the build system data” in the system as it currently exists.

Buildcloth is a nifty idea, and one that I’d like to expand upon. Also since the project is still pre-1.0, it seems reasonable to take these lessons and work on building a more usable implementation.

This post is a collection of thoughts on what I’d like to accomplish for 0.3:

pull out the job queue/running system from the build data organization.
collect more state in the dependency checking system/infrastructure.
separate data ingestion from build system organization.
remove ingestion and processing logic from the command line tool.
impose sub-module structure to make the interfaces for all of the different aspects of the program.

No clue about time frame. Feel free (and encouraged) to leave comments if you’re interested in helping or have a feature that you’d really like to see.

Onward and Upward!

Documentation Tooling

Sat, 09 Nov 2013 00:00:00 +0000

I’ve spent a little bit of time building some non-critical tools for my teammates on my work project, which has got me thinking about tooling for documentation systems.

This collection of tools is something that we’ve started to take for granted, but I think it’s pretty novel and worth talking about a bit more.

Documentation toolkits, traditionally refer to the system that deals with the production of the documentation for end use, which typically means taking the source text and rendering it into web sites, pdfs, ebooks, and embedded “online help” text.

These toolkits are really important and I think one of the best things you can do for a documentation tool kit is to produce documentation using a tool specifically designed to address the needs of documentation projects and technical writers.

Unfortunately, documentation production is mostly for readers and business owners of documentation and not really for the writers. Tooling for documentation, particularly the kind that I’ve been spending time on recently is about making documentation easier to maintain, and easier to improve at scale. For example:

tools to make common textual chunks easier to reuse.
tools that enforce common structures.
tools that detect common patterns of weak constructions.
tools that that analyize text for common readability conventions.

I was going to write this up as a blog post, but I think it makes sense as a collection of wiki pages:

Onward and Upward!

Large Sphinx Deployments

Sat, 28 Sep 2013 00:00:00 +0000

I use Sphinx a lot. Both in the sense that I have easily a dozen active (or reasonably so) projects that I maintain and work with on a regular basis. Sphinx is great, and I feel safe asserting that it’s probably the best documentation toolkit in existence and more generally the best tool kit for the production of structured text.

There are flaws. I’ve written here before with greater and lesser descriptions of the pain points of Sphinx. All of the problems are fixable, some fixes are delicate, and some small fixes require major work in some cases. In all, the challenges aren’t insurmountable and Sphinx remains extremely usable and effective.

Like any large and complex system, there are ways to manage resources with Sphinx with greater flexibility and ease. This post explores several ways that have helped me (and my collaborators!) manage big Sphinx deployments.

These suggestions fall into two general categories: suggestions for making Sphinx projects with large volumes of content manageable and strategies for handling and managing larger groups of Sphinx deployments.

Single Source Content

To avoid duplicated content, when possible, it makes sense to reuse content. reStructuredText has an include directive for this purpose. In general, the best strategy is:

maintain a directory of included content that’s distinct from the other directories that hold content. In our projects we use source/includes, where source/ is holds all content.
Use a different extension for the included text than you use for your content files. For example if all of your Sphinx processed rst files use .rst extensions, include files should have .txt extensions.
Smaller, more restricted resources are more effective, typically. Longer bits of text are more difficult to slipstream into the text. If your include snippet requires a section heading its probably too big.

This is most crucial for larger technical resources, and less crucial for other kinds of content, but in general, avoid duplicating common sections when possible.

To make this really awesome you might want to add tooling for yourself/writers so that you can:

see where a file is included in the larger text.
see if any include files are not being used.
detect if any two redirects are substantially similar.

Pre-process and Generate Content

reStructuredText is great for providing a human-friendly way to edit a structured text documents. However, for some kinds of structures its probably better to build the restructured text from some other common structure. For instance: repeated content, tables of contents, image declarations, and dealing with different output for different content types are all good cases for building content yourself.

My main mode of doing this is to use rstcloth to write scripts that read YAML files that contain the content and can be converted into rst content. As an alternative you could write extensions to the reStructuredText processors (docutils) to handle this content, but that may make the production of the documents more (differently?) fragile.

Minimize Configuration Differences

The best thing about Sphinx configuration is that the configuration file itself is a Python module. Which means you can inject pretty much whatever logic you want into the configuration and via html_template_options, the template. This is an awesome power, but if you need to manage more than one similar Sphinx site, the more complex your configuration is, the harder everything becomes. Therefore, tend toward minimal configurations.

I’ve been experimenting with a number of solutions, and I don’t have a “Sphinx Configuration Toolkit” established (yet,) but I’ve been trending toward where the canonical information about the project (urls, theme data, etc.) in a configuration object constructed from a metadata file. Then to populate site-specific lists (interspinx inventories; pdfs, manpages, etc.), I read from other data files. Keeping site-specific data seperate from the configuration code seems to work well.

Take Advantage of the Theme System

Sphinx’s HTML output uses jinja, which is incredibly flexible. To be honest, I kind of wish that the LaTeX builder was also Jinja based, but I’ll take what I can get. Sphinx gives you full access to build and customize really sophisticated display systems. If you’re using default templates, then you can skip this tip.

If you do have custom display code, then take some time to read through the Jinja Documentation and the Sphinx Templating Documentation so you know what’s possible. When developing a template for Sphinx (or in general,) remember the following:

minimize the amount of runtime logic required to render each template. While some template logic is unavoidable, and for some projects the performance hit may not be noticable.

However, putting logic elsewhere (e.g. in the values passed to the template.) makes the data handled by the template as a compile-time rather than run-time cost (plus or minus the memory costs of larger template memory.)
use template inheritance.

Basically, Jinja makes it possible to describe a complete template composed of “blocks.” The blocks don’t have any impact on the output; however, you take this template and use it as a “base” and then describe a new template that is like the base except that you can override some or all of the blocks. This is awesome, and makes it possible to reuse a lot template code without needing to duplicate anything or drive yourself crazy.

It’s easy to forget about this capability when you’re trying to hack something together and template inheritance, like class inheritance in object oriented programming can add complexity and fragility. So you’reprobably well justified in being wary of using inheritance, but give it a shot!

Evaluate Build System Requirements

Sphinx is a documentation tool kit, and it’s very extensible, and awesome. However, in the base configuration it’s not a complete end-to-end publishing system: it doesn’t have built in version control/maintenance, it’s in general only aware of the current build (i.e. the HTML version of your documentation is unaware of the PDF version (and so forth,)) and once Sphinx compiles a site you still have to deploy it somewhere.

In short, to build a website or resource using Sphinx, there are other things the build system before and after running Sphinx to get the product you need. You can reduce a great deal of complexity and provide a number of common points to synchronize multiple projects.

Also, avoid doing crazy things with Makefiles.

Other Useful Optimizations

Download intersphinx inventories independently of Sphinx.

Sphinx will attempt to download each intersphinx inventory each time you build your site and it will download each inventory serially. It’s trivial to do better on your own. Here’s what we do: intersphinx.py
The index objects (which Sphinx uses for all special objects) live in a flat namespace, and collisions are not well handled.

Buildcloth v0.2.0 Release

Sun, 08 Sep 2013 00:00:00 +0000

Today I did a (slightly) more formal release of a software project that I’ve been working on pretty consistently for the last two months. It’s an extension or elaboration on the buildcloth, and is the groundwork for some other projects I’ve been working on.

While there are some new fixes and improvements to the initial meta-build tool components of the project as I’d been working on 5 months ago, this one goes even further and includes a complete build automation tool.

The deal with this is that I’d been running a build system for months that had a bunch of very small tasks, and performance was awful for no really good reason. Well for one reason: process creation. Each task needed to create its own shell, run, and exit, which was awful. The solution to this problem was running each task (which was ultimately just a Python function) in a Python multi-processing pool. The new version of buildcloth is an attempt to build some common infrastructure around this practice.

It still needs some real world testing, and there are some missing features that I’d like to add, and always more documentation, but it’s good enough that I wanted to get it out there so that people could start using it and giving feedback.

I’ll post more later on the the experience and lessons learned here in a bit. While I work on that, see:

Onward and Upward!

Uphill Forever

Sun, 08 Sep 2013 00:00:00 +0000

In many ways this is the follow up to hard is good and the post I promised recounting the lessons of the buildcloth v0.2.0 release

This release of buildcloth is in some ways, the first real piece of software I’ve written from the ground up. I’ve written a bunch of code, and I’ve implemented a decent amount of functionality as extensions and additions to other programs, written some very small programs, and written an endless number of throw away scripts, but never something quite on this scale. The remainder of this post is

Whats the coolest thing about Buildcloth 0.2.0?

The buildsystem feature is pretty awesome, mostly because it makes it possible to have legitimate, honest-to-goodness build systems running with an Python project. Integration is a sweet thing.

Why make another build tool? Aren’t there enough of those already?

I started working on this for two reasons, first because the MongoDB documentation build process was lagging under some process creation overhead and using buildcloth as a meta-build system was clearly not holding up.

Second, I wanted to write a static site generator that used a fully concurrent internal model. My initial plan was to use the buildcloth meta-build system, but that clearly wouldn’t hold up at scale so I needed something like buildcloth.

Finally, there aren’t actually a lot of generalized build automation tools: Make, Ninja, SCons, Waf, and Rake plus a small cluster of Java tools (Ant, Maven, sbt, Gradle). See the wikipedia list of build automation tools.

How can I use Buildcloth? Is Buildcloth right for my project?

If you want, you can use Buildcloth as a Make replacement, using the buildc front end. For build systems that already have Python code to wrap or implement build steps, Buildcloth may be much more efficient than using something like Make. For other kinds of builds, the benefits may be less pronounced.

You can also use Buildcloth as a library in your own Python programs if you need a way of ruining build-jobs in a parallel, dependency aware mode.

I’ve started to think think buildcloth is really a sort of embedable, small-scale, local version of something like celery. Or maybe it’s just a collections of decent wrappers around multiprocessing.Pool. Regardless, there aren’t a lot of really intuitive tools around that make async processing/parallel execution easy and fun in Python, so if you need to do this kind of processing work, take a look.

How well tested is Buildcloth?

There’s a complete unit test suite with 400 tests (last count) that should ensure that things stay stable as the product continues to develop. In this respect buildcloth is really well tested (particularly for a project of its age.) In other respects, its less well tested.

Now that things are comparatively stable, I’m pretty eager to begin using the buildsystem and make sure all of the higher-level aspects work well.

What aspects need the most improvement?

I need to devise a way to save some state between builds so that buildcloth can check to see if a target needs to be rebuilt by seeing if the content of the dependent files has changed. Currently, (like many tools,) buildcloth must rebuild things based on a comparison of modified times from the file system, which do not necessarily indicate a required rebuild.

This isn’t really a buildcloth problem, but I also find myself frustrated by the error reporting of tasks running inside the multiprocessing pool. I’m thinking of wrapping tasks right before calling them in a way that will capture output and make it easier to kill zombie tasks and dead pools.

What would you do differently next time?

I wrote the implementation in a very bottom-up sort of way, and as a result the design and testing suite feels a little bottom up. In the long term I think it was the right decision, but I think that in the medium term it will lead to some awkwardness.

Furthermore, build systems are fundamentally static and there’s no good way to “add jobs to the top of the pipe.” I don’t yet have a good answer to this problem (yet,) but shouldn’t be insurmountable.

Hard is Good

Sun, 04 Aug 2013 00:00:00 +0000

In the hard part of software you could easily chose to read the word “hard” as “sucky,” which makes it seem like a big whine on the topic of “polishing your work is hard and annoying,” but really, that’s unfair.

I think it’s probably better to read the word “hard” as “important” rather than “sucky”.

While you may be able to get someone to try your software on the basis of its core implementation or design, people keep using software because it’s reliable (i.e. “has tests”) understandable (i.e. “has documentation”) and is easy to operate (i.e. “has a user interface.")

Furthermore, doing the hard work of adding infrastructure to a project is what allows software to grow in awesome ways. When you do the hard work, you make it possible to:

add functionality and utility without needing to rewrite large amounts of code.
add options and globules to expose features and behaviors in response to users needs.

Basically infrastructure begets agility.

The Hard Part of Software

Sat, 03 Aug 2013 00:00:00 +0000

I’ve been writing build system tool that allows users to specify concurrent build processes using a lightweight, Python-based system that minimizes overhead.

Progress is decent. I hope to use this to replace a hodgepodge of fabric and Makefile for my work and personal projects. I have a decent spec (3 hours), an initial implementation of the internal parts (3 hours,) a good first draft at a command line utility (1.5 hours,) internal/APO documentation (10 hours,) and none of the unit tests and procedural and conceptual documentation. In essence the hard stuff.

Basically what happened, is I spent a lot of time thinking about the problem, a little bit of time coding, and if all goes according to plan a lot of time writing rather droll code and good if uninteresting documentation.

Which is, all things considered, what all software boils down to.

Writing the core implementation is (often) this intense impassioned process that is necessarily flow-like, because there’s a bunch of state that you have to keep hot in your mind while solving hard problems, and if your attention drifts too far, you start breaking things.

Not that flow-like states are the best or only way to write code for core functionality, but it works and it’s enjoyable.

Everything else, is different:

Writing documentation is an exercise in context switching: you have to read code, or poke at a running program to figure out how it works, and then turn that information on its head so you can tell people how to use it.

It’s fun, but it’s much more fussy.
Writing tests is similarly hard: it’s also about balancing “how it works” and “how its used,” but rather than describing something for future users, tests are about defining what constitutes “correctness” and what’s incorrect.¹

Writing test-code is intellectually challenging work, and requires many of the same base skills as writing implementation-code but requires a different kind of focus and thinking.
There’s a lot of code that remains once the core logic exists, including: user interfaces, logging, test, managing edge cases, optimization, and tuning the parameters of the behavior (business logic tweaks.)

Which isn’t to say that any particularly portion of the work is more or less difficult or important. But, if you don’t work in this world every day it’s easy to see the hard initial work as being “the real part of software development,” and allow all the other work to sort of fade into the background. Which is unfair, and I think is representative of a larger misunderstanding of how software works and gets made.

Another project for another day.

Onward and Upward!

This assumes that you’re not writing code in a test-driven manner, which is I think is probably statistically likely, if somewhat in-ideal. ↩︎

Topic Based Authoring Failure

Sun, 21 Jul 2013 00:00:00 +0000

I wrote a long time ago, about /technical-writing/atomicity which (more or less) is the same as topic based authoring. Both describe the process of breaking information into the smallest coherent blocks and then using the documentation toolkit to compile the kind resource.

Topic based approaches to documentation promise reduced maintenance costs and greater documentation reuse. I’m not sure if anyone’s used “ease of authorship,” as an argument in favor of topic based approaches (they’re conceptually a bit difficult for the author,) but you get the feeling that it was part of the intention.

The obvious parallel is object orientation in programming, and I think the comparison is useful: they both present with optimism about reuse and collaboration through modularity and modern tool chains. While object oriented programming predates topic based authoring, both have been around for a while and even if you aren’t an adherent of object orientation or topic-based authoring, I think it’s impossible to approach programming or documentation without being influenced by either of these paradigms.

Unless you’re working with a really small resource, without some topic-based you end up with redundant documentation that looses consistency and a maintenance nightmare.

The downfalls of “topics,” don’t negate it’s overall utility, but they are significant:

topic based authoring makes it harder for non-writers to contribute to the documentation. This makes it more challenging to keep documentation up to date and can hurt overall accuracy.
topics force writers to focus on the “micro” documentation at the expense of the “macro” documentation experience. The content is clear, the completeness is good, but the overall experience for users is awful.
topic-centrism sometimes leads to deeper hierarchies which leads to duplicated content across the hierarchy as “cousin” nodes address related concepts.

What’s the solution? I’m not sure there is a single one, but:

it’s important to avoid duplication, by having great support for “single sourcing” (inlining/inclusion,) and simple cross referencing.
isolate all content in concrete topical units.
start with flat global organization and add interpage hierarchy only when necessary.
use as much intrapage organization and hierarchy as you need, and allow intrapage hierarchy.
build great reference material first. Everything else is gloss, and you should layer the gloss on top of strong reference rather than try and build reference under an existing structure.

Build Stages

Sat, 20 Jul 2013 00:00:00 +0000

For work, I’ve been working on revising our build system so that less of the build definition happens in Makefiles and more of it happens in Python scripts. This post is an elaboration.

I’m a complete partisan of reusing standard tools, and so moving away from Make felt like a big/hard jump. However:

1. Process creation is expensive, and every “job” starts a new shell and process, which takes time.

2. Most of the build logic was in Python anyway: over time most shell lines called Python code rather than commands directly. This seems like a common artifact of more complex build processes.

Beyond this, the generation of the Makefiles itself was encoded in Python code.

For our project, at least, we were indirecting through Make, sort of through the hell of it.

3. It turns out that multiprocessing in Python is crazy easy to use.

The transition isn’t complete of course, we’re still using make to handle dependencies between groups of tasks, and there’s no particular rush or need to rid ourselves of Make, but the gains are huge. Things build faster one or two orders of magnitude in some cases. There’s less flakiness. Rebuild times are much faster, and there are fewer moving parts.

Great win!

This has me thinking about ways of doing build systems in a generic, maintainable way without relying on something like Make. I have a prototype on my Laptop at the moment that provides a way to specify build processes concurrently. The rest of this post will be a high level overview of the design of this system. Please provide feedback and enjoy!

Build systems are basically collections of tasks expressed in a graph structure. The tools exist to enforce and encode the graph structure, or less abstractly to ensure that tasks run in the proper order. If you’re paining a wall, the build system ensures that you spackle, apply the primer, and then apply the final coat, in that order.

There are, as near as I can tell, three different kinds of relationships among/between groups of tasks in a build process:

1. There are groups tasks that don’t depend on each other and can run concurrently with each other.

2. There are some tasks or groups of tasks that must not run before or after another group of tasks.

3. There are sequences of tasks that must run in a specific order, but can run at the same time as other tasks or sequence of tasks.

What Make, and related systems do is provide a mechanism to specify “dependency” relationships between files (and tasks after a fashion,) or groups of files/tasks. After a fashion, Make takes the dependency information and runs tasks more or less according to one of those patterns. In many ways, my project is an experiment to see if it’s possible to “outsmart Make,” by generalizing the kinds of operations and forcing users to specify the concurrency constraints of the tasks explicitly, rather than letting the concurrency emerge out of the dependency graph. Thoughts:

This depends on users being able to intuit the actual dependencies abstractly, rather than rely on the emergence properties of Make. Arguably, Make also requires you to think abstractly about the potential concurrent modeling of the build, but allows you to avoid it in some situations.
If some large portion of the compilation process relies external processes, the performance gains will probably be more modest. Process creation is still expensive, but it’s probably marginally cheaper to use subprocess than it is to start a full shell.

In addition to the basic machinery, I’ve written a few helper functions to read build definitions from a YAML file, which will produce a usable build system. I’ll release this once: I’ve written some tests, there’s better logging, and some basic README-level documentation.

Onward and Upward!

On Generic Build Systems

Fri, 19 Jul 2013 00:00:00 +0000

I spent a bunch of time this week taking a bunch of my work project’s build system. We’ve gone from having most of the heavy lifting done by Make, to having only doing the general high level orchestration with Make and doing all of the heavy lifting with (simple) custom Python code.

The logic in the previous system was:

Make is everywhere, stable, and consistent. In the spirit of making the project as compatible and accessible to everyone it made sense to use common tools and restrict dependencies.
Concurrency and parallelism are both super hard, and Make provided a way to model the build in a knowable way, and the parallels the build as much as possible. Before starting this project, I’d spent two years being a write working with a build system that was not concurrently and ran in one thread, and I was eager to avoid this problem.

It turns out:

If you write Makefiles by hand they’re the inverse of portable. In the same way that “Portable Bash Script” is a thing that can lead only to insanity.

As Make-based build systems grow, the only thing you can do to preserve your sanity is wrap up all build instructions as scripts of some kind and/or use some sort of meta-build tool to generate the Makefiles programatically, using some sort of meta-build tool.

Complexity abounds.
Forcing you to model your build process as a graph, is actually not a bad thing, and frankly is the strongest selling point. Make doesn’t enforce graphs (and how could you really?) but if you pay attention to the ordering and build performance it’s not hard to keep things running in parallel.

By contrast, Make’s parallel execution is SO BAD. I think the problem is mostly shell/process creation overhead rather than scheduling. Regardless, for build systems with lots of small pieces, you loose a lot to overhead.

So I took the logic that’d I’d been using to generate the Makefiles, implemented some simple mtime based dependency checking, and used it to call the build functions directly in Python.

The results were huge. Speed gains of 300x, using 30% of the code and better processor utilization. Can’t argue with that. Even the steps that required external (i.e. non-Python) sub-processes components were considerably faster

So I was thinking: build systems tend to be big sources of blight, they tend to be hard to maintain and require a bunch of specialized knowledge that’s distinct from the actual domain knowledge of a project, and most generic build tools have problems, (like Make,) so what gives?

If the generic tools had better performance or were significantly easier to maintain, there’d be a rather convincing argument in their favor. As it is, I’m not really seeing it.

Micro Events

Tue, 02 Jul 2013 00:00:00 +0000

I’ve been enjoying blogging over on the micro tychoish site and thought I’d catalog these posts here.

More to come!

Weekend Accomplishments

Sun, 05 May 2013 00:00:00 +0000

(Note: Because I’m terrible at remembering to post entries during the week, this post is actually from last week. But it’s still interesting!)

The past few weeks have been somewhat disjointed for me. I’d been working a lot to wrap up a long expected release, followed by a vacation without a project plan, and a few more busy weeks. On top of this, I spent a bunch of time working on wrapping up, or at least releasing a few personal project to assuage some guilt.

After all that, I found myself at loose ends: I didn’t have new projects because I hadn’t had enough time to think about them or more importantly I was so interested in finishing something that I’d been trying to suppress thinking about new projects.

Well that was a great idea. Not. Now, finally after spending too long lolling about and trying to restart the creative (and project planning) engines, I’ve actually done some things:

RstCloth

Basically this the first break at a very very simple API for generating reStructuredText. It’s modeled on the interface for my buildcloth which does the same sort of thing for generating Makefiles.

reStrucutredText exists to make text easier for humans to write well formed documents, which is great and useful for about 95% of use cases: human editable text formats for machine parsing are an amazing boon to documentation productivity.

There are cases, where it makes more sense to store content in a regular format, like JSON or YAML and build the content programatically, tabular data, integrating content from external sources. If most of your tool chain uses reStructuredText, then something like RstCloth is probably exactly what you need.

And because it’s a second-generation *Cloth tool, I already have most of the awkwardness worked out.

It’s still dev, and I’ll be getting documentation, a readme, and some examples nailed out in the next few weeks. In the meantime:

tumblr -- m.tycho.co

I reactivated my tumblr account, hooked it up to the awesome tumblesocks plugin for emacs, and have attached it to the concise m.tycho.co domain. I’m also mirroring the content at tychoish.com/micro.

I’m going to try to avoid over thinking this, but:

While I’ve had some struggles with the emacs integration, it’s generally really idiomatic.
I really like that you can queue posts. This is the feature that I miss the most about systems that I’ve used in the past to host tychoish.
I like that there are some community features, and that the tagging isn’t worthless in light of being able to use tags to jump to relevant posts from other people.

While I like self-hosted websites, and am kind of freaked out by the whole “my blog is a service,” I think that the connection to a community/audience is useful and powerful, and is not to be overrated.
I like that tumblr does automatic integration with facebook and twitter. You can sort of do this manually, but baking things in leads to a better experience.

New Knitting Project: Cardigan

Mon, 08 Apr 2013 00:00:00 +0000

I’ve mentioned that I was working on a new sweater a few months ago, but I’ve neglected to post or write about the project at all. Let’s change that now:

In most respects it’s just like a number of existing sweaters that I’ve made: two color patterns, using a combination of mid-sized extrapolation of Scandinavian mitten patterns, with some influence of Turkish stocking patterns arranged in panels to convey strong vertical lines. The yarn is Harrisville Shetland, and another unidentified Shetland from a cone I got years ago and have now used in three sweaters. The plan is to have a simple fisherman’s-style drop shoulder construction with a simple short crew neck color.

The plan diverges somewhat from “tychoish standard” in two respects:

The biggest change is that it’s going to be a cardigan. I’ve never made a cardigan that I’d call a rocking success. I can do it, but the finishing always leaves something to be desired and it hangs funny or flares in a way that I don’t want.

The plan for finishing the cardigan opening this time around is to use the steek (the bit that you cut open) as the facing for a hem. the idea is minimal prep and let the yarn do its thing. For closure, I’ll do an attached i-cord band with room for buttons.

The slightly smaller change is that rather than use a hem, I used the “purl-when-you-can-and-want-to” for bottom hem treatment. The idea is that if you purl occasionally for the first few inches you can counteract the tendency of knitted fabric from rolling. It’s not perfect yet, but I’ve not steamed it, so we’ll see.

It’s fun to knit so far, and I look foraward to finally conquering my fear/avoidance of cardigans and perhaps finding the perfect lower edge finishing approach for stranded sweaters.

Onward and Upward!

While I've Been Gone...

Fri, 05 Apr 2013 00:00:00 +0000

… from blogging. See this post for the background.

I sometimes look at other people’s blogs, and think “wow, that’s sharp,” and while I really like the current tychoish theme, there’s a distinct lack of gradients, really polished typography, strong crisp lines, and elegant side bars.

Not that I have a clue what I’d put in a side bar: Hell, I can’t even find good things to put in the Cyborg Institute side bar. But it’s not just that my design has grown dated (I don’t think it has, that much,) and more that the practice of blogging has changed in a few ways:

The State of Blogging

self-hosted blogs are the exception rather than the rule.
it’s become increasingly difficult to aggregate content, the demise of Google Reader, both the removal of the product and the declining trend it its use point to the idea that RSS isn’t a user facing transmission method.

People are getting content through other means, and publishers probably can’t depend that users will poll any content, which changes the role of the publishing system.
In fact, I don’t have a real clue what the current state of the art for publishing tools for blogs is these days. My sense is that a greater portion of blogs are hosted on services like Tumblr and WordPress.com.

The big “advancements,” in blogging technology are probably related to integration and distribution of content to third party systems, which services can probably do better than hosted solutions.
There are fewer long-lived personal blogs, and even fewer that stray beyond a single niche.

Are there blogs that you read regularly? How do you know when there’s a new post?

Changes Afoot

Given these changes, and the chance to rethink how I approach this blog:

I’m curious as to the state of commenting and discourse related to blogs. Do people actually comment, in anything other than exceptional situations? Are most conversations on hacker-news/reddit or other domain specific common space and other blogs?

I’ve been thinking about the prospect of even turning off the discussion/discourse pages here. They don’t get used, they’re kind of weird, people don’t really know how to use them, and I’m not sure they get used. At the same time, providing a space for conversation seems essential. More on this on a later post.

Edit: I totally did this, and while I have some regrets, I think it’s generally a good move.
While it’d be nice to automate submitting content to various aggregation sites and social network-sites, I’ve added various browser extensions to do these submissions. It’s a pain in the ass, but I guess auto-submits makes for less useful content aggregator.
Just as tagging systems are inefficient and broken for wikis and “real” technical resources (see /posts/taxonomic-failure/ for my thoughts,) they’re not all that great for blogs. I’m considering completely removing the tagging system on tychoish, and just letting the search tool (which is pretty good) make content easy to discover.

Onward and Upward!

Doing versus Talking

Tue, 02 Apr 2013 00:00:00 +0000

In On my Return to Blogging post I attributed the fact that I’d taken a break from blogging because I wanted to get out and do things rather than just spend my free time writing and thinking about things.

A Critique

The problem with this kind of statement is that it evokes a certain kind of anti-intellectualism: thinking isn’t as good as doing things, which is counter productive. Actions, creation, feed and grow out of thinking (and vice versa.)

In light of this it’s difficult to re-calibrate ones practice without on the one hand taking an anti-intellectual stance or becoming too ungrounded in practice.

Cogitative Side Effects

I read an article a while back (source lost to the depths of the internet,) that mentioned the following effect; when you talk about something publicly the recognition and validation you get from talking about it is pretty much the same as the recognition and validation you’d get from actually doing something. The result is, if you talk about doing something, you become less likely to actually do it because you’ve already experienced most of the gratification of doing something.

(Sorry for the poor translation.)

In any case, it seems plausible, and certainly worth testing. So when I say “I want to spend time doing things,” rather than theorizing about possible future projects or talking about things I want to work on, as has been my wont, I’m just not.

This is an interesting conundrum for free software/open source: how do you start developing a project in a community centered way without shooting yourself in the proverbial foot. Sometimes it works (e.g. GNU MediaGoblin,) but often people hack a working prototype (and often a lot more) before talking about the project. There are too many examples to list.

There are also a large number of examples of projects that started that languish because they were clearly announced too soon. On the other hand, maybe early-public discussion or announcements is purely epiphenomenal and early public discussion is just a symptom of an always already weak project, that you’re more interested in talking about something that doing something. (Which might just prove the point?)

The Take Away

Don’t blog about something until it exists, and is in a form that you’d be willing to share and discuss.

Corollary: code names are probably the same as real names.

Strive for balance between “project work,” and meta-work. The ideal proportions are unclear.
Avoid anti-intellectualism when possible.

Buildcloth Release, No. 1

Sat, 30 Mar 2013 00:00:00 +0000

Today I released the first version of Buildcloth which is a tool that I’ve been using at work to programatically (and in some cases) dynamically generate build systems (i.e. Makefiles.)

Background

It’s obviously been “production ready” in some sense for a while, but I recently finished the API documentation, and a lot of the infrastructure for packaging and distribution, so it seemed like this was a good starting point.

The initial idea was basically that while Make syntax can be really powerful, in a number of situations:

to specify conditional elements,
to generate build targets and procedures based on system configuration or project state,
for large numbers similar of targets, and
for build with where single targets have a group of related rules,

defining build systems programatically ends up producing a much more reliable and maintainable build system. The wins are pretty big in terms of maintainability, clarity, and flexibility.

The idea, and naming, is sort of: do what fabric does for shell scripts and deployment but for build system generators. Maybe this is exactly what you’re looking for.

More Information

Check it out:

Bugs go here, and patches/pull requests are always welcome.

Cool Improvements:

full documentation.
support for specifying targets/dependencies as a list.
a build-rule abstraction called RuleCloth.
improved ninja support.

The Roadmap

making the tutorial and high level documentation better.
improving the “RuleCloth.”
adding some preliminary tools for managing data interactions.
pypy support (why not?)

On My Return to Blogging

Sat, 30 Mar 2013 00:00:00 +0000

I’ve been a blogging slacker in the last few months. I’ve been working a lot (software releases! content migrations!) and spending my free-time singing and working on a few odds-and-ends projects. And not blogging.

But I did this /posts/delegated-builds project and it seemed like blogging about it would be good.

And it was…

I’ve had this blog, in one form or another for 10 years, and my relationship to this blog has grown and changed a lot in that time and I don’t think it’s useful to really think about all the turns too much, but the recent developments are novel:

I write for a living, and have pretty consistently for the last 4-ish years (holy crap!) It’s not exactly that I’m burnt out on writing, but it does mean that I write differently now, which is a good thing, but I don’t always have the same ability to sit down after work and want to write more, for fun.
I noticed that blogging rigorously meant that I didn’t really have time to work on “research” projects, which is to say, I was putting a lot of energy into writing about ideas and theories but not too much time into actually doing things.

I love theory. I love working on theory, but I’m not sure I see use for theory that doesn’t interact with the world outside of it. For example, before I started my current job I wrote some about technical writing here, and those theories definitely guide what I’m working on today and I think I can stand by what I said, but I think my understanding and knowledge of documentation has grown a lot for the experience of working on it.
To continue on this theme, the programming projects I’ve been working on, which I haven’t blogged about too much here, have been helpful in teaching me a lot about software development, (which I’ve long been fascinated by,) which improves the documentation I write and the way I approach problems.

Furthermore, it means that things I used blog about and say “wouldn’t it be nice if a tool that did existed?” and now I spend a lot of time thinking about how to make them exist. Which isn’t to say that I’m a really fluent programmer (yet,) but I’m not helpless.

So in short, I’m back, and I’m hopeful that in the coming weeks and months I can use this space to talk about the things I’m working on and help build a little bit of (mostly personal) momentum behind these projects.

Onward and Upward!

Delegated Build Questions

Tue, 26 Mar 2013 00:00:00 +0000

This post accumulates what I thought would be the common questions about the /posts/delegated-builds post/tool. For more background see the /posts/build-woes post.

Couldn’t you just have a separate build-only repository?

Sure, but you’d still have to manage that repository which would probably require a non-trivial amount of code and wouldn’t support building/testing topic branches. Furthermore unless you linked the build directories in some way, which this solution does, you’d end up chronically overbuilding.

Doesn’t this use lot of disk space?

Sure, some. But I think in most competitions between disk space and improved productivity, productivity always wins. That not withstanding:

1. Little known fact: when run git clone and specify the remote as a “local” repository, git uses hardlinks, if possible, for it’s objects database. This means, that you’re only copying the source tree, indeed there are two or three copies of the source tree lying around as it is.

2. Most build processes aren’t terribly space intensive: the second checkout is only 7 megs, our .git directory is 7.3 megs (packed), which translates to a 5.4 meg source directory. By contrast the output of a full build of a branch is about 150 megabytes plus production staging.

At least in our case, the additional space costs are effectively trivial both given the size of contemporary hard drives and scale of other size requirements.

Source code may be larger: the MongoDB source tree has a 16 megabyte source tree (not counting 50+ megs of in-tree third party libraries) that becomes tens of gigabytes with build artifacts. Even so, given a project of this scale space costs wouldn’t be hard to justify. Having said that, most software build problems (that I’m aware of,) don’t face this kind of contention, so it’s pretty irrelevant.

This doesn’t make anything faster, so how does it help?

Indeed it probably makes things slower (tests are not yet conclusive,) but it means that any build process can happen entirely in the background and without possibly affecting your current work.

Sometimes the best way to optimize an inefficient process is to apply intelligence and actually make something slow faster. This is great, but it’s also quite hard (and time consuming) and often intelligence can only increase performance by a few percentage points. As a caveat, always make sure that things aren’t slow for a dumb and simple reason, but if an improvement isn’t obvious or there isn’t a simple easy to fix source of slowness, intelligence is often overrated in this regard.

Other times, perhaps even often, the best way to optimize an inefficient process is to make it not matter that it’s slow. Some things take a long time to do, and while it’s great to do things synchronously, it’s not always a real requirement.

This is a smart hack that falls into the second category: if builds are going to take 4 to 6 minutes to run, I don’t want that to prevent things from happening in that time. I don’t want to have to think about coordinating activities around a given period of dead time: this hack solves this handily.

Four to 6 minutes isn’t that long, but it’s starting to get to a point where it’s too long to maintain focus on a task and wait around for a build to finish, particularly for the longer ends.

With this I think we could tolerate ~15 minute builds without really causing a problem. Beyond that and we might need to reopen this case.

Build Woes

Mon, 25 Mar 2013 00:00:00 +0000

I thought I’d back up after the /posts/delegated-builds post and expand on the nature of the build engineering problem that I’ve been dealing with (at my day job) on a documentation related problem.

we do roughly continuous deployment. All active development and editing happens in topic branches, and there’s no really good reason to leave typos and what not on the site any longer than we need to.
we publish and maintain multiple versions of the same resource in parallel, and often backport basic changes to maintenance branches. This is great for users, and great for clarity, but is awful practically, because to deploy continuously, you have to be rebuilding.
all build is self-contained. This isn’t strictly a requirement, and we do use some internal continuous integration tools for internal development, but at the core, for a number of reasons I think it’s important that all writers be able to build the project locally:
as an open source project, it’s important that users can easily contribute at any level. We do lots of things to make it easy for people to submit patches, but if the build isn’t portable (within reason,) then it’s difficult for developers to work as peers.
if it’s difficult to view rendered content while developing, it’s hard to develop well or efficiently. While I think the what-you-see-is-what-you-get model (WYSIWYG) is the wrong answer, good feedback loops are important and being able to build locally, after you make changes, whenever you want, regardless of the availability of a network connection, is terribly important.
the tool we use, Sphinx, in combination with the size of our resource is a bottleneck. A single publication run takes anywhere from 4:30-6 depending on the hardware, and has grows on average 30 seconds every six months. I could rant about parallelism in documentation, but basically, if you want a system that handles cross referencing and internal links, and you want to generate static content, long compile times are mostly unavoidable.

Now there are a number of tricks that we’ve established to fight this underlying truth: Sphinx does some dependency checking to avoid “overbuilding,” which helps some, and I’ve done a lot of mangling in the Makefiles to make the build process more efficient for most common cases, but even so, long growing build times are inevitable.

The Sphinx part of the build has two qualities that are frustrating:

each build is single threaded, so it has to read all the files one by one, and then write each file one by one. You can build other output formats in parallel (with a small hack from the default makefile,) but you can’t get around the speed of a single build. There is a patch in consideration for the next version that would allow the write-stage of the build to run concurrently, but that’s not live yet.
during the read stage of the build, you can’t touch the source files, and extra files in the source tree can affect or break the build, which means that for the most part you can’t build and work at the same time, until now.

The solutions are often not much better than the problem:

use a different build tool, that was built to do incremental builds. The problem is that there aren’t a lot of good options in this area, and the build is really the primary objectionable feature of the build.
improve the build tool, or wait for it to improve. The aforementioned patch to let the write phase run concurrently will help a lot. Having said that, it’s important to keep the project on a standard public release of Sphinx and it’s difficult to modify core Sphinx behavior from the extension system.

Perhaps I have Stockholm Syndrome with the build, but I tend to thing that on some level this is a pretty difficult problem, and building a safe concurrent build system is hard there aren’t a lot of extant solutions. At the same time, this blog is about 2.5 times as large as the documentation project and can do a complete rebuild in 20% of the time or less as much time. While the blog is probably a little less complex, they’re largely similar and it’s not 5-6 times less complex.

I think the problem is that people writing new documentation systems have to target and support new users and smaller test projects, that by the time people have serious problems with the road blocks, the faulty designs are too baked in.
brute force the problem by making use non-local build infrastructure that has faster less contentious processor and disk resources. This sounds good, except our test machines are pretty fast, and the gains made by better hardware don’t keep up with continued growth. While we might gain some speed by moving builds off of our local machines, the experience is quite worse. Furthermore, we do build-non locally, and that’s great, but it’s not a replacement.

There aren’t a lot of solutions and most of them seem to come down to “deal with it and build less,” which is hardly a solution.

This is the foundation of the /posts/delegated-builds script that I wrote, which addresses the problem by making it less intrusive. I’m also working on a brief FAQ, which might help address some of the big questions about this project.

Delegated Builds

Sun, 24 Mar 2013 00:00:00 +0000

Introduction

I cooked up something at work that I think is going to be awesome for building the project that I work on a day to day basis. Here’s the basic problem, in a different post, I’ll expand on these in more depth:

we do continuous deployment.
we maintain and publish multiple branches.
our builds take a non-trivial amount of time (4-6 minutes depending on hardware,) and will continue to get longer.
our documentation toolkit, Sphinx, lacks concurrency for some steps, which means builds take too long and leave most of a contemporary computer idle. Furthermore, given our use of topic-branches it can be hard to get work done during a build.

So there are a couple of notable hacks that I’ve come up with, over the past few months that help:

duplicate some of the initial work so that different output formats can build in parallel (using Make’s job control) at the expense of disk some space.
using a source proxy, (i.e. copying the source content into the build directory and building from this copy so that the actual source files can change during the build.)

These changes are simple and amount to some really minor changes to commands and Make files. This next fix required a non-trivial amount of code, but is really awesome:

building content, if it’s already committed, from a local checkout in the build directory. This way you can build (and publish!) from a different branch without doing anything to your current working directory.

The Code

See the gist for a basic overview, and keep your eyes on the repositories:

Some implementation notes:

it needs a bit more clean up and configuration with regards to a few hard-coded directory names, and assumptions about projects.
in practice it should work fine with Python 2.7 and 3.0. If you have the backported argparse module for 2.6, that should work too.
this plugs in really nicely with some existing infrastructure: becasuse we generate most of our Makefiles, it’s trivial to make this script smart and only permit sane things with regards to branch creation/management, and build targets.

Building on this, I’ve written up a separate script to generate makefile targets to invoke these commands, which allows the script to fit more nicely into the existing idiom. That’s not included here, yet.
this, so far, has been the best introduction I’ve gotten to the subprocess module, so perhaps this will be useful to you.
there’s no good way to queue builds in make, except to use the blocking mode and use a bunch of make calls, which is reasonably inefficient. To get better at this, we’ll need to make some underlying build changes, but the gains could be pretty significant.
finally, this is the first bit of Python I’ve written since I had the breakthrough where I finally understood classes of a moderate amount of complexity without classes. No harm, and it’s not like there’s any internal state; at the same time, a bit of encapsulation around the interactions with git might be useful.

Pull request, suggestions, are always welcome.

More On Delegated Builds

I’ve written a bunch more about this problem and script and will be posting some of that very soon!

(Also it’s good to return to blogging/posting. Thanks for sticking around!)

git raspberry

Sun, 24 Feb 2013 00:00:00 +0000

This is an awful pun, but I’ve recently written the following script to help with some of the work of back-porting patch sets to maintenance branches. Basically you pass it a bunch of commit identifiers and it cherry picks them all in order.

#!/usr/bin/python

import os import sys import subprocess

for commit in sys.argv[1:]: with open(os.devnull, “w”) as fnull: subprocess.call( [‘git’, ‘cherry-pick’, commit ], stdout=fnull, stderr=fnull )

What I’d been doing, previously is assembling commit hashes in an emacs buffer, and then copy-pasting git cherry-pick before each line and then pasting those lines into the shell and hoping nothing goes wrong. The script isn’t much better but it’s a start.

To use, save in a file named git-raspberry in your $PATH, chmod +x the file, and then just run “git raspberry”. Turns out git runs any program in the path that starts with git- as a sub command.

The more you know.

New Knitting Project: Ballstown

Sat, 09 Feb 2013 00:00:00 +0000

I’ve started a new project, much to my own surprise. After many years of looking at the merino/tencel blend “colrain” I ordered a cone of it, and have cast on a project: a plain tube using size 0s..

I think I may be crazy.

The thing is, I got one of these neck tubes a month or two ago, and it’s the most amazing thing ever. Looks good with most things, not weird, very comfortable, etc.

So I’m making myself one…

I’m calling it “Ballstown” after a tune in the Sacred Harp of the same name. The tune is named after the town in the capital region of New York State, now known as “Ballston Spa.” Why? Because I cast on 217 stitches.

It turns out, I’ve really rather missed plain knitting that you can just knit on for hours without really thinking about, or can knit on in the dark.

One of the reasons that I’ve not been knitting as much recently, other than available time is that I’ve found it difficult to actually wear or use the things I knit. Sweaters, even finer weight ones are too warm to wear inside, and not windproof enough to keep me warm outside without substantial jacket.

The answer is to knit finer fabrics, of course, but this has been easier said than done, for me. Mostly I’ve stuck to fair isle sweaters, which are great fun to knit, and reasonably wearable, but difficult to knit on casually: lots to lug around, and starting to knit something with a pattern requires some “spin up time,” as you remember where you were and what you’re supposed to be doing.

In most ways this plain tube is the perfect answer to this problem….

I’ll blog more about this (or not,) as I progress.

Sphinx Caveats

Fri, 08 Feb 2013 00:00:00 +0000

This is a rough sketch of some things that I’ve learned about the Sphinx documentation generation system. I should probably spend some time to collect what I’ve learned in a more coherent and durable format, but this will have to do for now:

If you describe a type in parameter documentation it will automatically link to the Python documentation for that type when using the Python Domain and if you have intersphinx connected. That’s pretty cool.
Sphinx let’s you define a scope for a file in some cases. If you’re documenting command-line options to a program. (i.e. with the “program” with subsidiary “option” directives,) or if you’re documenting Python objects and callables within the context of a module, the module and program directives have a scoping effect.

Cool but it breaks the reStructuredText idiom, which only allows you to decorate and provide semantic context for specific nested blocks within the file. As in Python code, there’s no way to end a block except via white-space,¹ which produces some very confusing markup effects.

The “default-domain” directive is similarly… odd.
Sphinx cannot take advantage of multiple cores to render a single project, except when building multiple outputs (i.e. PDF/LaTeX, HTML with and/or directories.) if with a weird caveat that only one builder can touch the doctree directory at once. (So you either need to put each builder on its own doctree directory, or let one build complete and then build the reset in parallel.)

For small documentation sets, under a few dozen pages/kb, this isn’t a huge problem, for larger documentation sets this can be quite frustrating.²

This limitation means that while it’s possible to write extensions to Sphinx to do additional processing of the build, in most cases, it makes more sense to build custom content and extensions that modify or generate reStructuredText or that munge the output in some way. The include directive in reStructuredText and milking the hell out of make are good places to started.
Be careful when instantiating objects in Sphinx’s conf.py file: since Sphinx stores the pickle (serialization) of conf.py and compares that stored pickle with the current file to ensure that configuration hasn’t changed (changed configuration files necessitate a full rebuild of the project.) Creating objects in this file will trigger full (and often unneeded) rebuilds.
Delightfully, Sphinx produces .po file that you can use to power translated output, using the gettext sphinx builder. Specify a different doctree directory for this output to prevent some issues with overlapping builds. This is really cool.

Sphinx is great. Even though I’m always looking at different documentation systems and practices I can’t find anything that’s better. My hope is that the more I/we talk about these issues and the closer I/we’ll get to solutions, and the better the solutions will be.

Onward and Upward!

In Python this isn’t a real problem, but reStructuredText describes a basically XML-like document, and some structures like headings are not easy to embed in rst blocks. ↩︎
reality documentation sets would need to be many hundreds of thousands of words for this to actually matter in a significant way. I’ve seen documentation take 2-3 minutes for clean a regeneration using Sphinx on very powerful hardware (SSDs, contemporary server-grade processors, etc.), and while this shouldn’t be a deal breaker for anyone, documentation that’s slow to regenerate is harder to maintain and keep up to date (e.g. difficult to test the effect of changes on output, non-trivial to update the documents regularly with small fixes.) ↩︎

Stability is a Crutch

Wed, 23 Jan 2013 00:00:00 +0000

I don’t think the tension between having good, robust, and bug-free software and having software with new features and capabilities is solvable in the macro case. What follows is a musing on this subject, related in my mind to the On Installing Linux post.

I’m not exactly making the argument that we should all prefer to use unstable and untested software, but I think there is a way in which the stability¹ of the most prevalent Linux distributions is a crutch. Because developers can trust that the operating system will effectively never change, there’s no need to write code that expects that it might change.

The argument for this is largely economic: by spacing updates out to once a year or once every 18 months, you can batch “update” costs and save some amount of overhead. The downside here is that if you defer update costs, they tend to increase. Conversely, its difficult to move development forward if you’re continuously updating, and if your software is too “fresh,” you may loose time to working out bugs in your dependencies rather than your system itself.

The logic of both arguments holds, but I’m not aware of comparative numbers for the costs of either approach. I’m not sure that there are deployments of significant size that actually deploy on anything that isn’t reasonably stable. Other factors:

automated updating and system management.
testing infrastructure.
size of deployment.
number and variety of deployment configurations.

Reliably updated and patched regularly for several years of maintenance, but otherwise totally stable and static. ↩︎

Markdown Standardization

Sun, 20 Jan 2013 00:00:00 +0000

I (mostly) lurk on the markdown discussion list, which is a great collection of people who implement and develop projects that use markdown. And there’s been a recent spate of conversation about standardization of markdown implementations and syntax. This post is both a reflection on this topic and a brief overview of the oft-forgotten history.

A History of Markdown Standardization

Markdown is a simple project that takes the convention that most people have been using to convey text formatting and style in plain text email, and providing a very minimalist and lightweight script that translates this “markup” (i.e. “markdown”) into HTML. It’s a great idea, and having systems that make it possible for people to focus on writing rather than formating is a great thing.

People should never write XML, HTML, or XHTML by hand. *Ever*.

Here’s the problem: the initial implementation is a Perl script that uses a bunch of pattern matching regular expressions (as you’d expect) to parse the input. It’s slow, imprecise, there are a few minor logical bugs, there’s no formal specification, and the description of the markdown language are ambiguous on a few key questions. Furthermore, there are a number of features that are simple, frequently requested/desired, with no official description of the behavior.

As people have gone about developing markdown implementations and extensions in other languages, to fix up the inconsistencies, to provide markdown support in every programming language under the sun, without a formal specification and disambiguation of the open question, the result is fragmentation: all the implementations are subtly different. Often you’ll never notice, but if you use footnotes (which are non-standard,) or want to have nested lists, you will end up writing implementation-dependent markdown.

The result is that either you tie your text to a specific implementation, or you go blithely on with the knowledge that the markdown that you write or store today will require intervention of some sort in the future. If you need to extend markdown syntax, you can’t without becoming an implementer of markdown itself.

That’s an awful thing. And there’s no real path out of this: the originator of markdown has publicly stated that he has no interest in blessing a successor, continuing development of the reference implementation, or in contributing to a specification process. Insofar as he controls the authoritative definition of markdown, the project to standardize markdown is dead before it even begins.

The problem is that while most people involved (implementers, application developers, etc.) in markdown want some resolution to this problem: it’s bad for users and it makes implementing markdown difficult (which markdown flavor should you use? should you reimplemented bugs for consistency and compatibility, or provide a correct system that breaks compatibility?) At the same time, markdown implementations are not commercial products and were built to address their author’s needs, and none of those maintainers really have the time or a non-goodwill interest in a standardization process.

If markdown standardization weren’t doomed from the start, the fact that the only people with any real ability to rally community support for a standardized markdown, are not inclined to participate in a standardization process.

Markdown Isn’t For Text That Matters

If markdown were better, more clear, and more rigorously defined and implemented, this wouldn’t be a problem, but the truth is that markdown’s main role has been for README files, blog posts, wikis, and comments on blog posts and in discussion forms.

It’s a great “lowest common denominator” for multi-authored text that needs rich hypertext features but needs markdowns simplicity and intuitiveness. Big projects? Multi-file projects? Outputs beyond single files?

Sure you can hack it with things like maruku and multi-markdown to get LaTeX output, and footnotes, and more complex metadata. And there are some systems that make it possible to handle projects beyond the scope of a single file, but they’re not amazing, or particularly innovative, particularly at scale.

To recap, markdown is probably not an ideal archival format for important text, because:

The implementation-dependency means that markdown often fails at genericism, which I think is supposed to be it’s primary features.

Generic text representation formats are a must.
If you need output formats beyond HTML/XHTML then markdown is probably not for you.

You can get other formats, but it’s even more implementation specific.

The Alternatives

Don’t standardize anything. While markdown isn’t perfect the way it is now, there’s no real change possible that wouldn’t make markdown worse. There are two paths forward, as I see it:

Give up and use reStructuredText for all new projects.

RST is fussy, but has definite and clear solutions to the issues that plague markdown.
- It has support for every major output format, and it wouldn’t be too hard to expand on that.
- It’s fast.
- In addition to the primary implementation, Pandoc supports python and there are early stage Java/PHP implementations. Most tools just wrap the Python implementation, which isn’t really a problem.
- There are clear paths for extending rst as needed for new projects.
Design and implement a new markdown -like implementation. I think reMarkdown would be a good name. This will be a lot of work, and have the following components:
- a complete test suite that other implementations could use to confirm compatibility with reMarkdown.
- a formal specification.
- a lexer/parser design and reference implementation. With an abstract XML-like output format. We want a realistic model implementation that isn’t overly dependent upon a single output format.

an explicit and defined process for changing and improving the syntax.

On Installing Linux

Tue, 08 Jan 2013 00:00:00 +0000

(alternately, “Installing Linux the Hard Way")

I’ve had the occasion to install Linux on three systems in the recent past. People don’t really install Linux anymore, it seems: with “cloud” instances and provisioning that’s based on images means that no one really has to install Linux as such. My experiences have been mostly awful:

I couldn’t make my current laptop do a full LCM boot for the life of me. I partitioned the hard drive in the conventional way, and while the system works fine, I think non-abstracted disk volumes are bad practice.

Disk partitioning and bootloaders remain the most difficult and frustrating aspect of the installation process, and there’s no automation to support this work. Furthermore, even if it takes you a day to get it right, usually you don’t have to mess with it for a year or two. Which makes it difficult to improve practically.

The Debian installer will do this pretty well, but you can’t get the auto partitioning tool to not use the full disk. Or I can’t figure it out.
I recently tried to install Arch Linux on an infrastructural system. Apparently in the last couple of months Arch totally did away with the installation system. So it dumps you into a mostly working shell and provides a couple of shell scripts to “automate” the installation.

It’s a great idea, as long as you never have to use it.

Conversely, it’s a great idea if you’re constantly running installations.

If you install Arch once every year or two, as I suspect is the most common case, good luck.

I need to do it again: to update an older laptop to the 64-bit version of Arch, and I fear this is going to be terribly painful. I’m left with two main questions:

1. Have we given up on the idea that desktop Linux may be viable for people who aren’t already familiar with Linux, or who aren’t software developers (or the next best thing?)

Does the desktop experience actually matter?

I’m asking this in a more narrow line of questioning. There’s computer usage that revolves around things that happen in the browser, which is (probably) better suited for embeded systems (i.e. Android and iOS based devices,) and it’s not clear where the line between that and “General Purpose” computing will fall.

If we end up using embeded systems for most of the computers that we actually touch, this fundamentally changes the desktop experience as we know it, particularly for things like installation.

Three Way Merge Script

Sat, 05 Jan 2013 00:00:00 +0000

Note: This is an old post about a script I wrote a few months ago about a piece of code that I’m no longer (really) using. I present it here as an archival piece with a boatload of caveats. Enjoy!

I have a problem that I think is not terribly unique: I have a directory of files and I want to maintain two distinct copies of these files at once, and I want a tool that looks at both directories and makes sure they’re up to date. That’s all. Turns out nothing does exactly that, so I wrote a hacked up shell script, and you can get it from the code section:

merge-script

I hope you enjoy!

Background

You might say, “why not just use git to take care of this,” which is fair. The truth is that I don’t really care about the histories as long as there’s revision. Here’s the situation:

I keep a personal ikiwiki instance for all of my notes, tasks, and project stuff. There’s nothing revolutionary, and I even use deft, dired, and some hacked up lisp to do most of the work. But I also work on a lot of projects that have their own git repositories and I want to be able to track the notes of some of those files in those repositories as well.

Conflicts.

There are some possible solutions:

1. Use hard links so that both files will point at the same data on disk.

Great idea, but it breaks on multiple systems. Even if it might have worked in this case, it freight ens me to have such fragile systems.

Note: the more I play with this, the less suitable I think that it might be for multi system use. If one or both of the sides is in a git repo, and you make changes locally and then pull changes in from a git upstream, the git files, may look newer than the files that you changed. A flaw.

2. Only edit files in one repository or the other, and have a pre-commit hook, or similar, that copies data from the new system to the old system.

I rejected this because I thought I’d have a hard time enforcing this behavior.

3. Write a script that uses some diff3 to merge (potential) changes from both sources of changes.

This is what I did.

The script actually uses the merge command which is a wrapper around diff3 from rcs. shrug.

Beyond my somewhat trivial and weird use-case, I actually think that this script is more useful for the following situation:

You use services like Dropbox as a way of getting data onto mobile devices (say,) but you want the canonical version of the file to live in a git repository on your system.

This is the script for you.

I hope you enjoy it!

Today's Bottleneck

Fri, 04 Jan 2013 00:00:00 +0000

Computers are always getting faster. From the perspective of the casual observer it may seem like every year all of the various specs keep going up, and systems are faster.¹ In truth, progress isn’t uniform across all systems and subsystems, and thinking about this progression of technology gives us a chance to think about the constraints that developers² and other people who build technology face.

For most of the past year, I’ve used a single laptop, for all of my computing work, and while it’s been great, in this time I lost touch with the comparative speed of systems. No great loss, but I found myself surprised to learn that all computers did not have the same speed: It wasn’t until I started using other machines on a regular basis that I remembered that hardware could affect performance.

For most of the past decade, processors have been fast. While some processors are theoretically faster and some have other features like virtualization extensions and better multitasking capacities (i.e. hyperthreading and multi-core systems) the improvements have been incremental at best.

Memory (RAM) manages to mostly keep up with the processors, so there’s no real bottleneck between RAM and the processor. Although RAM capacities are growing, at current volumes extra RAM just means services/systems that had to be distributed given RAM density can all run on one server. In general: “ho hum.”

Disks are another story all together.

While disks got faster over this period, they didn’t get much faster during this period, and so for a long time disks were the bottle neck in computing speed. To address this problem, a number of things changed:

We designed systems for asynchronous operation.

Basically, folks spilled a lot of blood and energy to make sure that systems could continue to do work while waiting for the disk to reading or writing data. This involves using a lot of event loops, queuing systems, and so forth.

These systems are really cool, the only problem is that it means that we have to be smarter about some aspects of software design and deployment. This doesn’t fix the tons of legacy sitting around, or the fact that a lot of tools and programmers are struggling to keep up.

We started to build more distributed systems so that any individual spinning disk is responsible for writing/reading less data.
We hacked disks themselves to get better performance.

There are some ways you can eek out a bit of extra performance from spinning disks: namely RAID-10, hardware RAID controllers, and using smaller platters. RAID approaches use multiple drives (4) to provide simple redundancy and roughly double performance. Smaller platters require less movement of the disk arm, and you get a bit more out of the hardware.

Now, with affordable solid state disks (SSDs,) all of these disk related speed problems are basically moot. So what are the next bottlenecks for computers and performance:
Processors. It might be the case that processors are going to be the slow to develop bottleneck. There are a lot of expectations on processors these days: high speed, low power consumption, low temperature, high amount of parallelism (cores and hyperthreading.) But these expectations are necessarily conflicting.

The main route to innovation is to make the processors themselves smaller, which does increase performance and helps control heat and power consumption, but there is a practical limit to the size of a processor.

Also, no matter how fast you make the processor, it’s irrelevant unless the software is capable of taking advantage of the feature.
Software.

We’re still not great at building software with asynchronous components. “Non-blocking” systems do make it easier to have systems that work better with slower disks. Still, we don’t have a lot of software that does a great job of using the parallelism of a processor, so it’s possible to get some operations that are slow and will remain slow because a single threaded process must grind through a long task and can’t share it.
Network overhead.

While I think better software is a huge problem, network throughput could be a huge issue. The internet endpoints (your connection) has gotten much faster in the past few years. That’s a good thing, indeed, but there are a number of problems:
Transfer speeds aren’t keeping up with data growth or data storage, and if that trend continues, we’re going to end up with a lot of data that only exists in one physical location, which leads to catastrophic data loss.

I think we’ll get back to a point where moving physical media around will begin to make sense. Again.
Wireless data speeds and architectures (particularly 802.11x, but also wide area wireless,) have become ubiquitous, but aren’t really sufficient for serious use. The fact that our homes, public places, and even offices (in some cases) aren’t wired correctly to be able to provide opportunities to plug in will begin to hurt.

Thoughts? Other bottlenecks? Different reading of the history?

By contrast, software seems like its always getting slower, and while this is partially true, there are additional factors at play, including feature growth, programmer efficiency, and legacy support requirements. ↩︎
Because developers control, at least to some extent, how everyone uses and understands technology, the constrains on the way they use computers id important to everyone. ↩︎

Github without Github

Wed, 02 Jan 2013 00:00:00 +0000

Github is great, and I think they’ve done a lot--for the better--to change and shape the way that everyone uses and does really awesome things with git.

But I worry about lock-in, I worry about having a project that relies on some feature of github that can’t be easily accomplished on another platform.

This post is an index of “git ecosystem” tools that let you get something that looks a bit like github on your own servers. Feel free to edit this page (it’s a wiki!) if you have other tools you like or can recommend!

Permissions Control

Wiki / Pages

Github has a wiki system that’s open source. I’ve never played around with it, because there’s ikiwiki which is better anyway.

Their page’s functionally is also open source, it’s Jekyll there’s no particular shortage of programs that do this kind of thing, and most aren’t that good but that’s an orthogonal point.

Hosted Solutions

There are about a million different repository viewers, but the magic of the github website is that there’s a lot of other integrated functionality (bug tracking, merge request queues, automatic forking/branching/etc.

gitorious - functional but inelegant.
gitlab - promising but untested.
repo.or.cz - functional but not practical for casual administration.

Web hooks

I’ve only recently found notify-webhook, but it basically implements something like github’s service hooks, as a traditional git post-recieve hook.

The Rest

There’s no great stand alone merge (pull) request system. Other code review tools are uneven, but the truth is that pull requests are not a sufficiently advanced code review tool.

Patchwork might work, but its a bit rustic for contemporary workflows.

Integrated issue tracking, hell any kind of issue tracking, remains an unsolved problem, but I think github’s approach is a good start, and that feature set isn’t as easily available from other projects/products/tools.

Knitting Shopping

Wed, 02 Jan 2013 00:00:00 +0000

… and planning

I did a little bit of holiday knitting shopping. Given how infrequently I buy yarn and knitting things and the fact that shopping for knitting things correlates strongly with my project planning, it seems worth sharing:

I got a cone of merino/tencel lace weight yarn in a steel blue color to knit a long plain tube to wear as a neck tube/scarf thing. I bought one of these a few months ago knit out of a jersey tencel knit, and I adore it, so it makes sense to knit something similar.

Hopefully knitting these scarves will prove successful and useful. I’m not much of a knitted sock wearer, I find most flat scarves dreadful to knit, I find shawls difficult to pull off, and I enjoy knitted hats but don’t find them windproof enough for common use. Having good, small, lightweight, and plain knitting projects would probably be very good thing indeed.
I bought a couple of carbon fiber knitting needles, in sizes 2.5 (the size that my sweaters have been and will be for a little while,) and 0s (for the scarfs and hem facings as needed.)

I’m a chronic needle bender and like sharp points and reasonably slippery needles. I also have a set of carbon fiber needles for socks which are great. Very much looking forward to trying these out.
Finally, I’ve procured a few cones of HD Shetland to complement some of my left overs. On the sweater queue:
- a few cardigans. One for my mother, and a second one for me?
- maybe something with shoulders/sleeves in a different color? I’ve used this kind of shading on otherwise plain sweaters, but it seems interesting to see how it might look on a two color sweater.

Sweater Evolutions

Wed, 02 Jan 2013 00:00:00 +0000

I’ve been knitting! Here’s an update:

I finished a sweater. Still need to block it, but it looks great so far. It’s a (near) duplicate of a sweater that I made a few years ago in blues. The biggest difference in construction is that I did the hem in a very slightly different way. Other than that, it’s very much the exemplar of “the default tychoish sweater.”

Note to self, it would be good to have a version of this sweater in brown.
I started another sweater. This one is a medium gray and light blue-gray as a cardigan. Rather than do hems, the idea with this is:
- purl when you can and want to border.
  
  This is an Elizabeth Zimmerman and Meg Swansen technique for colorwork to avoid ribbing or hems. If you purl occasionally in the first few inches, you can prevent rolling. Seems to work well enough, and it makes the bottom edge less bulky and more integrated into the sweater.
- Crocheted front steek with knitted cord (i-cord) edge, using the steek as facing.
  
  Another Meg Swansen technique where you use the steek as facing, by crocheting along the edges and then knitting an i-cord to cause the steak to “fold” under and act as a facing. Blocking does the rest. Again, the end result is lightweight, flexible, and easy to handle.
I have about five inches done, and I expect slow but steady progress on this over the next few months.

Documentation Maximalism

Tue, 01 Jan 2013 00:00:00 +0000

You may hear people, particularly people who don’t like to write documentation, something like:

Users need minimalist documentation that only answers their questions, and there’s no point in overwhelming users with bloated, maximalist documentation that they’ll never read.

Which sounds great, but doesn’t reflect reality or best practice. Consider the following:

Documentation is as much for the producers of the software as it is for the users. Having extensive documentation contributes too, and reflects a sane design process. Collecting and curating the documentation helps ensure that the software is usable and knowable.
Having complete documentation reduces support costs, both by reducing the volume of support requests and by lowering the complexity of the work associated with support.
Good extensive documentation drives adoption of software. Products with better documentation will always see better adoption than comparable products with worse documentation.

Without users, software is useless.

Does this mean that you shouldn’t value minimalism, particularly conceptual minimalism, and visual minimalism? Does this mean you should avoid customizing the documentation to fit the needs and patterns of your users?

No, of course not.

But minimalism for the sake of minimalism, without a particular strategy is an awful and ill-gotten ideology.

Even if it looks good, and particularly if it sounds good.

Information Debts

Tue, 18 Dec 2012 00:00:00 +0000

Like technical debt, information debt is a huge problem for all kinds of organizations, and one that all technical writers need to be aware and able to combat directly. Let’s backup a little…

Information debt is what happens when there aren’t proper systems, tools, and processes in place to maintain and create high quality information resources. A number of unfortunate and expensive things result:

People spend time recreating documents, pages, and research that already exists. This is incredibly innefficent and leads to:
Inaccurate information propagates throughout the organization and to the public.
Information and style “drifts,” when information and facts exist in many places.
Organizations spend more money on infrastructure and tools as a band-aid when data is poorly organized.
People lose confidence in information resources and stop relying on them, preferring to ask other people for information. This increases the communication overhead, noise level, and takes longer for everyone, than using a good resource.

To help resolve information debt:

Dedicate resources to paying back information debts. It takes time to build really good resources, to collect and consolidate information, and to keep them up to date. But given the costs of the debt, it’s often worth it.
Documents must be “living,” usefully versioned, and there must be a process for updating documents. Furthermore, while it doesn’t make sense to actually limit editing privileges, it’s important that responsibility for editing and maintaining documents isn’t diffused and thus neglected.
Information resources, must have an “owner” within an organization or group who is responsible for keeping it up to date, and making sure that people know it exists. You can have the best repository for facts, if no one uses it and the documents are not up to date, it’s worthless.
Minimize the number of information resources. While it doesn’t always make sense to keep all information in the same resource or system, the more “silos” where a piece of information or document might live the less likely a reader/user will find it.

… and more.

I’m working on adding a lot of writing on information debt in the technical writing section of the wiki. I’ll blog more about this, while I continue to work through some of these ideas, but I’m quite interested in hearing your thoughts on this post and on the information-debt pages as well.

Onward and Upward!

Novel Automation

Tue, 11 Dec 2012 00:00:00 +0000

This post is a follow up to the interlude in the /posts/programming-tutorials post, which part of an ongoing series of posts on programmer training and related issues in technological literacy and education.

In short, creating novel automations is hard. The process would have to look something like:

Realize that you have an unfulfilled software need.
Decide what the proper solution to that need is. Make sure the solution is sufficiently flexible to be able to support all required complexity.
Then sit down, open an empty buffer and begin writing code.

Not easy.¹

Something I’ve learned in the past few years is that the above process is relatively uncommon for actual working programmers: most of the time you’re adding a few lines here and there, testing various changes or adding small features built upon other existing systems and features.

If this is how programming work is actually done, then the kinds of methods we use to teach programmers how to program should hold some resemblance to the actual work that programmers do. As an attempt at a case study, my own recent experience:

I’ve been playing with Buildbot for a few weeks now for personal curiosity, and it may be useful to automate some stuff for the Cyborg Institute. Buildbot has its merits and frustrations, but this post isn’t really about buildbot. Rather, the experience of doing buildbot work has taught me something about programming and about “building things,” including:

When you set up buildbot, it generates a python configuration file where all buildbot configuration and “programming” goes.

As a bit of a sidebar, I’ve been using a base configuration derived from the buildbot configuration for buildbot itself, and the fact that the default configuration is less clean and a big and I’d assumed that I was configuring a buildbot in the “normal way.”

Turns out I haven’t, and this hurts my (larger) argument slightly.

I like the idea of having a very programmatic interface for systems that must integrate with other components, and I really like the idea of a system that produces a good starting template. I’m not sure what this does for overall maintainability in the long term, but it makes getting started and using the software in a meaningful way, much more possible.
Using organizing my buildbot configuration as I have, modeled on the “metabuildbot,” has nicely illustrated the idea software is just a collection of modules that interact with each other in a defined way. Nothing more, nothing less.
Distributed systems are incredibly difficult to get people to conceptualize properly, for anyone, and I think most of the frustration with buildbot stems from this.
Buildbot provides an immediate object lesson on the trade-offs between simplicity and terseness on the one hand and maintainability and complexity on the other.

This point relates to the previous one. Because distributed systems are hard, it’s easy to configure something that’s too complex and that isn’t what you want at all in your Buildbot before you realize that what you actually need is something else entirely.

This doesn’t mean that there aren’t nightmarish Buildbot configs, and there are, but the lesson is quite valuable.
There’s something interesting and instructive in the way that Buildbot’s user experience lies somewhere between “an application,” that you install and use, and a program that you write using a toolkit.

It’s clearly not exactly either, and both at the same time.

I suspect some web-programming systems may be similar, but I have relatively little experience with systems like these. And frankly, I have little need for these kinds of systems in any of my current projects.

Thoughts?

Indeed this may be why the incidence of people writing code, getting it working and then rewrite it from the ground up: writing things from scratch is an objectively hard thing, where rewriting and iterating is considerably easier. And the end result is often, but not always better. ↩︎

Programming Tutorials

Mon, 10 Dec 2012 00:00:00 +0000

This post is a follow up to my :doc`/posts/coding-pedagogy` post. This “series,” addresses how people learn how to program, the state of the technical materials that support this education process, and the role of programming in technology development.

I’ve wanted to learn how to program for a while and I’ve been perpetually frustrated by pretty much every lesson or document I’ve ever encountered in this search. This is hyperbolic, but it’s pretty close to the truth. Teaching people how to program is hard and the materials are either written by people who:

don’t really remember how they learned to program.

Many programming tutorials were written by these kinds of programmers, and the resulting materials tend to be decent in and of themselves, but they fail to actually teach people how to program if they don’t know how to program already.

If you already know how to program, or have learned to program in a few different languages, it’s easy so substitute “learning how to program,” with “learn how to program in a new language” because that experience is more fresh, and easier to understand.

These kinds of materials will teach the novice programmer a lot about programming languages and fundamental computer science topics, but not anything that you really need to learn how to write code.

people who don’t really know how to program.

People who don’t know how to program tend to assume that you can teach by example, using guided tutorials. You can’t really. Examples are good for demonstrating syntax and procedure, and answering tactical questions, but aren’t sufficient for teaching the required higher order problem solving skills. Focusing on the concrete aspects of programming syntax, the standard library, and the process for executing code isn’t enough.

These kinds of documents can be very instructive, and outsider perspective are quite useful, but if the document can’t convey how to solve real problems with code, you’ll be hard pressed to learn how to write useful programs from these guides.

In essence, we have a chicken and egg problem.

Interlude:

Even six months ago, when people asked me “are you a programmer?” (or engineer,) I’d often object strenuously. Now, I wave my hand back and forth and say “sorta, I program a bit, but I’m the technical writer.” I don’t write code on a daily basis and I’m not very nimble at starting to write programs from scratch, but sometimes when the need arises, I know enough to write code that works, to figure out the best solution to fix at least some of the problems I run into.

I still ask other people to write programs or fix problems I’m having, but it’s usually more because I don’t have time to figure out an existing system that I know they’re familiar with and less because I’m incapable of making the change myself.

Even despite these advances, I still find it hard to sit down with a blank buffer and write code from scratch, even if I have a pretty clear idea of what it needs to do. Increasingly, I’ve begun to believe that this is the case for most people who write code, even very skilled engineers.

This will be the subject of an upcoming post.

The solution(s):

1. Teach people how to code by forcing people to debug programs and make trivial modifications to code.

People pick up syntax pretty easily, but struggle more with the problem solving aspects of code. While there are some subtle aspects of syntax, the compiler or interpreter does enough to teach people syntax. The larger challenge is getting people to understand the relationship between their changes and behavior and any single change and the reset of a piece of code.

2. Teach people how to program by getting them to solve actual problems using actual tools, libraries, and packages.

Too often, programming tutorials and examples attempt to be self-contained or unrealistically simple. While this makes sense from a number of perspectives (easier to create, easier to explain, fewer dependency problems for users,) it’s incredibly uncommon and probably leads to people thinking that a lot of programming revolves around re-implementing solutions to solved problems.

I’m not making a real argument about computer science education, or formal engineering training, with which I have very little experience or interest. As contemporary, technically literate, actors in digital systems, programming is a relevant for most people.

I’m convinced that many people do a great deal of work that is effectively programming: manipulating tools, identifying and recording procedures, collecting information about the environment, performing analysis, and taking action based on collected data. Editing macros, mail filtering systems, and spreadsheets are obvious examples though there are others.

Would teaching these people how programming worked and how they could use programming tools improve their digital existences? Possibly.

Would general productivity improve if more people new how to think about automation and were able to do some of their own programming? Almost certainly.

Would having more casual programmers create additional problems and challenges in technology? Yes. These would be interesting problems to solve as well.

Project Orientation

Sun, 02 Dec 2012 00:00:00 +0000

(or my latest attempt to do things in a more “project oriented way.")

This post is about recent projects, projects that I’m working on, and how my work has changed in recent months.

A couple of weeks ago, I finally posted all of the content that I’ve been working on for the new, revived Cyborg Institute. While the book on systems administration itself had been mostly done for a while, I’d delayed for two reasons:

I wanted to have a couple of other projects completed to demonstrate that the Institute as a project wasn’t just isolated to book-like objects.
I wanted to have some infrastructure in place to be able to sanely publish the Institute site without using some gnarly content management system.¹

The end result is that in addition to the book, I’ve put together a few other projects and documentation. The more exciting thing is that I might do more things like this in the future.

In addition to a lot of day-job work--forthcomming releases and team growth are eating a lot of my time--I’ve been working on a series of wiki pages (and related blog posts,) that address “information debt that happens when organizations don’t put resources and energy into maintaining resources and “knoweldge.” Expect to hear more on this topic.

The truth is that I really like working on bigger projects. Writing blog posts and participating in online conversations has been very rewarding to me over the past ~10 years, I feel like I’ve hit a wall: I’ve written ~830,000 words on tychoish.com, and am frustrated that there’s not a lot to show for it:

readership is steady, even increasing, but not inspiring,
I don’t actually want to work as a blogger, and
most importantly the work I’ve done here doesn’t really build to anything more than a half-dozen or so blog posts.

While there are themes throughout all of the posts, the work isn’t very rigorous, and it lacks a certain kind of depth.

So here I am, writing books-like objects things about technology that I hope are and will be useful for both technical and non-technical audiences, as well as compiling the little things that I hack on for other people to improve and benefit fromm, and writing fiction (that I may try and publish conventionally, but I may end up self-publishing using a similar proccess.) The goal is to:
Write things with more rigor, including better citations and research.
Work on projects that address topics more comprehensively.
Produce, document, and maintain scripts and other programs that I write rather than endlessly critique existing tools and approaches. In short, less talking about stuff and more making stuff.

Let’s see how this goes!

All content management systems are gnarly. ↩︎

imenu for Markdown

Sun, 25 Nov 2012 00:00:00 +0000

For a while, I’ve been envious of some of the project and file navigation features in emacs for browsing bigger projects/programs, things like imenu and tags have always seems awesome but given that I spend most of time editing restructured text and markdown files (I’m a technical writer), these tools have been distant and not a part of my day to day work.

It’s not that it would be impossible to write interfaces for imenu or etags, for the formats I use regularly, but more that I’ve never gotten around to it until now.

We’re still a ways away on the question of etags, but it turns out that when I wasn’t looking rst mode got imenu support, and with the following little bit of elisp you can get imenu for markdown.

(setq markdown-imenu-generic-expression
   '(("title"  "^\\(.*\\)[\n]=+$" 1)
     ("h2-"    "^\\(.*\\)[\n]-+$" 1)
     ("h1"   "^# \\(.*\\)$" 1)
     ("h2"   "^## \\(.*\\)$" 1)
     ("h3"   "^### \\(.*\\)$" 1)
     ("h4"   "^#### \\(.*\\)$" 1)
     ("h5"   "^##### \\(.*\\)$" 1)
     ("h6"   "^###### \\(.*\\)$" 1)
     ("fn"   "^\\[\\^\\(.*\\)\\]" 1)
))

(add-hook 'markdown-mode-hook
   (lambda ()
      (setq imenu-generic-expression markdown-imenu-generic-expression)))

Pretty awesome! I hope it helps you make awesome things.

Practical Branch Layout

Sat, 17 Nov 2012 00:00:00 +0000

I’ve recently gone from being someone who uses git entirely “on my own,” to being someone who uses git with a lot of other people at once. I’ve also had to introduce git to the uninitiated a few times. These have both been notable challenges.

Git is hard, particularly in these contexts: not only are there many concepts to learn, but there’s no single proscribed workflow and a multitude of workflow possibilities. Which is great from a philosophy perspective, and arguably good from a “useful/robust tool perspective,” but horrible from a “best practices” perspective. Hell, it’s horrible for a “consistent” and “sane” practices perspective.

There are a lot of solutions to the “finding a sane practice,” when working with git and large teams. Patching or reformulating the interface is a common strategy. Legit is a great example of this, but I don’t think it’s enough, because the problem is really one of bad and unclear defaults. For instance:

The “master” branch, in larger distributed systems is highly confusing. If you have multiple remotes (which is common), every remote has its own master branch, which all (necessarily) refer to different possible points in a repository’s history.
The names of a local branch do not necessarily refer to the names of the remote branch in any specific repository. The decoupling of local and remote branches, makes sense from a design perspective, but it’s difficult to retain this mapping in your head, and it’s also difficult to talk about branch configurations because your “view of the universe,” doesn’t often coincide with anyone else’s?

Here are some ideas:

Have two modes of operation: a maintainer’s mode that resembles current day git with a few basic tweaks described in later options, and a contributors mode, that is designed with makes the following assumptions:
- The “mainline” of the project’s development will occur in a branch to which this user only has read-only access.
- Most of this user’s work will happen in isolated topic branches.
Branch naming enforcement:
- All branches will be uniquely named, relative to a username and/or hostname. This will be transparent (largely) to the history, but will make all conversations about branches less awkward. This is basically how branches work now, with [remote]/[branch], except that all local branches need self/[branch], and the software should make this more transparent.
- Remote branches will implicitly have local tracking branches with identical names. You could commit to any of the local tracking branches, and pull will have the ability to convert your changes to a self/[branch] if needed.
- All local branches, if published, will map to a single remote branch. One remote, will be the user’s default “publication target,” for branches.
This is basically what the origin remote does today, so again, this isn’t a major change.
When you run git clone, this remote repository should be the upstream repository, not the origin.

Use the origin remote, which should be the default “place to publish my work,” and would be configured separately.
Minor tweaks.
- Map local branches directly to remote branches.
- Be able to specify a remote branch as a “mirror” of another branch.
- Make cherry-picked commits identifiable by their original commit-id internally. The goal is to push people to cherry-pick commits as much as possible to construct histories without needing to rebase.¹
- Have sub-module support automatically configured, without fussing.
- Have better functions to cleaning up branch cruft. Particularly on remotes.
- Have some sort of configurable “published pointer,” that users can use as a safe block against rebases before a given point.
The goals here are to:
- Make working with git all about branches and iteration rather than a sequence of commits.
- Provide good tools to prevent people from rebasing commits, which is always confusing and rarely actually required.
- Make branch names as canonical as possible. The fact that there can be many possible names for the same thing is awful.

Who’s with me? We can sort out the details, if you want in comments.

To make this plausible, github needs to allow cherry-picked commits to close a pull request. ↩︎

Hypertextuality

Mon, 08 Oct 2012 00:00:00 +0000

I recently took some of my writing time to create a makefile (Novel Makefile) to manage work I hope to be doing on a new novel project. I’ve started outlining and researching the story in earnest after having spent the past few couple of years talking about it, and I think writing will commence soon. In another post I’d like to write up some thoughts on the tooling and technology of writing non-technical/non-manual long-form.

This post, drawing from the spending some time buried deep in production is about the state of (conceptually) longer form work in digital mediums. Or, at least a brief commentary on same.

The tools that I use to write technical materials do all sorts of cool things, like:

provide instant cross referencing,
generate great indexes, and
automatically generate and link glossaries.

This is not particularly unusual, and in fact Sphinx is somewhat under-featured relative to other documentation generation systems like DocBook.¹

And yet people publish ebooks that virtually identical to paper books. Ebooks seem to say “*this electronic book is the best facsimile of a paper book that we can imagine right now,*” while totally ignoring anything more that a *hyper*text rightfully might be.

I enjoy reading ebooks, just as I have enjoyed reading paperbooks, but mostly because ebooks basically are paperbooks. I’ve written posts in the past challenging myself,and fiction writers in general, to actually do hypertext rather than recapitulating prior modalities in digital form.

At various points I’ve thought that wikis might be a good model of how to do hypertext, because the form is structurally novel. Any more, I don’t think that this is the case: wikis are unstructured and chaotic, and I’ve come to believe that the secret to hypertext is structure. There are so many possibilities in hypertext, and I think much experimentation in hypertext has attempted to address the chaos of this experience. This does serve to highlight the extent to which “the future is here,” but it obscures the fact that structure makes narratives understandable. Think about how much great, new, innovative (and successful!) fiction in the past decade (or so) is not structurally experimental or chaotic. (Answer: there’s a lot of it.)

The not-so-secret of hypertext, is (I suspect,) tooling: without really good tools the mechanics of producing a complex, interactive textual experience² is difficult for a single writer, or even a small group of writers. Most tools that manage the publication and writing of text are not suited to helping the production of large-multi-page and mutli-component texts. One potential glimmer of hope is that tools for developing programs (IDEs, build systems, compilers, templating systems, introspection tools, pattern matching, etc.) are well developed and could modified for use in text production.

The second non-so-secret of hypertext is probably that hypertext is an evolution of text production and consumption, not a revolution. Which only seems reasonable. We have the technology now to produce really cool text product. While tooling needs to get better, the literature needs to do some catching up.

Lets start making things!

It’s not that Sphinx is “bad,” but it’s clearly designed for a specific kind of documentation project, and if you stray too far outside of those bounds, or need formats that aren’t quite supported, then you end up without a lot of recourse. Having said that, the “normal,” well supported and most projects--documentation or otherwise--will only very rarely hit upon an actual limitation of Sphinx itself. ↩︎
To be clear, I’m partial to the argument that today’s computer games, particularly role-playing games, are the things that the futurists of the 1960s and 70s (e.g. Theodor Holm Nelson) called “hypertext.” ↩︎

Denormalize Access Control

Mon, 24 Sep 2012 00:00:00 +0000

Access control is both immensely useful and incredibly broken.

Access control, or the ability to constrain access to data and programs in a shared system is the only way that we, as users of shared systems, can maintain our identities, personal security, and privacy. Shared systems include: databases, file servers, social networking sites, virtualized computing systems, vendor accounts, control panels, management tools, and so forth all need robust, flexible, granular, and scalable access control tools.

Contemporary access control tools--access control lists (ACL,) and access control groups--indeed the entire conceptual framework for managing access to data and resources, don’t work. From a theoretical practice, ACLs that express a relationship between users or groups of users and data or resources, represent a parsimonious solution to the “access control problem:” if properly deployed only those with access grants will have access to a given resource.

In practice these these kinds of relationships do not work. Typically relationships between data and users is rich and complex and different users need to be able to do different things with different resources. Some users need “read only” access, others need partial read access, some need read and write access but only to a subset of a resource. While ACL systems can impose these kinds of restrictions, the access control abscration doesn’t match the data abstraction or the real-world relationships that it supposedly reflects.

Compounding this problem are two important factors:

Access control needs change over time in response to social and cultural shifts among the users and providers of these resources.
There are too many pieces of information or resources in any potential shared system to allocate access on a per-object or per-resource basis, and the volume of objects and resources is only increasing.

Often many objects or resources have the same or similar access control patterns, which leads to the “group” abstraction. Groups make it possible to describe a specific access control pattern that apply to a number of objects, and connect this pattern with specific resources.

Conceptual deficiencies:

There’s a volume problem. Access control data represents a many-to-many-to-many relationship. There are many different users and (nested) groups, many different kinds of access controls that systems can grant, and many different (nested) resources. This would be unmanageably complex without the possibility for nesting, but nesting means that the relationships between resources and between groups and users are also important. With the possibility for nesting access control is impossible.
ACLs and group-based access control don’t account for the fact that access must be constantly evolving, and current systems don’t contain support for ongoing maintenance. (we need background threads that go through and validate access control consistency.) Also all access control grants must have some capacity for automatic expiration.
Access control requirements and possibilities shift as data becomes more or less structured, and as data use patterns change. The same conceptual framework that works well for access control in the context of a the data stored in a relational database, doesn’t work so when the data in question is a word processing document, an email folder, or a spread sheet.

The fewer people that need access to a single piece of data, the easier the access control system can be. While this seems self evident, it also means that access control systems are difficult to test in the really large complex systems in which they’re used.
Group-based access control systems, in effect, normalize data about access control, in an effort to speed up data access times. While this performance is welcome, in most cases granting access via groups leads to an overly liberal distribution of access control rights. At once, its too difficult to understand “who has access to what” and too easy to add people to groups that give them more access than they need.

So the solution:

Denormalize all access control data,
don’t grant access to groups, and
forbid inheritance.

This is totally counter to the state of the art. In most ways, normalized access control data, with role/group-based access control, and complex inheritance are the gold standard. Why would it work?

If you have a piece of data, you will always be able to determine who has access to data, without needing to do another look-up.
If you can deactivate credentials, then a background process can go through and remove access without causing a large security problem. (For partial removes, you would freeze an account, let the background process modify access control and then unfreeze the account.)

The down side is that, potentially, in a large system, it may take a rather long time for access grants to propagate to users. Locking user accounts makes the system secure/viable, but doesn’t make the process any more quick.

As an added bonus, these processes could probably be independent and wouldn’t require any sort of shared state or lock, which means many such operation could run in parallel, and they could stop and restart at will.
The inheritance option should be fuzzy. Some sort of “bucket-based” access control should be possible, if there’s a lot of data with the same access control rules and users.

Once things get more complex, buckets are the wrong metaphor, you should use granular controls everywhere.

Problems/Conclusion:

Denormalization might fix the problems with ACLs and permissions systems, but it doesn’t fix the problems with distributed identity management.

As a counterpoint, this seems like a cryptography management problem.
Storing access control information with data means that it’s difficult to take a user and return a list of what these credentials have access to.

In truth, centralized ACL systems are subject to this flaw as well.
A huge part of the problem with centralized ACL derives from nesting, and the fact that we tend to model/organize data in tree-like structures, that often run counter to the organization of access control rights. As a result access control tools must be arbitrary.

Taxonomic Failure

Mon, 27 Aug 2012 00:00:00 +0000

I tell people that I’m a professional writer, but this is a bit misleading, because what I really do is figure out how to organize information so that it’s useful and usable. Anyone, with sufficient training and practice, can figure out how to convey simple facts in plain language, but figuring out how to organize simple facts in plain language into a coherent text is the more important part of my job and work.

This post and the “Information Debt” wiki page, begin to address some of these the problem of information resource maintenance, organization, and institutional practices with regards to information and knowledge resources.

Organization is hard. Really hard. One of the challenges for digital resources is that they lack all of the conventions of /technical-writing/books, which would seem to be freeing: you get more space and you get the opportunity to do really flexible categorization and organization things.

Great right?

Right.

Really flexible and powerful taxonomic systems, like tagging systems have a number of problems when applied to large information resources:

the relationship between the “scope” of the tag, and the specificity of the tag matters a lot. Too much. Problems arise when:
tags are really specific, pages include a number of pieces of information, and tags can only map to pages.
tags are general and the resources all address similar or related topics.
the size of the tag “buckets” matters as well. If there are too many items with a tag, users will find not the tag for answering their questions.
if your users or applications have added additional functionality using tags, tags begin to break as a useful taxonomic system. For example, if your system attaches actions to specific tags (i.e. send email alerts when content with a specific tag,) or if you use a regular notation to simulate a hierarchy, then editors begin adding content to tags, not for taxonomic reasons, but for workflow reasons or to trigger the system.

The resulting organization isn’t useful from a textual perspective.

If you have to have multiple tagging systems or namespaces.

Using namespaces is powerful, and helps prevent collisions. At the same Sat Aug 16 10:50:00 2014, if your taxonomic system has collisions, this points to a larger problem.
If the taxonomy ever has more than one term for a conceptual facet, then the tagging system is broken.

These problems tend to exacerbate as:

the resource ages.
the number of contributors grow.

There’s this core paradox in tagging systems: To tag content effectively, you need a fix list of potential tags before you begin tagging content, and you need to be very familiar with the corpus of tagged contentbefore* beginning to tag content.*

And there’s not much you can do to avoid it. To further complicate the problem, it’s essentially impossible to “redo” a taxonomic system for sufficiently large resources given the time requirements for reclassification and the fact that classification systems and processes are difficult to automate.

The prevalence of tagging systems and the promises of easy, quick taxonomic organization are hard to avoid and counteract. As part of the fight against information debt it’s important to draw attention to the failure of broken taxonomy systems. We need, as technical writers, information custodians, and “knowledge workers,” to develop approaches to organization that are easy to implement and less likely to lead to huge amounts of information debt.

Work Logging

Sun, 19 Aug 2012 00:00:00 +0000

Joey Hess' blog of his work on git-annex-assitant has been a great inspiration to me. Basically, Joey had a very successful Kickstarter campaign to fund his work on a very cool tool of his that he wants to expand to be a kind of drop-box replacement. As part of the project, he’s been blogging nearly every day about the work, his progress, and the problems he’s faced.

I really admire this kind of note taking, and think it’s really interesting to see how people progress on cool projects. More than that, I think it’s a cool way to make yourself accountable to the world, and help ensure things get done.

By the same token, when your project work is writing, increasing daily workload by writing notes/posts rather than actually doing work is a problem, on the other hand, given the right kind of templates, and a good end of day habit, it might be easy and a good habit.

Anyone else interested in this? Thoughts on using rhizome versus some other area of tychoish.com? So many choices!

In Favor of PDF

Fri, 17 Aug 2012 00:00:00 +0000

This is really a short rant, and should come as a surprise to no one.

I hate DOC files, and RTF files, to say nothing of ODF, DOCX, and their ilk because they have two necessarily conflicting properties:

1. They’re oriented at producing documents on paper. Which is crazy. Paper is an output, but it’s not the only output in common use, so it’s nuts that generic document representation formats would be so tightly coupled with paper.

2. The rendering of the content is editor specific, particularly with regards to display options. If I compile a document and send it to you, I have no guarantee whatsoever about the presentation or display of the document on your system, particularly if I’m not certain that your system is similarly configured. Particularly with respect to fonts, page breaks, etc.

This is particularly idiotic with respect to 1.

It’s not that PDF is great, or especially usable, but it’s consistent and behaves as expected. Furthermore, it does a good job of appropriately expressing the limitations of paper.

So use PDF and accept no substitutions.

Announcement Cycle

Tue, 14 Aug 2012 00:00:00 +0000

Things you should know about, dear readers:

I published a book and released a mess under the banner of the new Cyborg Institute. These projects are:

Systems Administration for Cyborgs

This is a high level guide that introduces key systems administration knowledge domains and concepts. Possible subtitles include: “How to think like a systems administrator,” or “Just add man pages.”

Read it now. Tell your friends. And please, send feedback and comments.
Taskfile

Rather than use org-mode, I’ve taken to using a sort of hacked together makefile that aggregates task items from other textfiles. It works surprisingly well.
stl

The idea behind this script is that it would provide a good way to report activity in a regular way so that I’d be able to look back over a log file and see how much I’d written (i.e. report word counts,) and also do a bunch of other straightforward reporting and logging.

I plan to do a revision of this that’s a bit less complicated and a bit more modular, but it’s a nice proof of concept.
stack

I’ve been meaning to publish this for a long time, but this project contains configuration files for emacs and StumpWM, with other systems to follow.

I’ve made a bunch of wiki pages on Information Debt. Current pages include:
- Books
- Taxonomy
- Wiki
- Document Management
- Hierarchy
I was planning on speaking at the Open Help Conference, on August 10-15th,

2012. But then, United Airlines canceled my flight and I spent the day in Terminal C of Newark Airport and ultimately couldn’t make it for the conference. Which is a huge shame. My talk was going to be is about involving developers in documentation processes, about maintaining documentation, and about working with documentation “as code.”

The site I made for the talk is at: http://tychoish.net/oh-2012

And the last draft of the slides are at: http://tychoish.net/oh-2012/slides/

(I was speaking on behalf of `10gen <http://10gen.com/>`_, my employer, `MongoDB <http://docs.mongodb.org/>`_ `documentation project <http://docs.mongodb.org/>`_ that I work on by day (and sometimes night.))

Free the Work

Sat, 11 Aug 2012 00:00:00 +0000

There’s a folk song with the line “let the toast go free,” to which an assembled crowd of singers will often stand up and cry “free the toast!” in response.

I’ve been spending the morning going through email, reading feeds, tweaking some of the issues raised in the site improvements page, and trying to rock getting things done. In the mean time I have a few links to share.

The Everyday Lives Of Video Game Developers
The Price of Free
This article, and the one before, are both open on my “things to read list. Imagine that’s what I’m doing this afternoon.
Branchable - ikiwiki (like the wiki that powers this site) hosting service, based on publicly available tools. I kinda want to set up a branchable instance.
Pybtex - a python/yaml drop in replacement for BibTeX, which is really cool, and I suspect that this will probably be my next citation management tool. I’ve been looking for one for a long time.
Kindle Feeder - a tool to crawl your feeds and move them to your kindle. I’m currently sifting through my feeds and seeing what might make sense to read on the kindle. Might as well, and the automatic delivery makes a lot of sense.
I played with ehsell for the first time today in response to this post, and I have to say that I really like it, and it’s much more intuitive and usable than I thought it’d be. I like the idea of being able to write shell functions in emacs-lisp. As an aside the post is, I think, an example of really great casual technical writing.

Coding Pedagogy

Fri, 10 Aug 2012 00:00:00 +0000

There are two parts to this post: first, the relationship or non-relationship between the ability to write code and technical literacy; and second, the pedagogical methods for teaching people how to program/code.

In some ways, I’ve been writing about this and related topics for quite a while: see /posts/objective-whatsis for an earlier iteration in this train of thought.

Programming and Technical Literacy

Programmers and other technical folks talk a lot about teaching young people to code as the central part of any young technical person’s education and basic computer literacy. Often this grows out of nostalgia for their own experience learning to program, but there are other factors at play.¹

In some cases, they even start or point to projects like Codecademy. Which are, in truth, really cool ideas, but I think that effectively equating the ability to write code with technical literacy is fraught:

There are many different kinds of technical literacy and writing code is really such a small part. Sure code gives us a reasonable way to talk about things like design and architecture, but actually writing code is such a small part of developing technology.
Writing code isn’t that important, really. In a lot of ways, code is just an implementation detail. Important as a way of describing some concepts pretty quickly, important because it’s impossible to iterate on ideas without something concrete to point to, but the implementation isn’t nearly as important as the behavior or the interface.
For the last ~40 years, code has been the way that people design behavior and specify interfaces for software. While there are a lot of reasons why this predominantly takes the form of code, there’s not particular reason that we can’t express logic and describe interfaces using other modalities.

There are many people who are very technically literate and productive who don’t write code, and I think that defining literacy as being able to write code, is somewhat short sighted. Also, there is another group of people who are actually programmers who don’t think of the things they do as “programming,” like people who do crazy things with spreadsheets, most librarians, among others. These non-coding programmers may shy away from programming or are mostly interested in the output of the program they write and less interested in the programming itself.

This is a huge problem. I hope that this /posts/computer-literacy-project that I’ve been planning will start to address some of these issues, but there’s even more work to do.

How to Teach People to Code

(This section of the post derives from and summaries the “How to Teach People to Program” wiki page.)

Most of the way that programming books and courses teach programming are frustrating and somewhat dire, for a few reasons:

Most examples in programming books are dumb.
Basic computer science/engineering knowledge is fundamental to the way that accomplished programmers think about programming but aren’t always required to teach people how to program.
Syntax isn’t that important, but you can’t ignore it either.
Slow reveals are really frustrating.
The kinds of code that you write when learning to programming bear little resemblance to the actual work that programmers do.

The solutions to these problems are complex and there are many possible solutions. As a starting point:

Separate the way you present core concepts (i.e. data structures, typing, functions, classes, etc.) from actual code examples and from actual explanations of the syntax.

Interlink/cross reference everything, but if you give people the tools to answer their own questions they’ll learn what they actually need to know, and you can then do a better job of explaining the syntax, basic concepts, and practical examples.
Provide longer examples that aren’t contrived.

Examples don’t need to start from first principals, and don’t need to be entirely self contained. Programming work rarely starts from first principals (relative,) and is rarely actually self contained. It’s foolish, then to use these sorts of pedagogical tools.

Thoughts?

In addition there’s a related fear that many people who don’t have experience with the technology of the 1980s and 1990s won’t have the required technological skills to innovate in another 10 or 20 years. ↩︎

Winning Your Todo List

Fri, 10 Aug 2012 00:00:00 +0000

I kind of want to title this piece “the remains of the day,” but I’ll spare you.

This week I found myself doing something sort of different with my task list. Since I use org-mode to manage my todo list, my explanation will include a bit of “introduction to org-mode,” but I think this practice may be generally applicable regardless of the software you use.

In org-mode you can take elements of an outline--any outline--and turn it into a “todo” item, and using the agenda feature, generate views of these tasks. This is great because you can do planning and brainstorming in a manner that makes sense for you, and when you’re ready to start working, the list that you work from is organized in way that’s conducive to doing things. It’s a great system.

When you create a task, org-mode provides the ability to schedule it for a particular day or set a deadline. You can generate agenda views organized by day. This is how I work, most of the time. In the morning (on the train) I open the agenda and I see about five things per day, and I start working. The key to success, for any situation, but particularly fragmented situations like mine, is figuring out how to structure your projects and tasks such that there’s always a task that’s “small enough” no matter how short your free time is.

This works pretty well, except sometimes, you run out of free time, something takes longer than you’d like, or you reprioritize. The system of scheduling tasks breaks. Solution, at the end of your work day (or evening,) spend a moment or two going though the remains of the day’s tasks and figuring out:

If you couldn’t get to a task was it because you simply didn’t have time, or because there was an unforeseen dependency.
Which tasks can be rescheduled for another day (and what days would be best for this.)
What no longer needs to be done?

At the end of the process there should be nothing left on your todo list. Now you may be tempted to engage in a little “productivity theater,” but the goal is less to “get everything done,” and more to check in with yourself more regularly, and make sure everything is on track. Also, I’m convinced being faced with scads of overdue tasks, particularly tasks that have grown stale is hardly a wining strategy either.

Writing about Science Fiction Writing

Fri, 10 Aug 2012 00:00:00 +0000

With the last post /posts/writing-about-technical-writing about the kind of writing I do every day for work (and work related) project, I thought it would be fun to muse, briefly about the kind of writing, I do for me.

That sounded pretentious. But it’s less pretentious, I think, than saying I write fiction for art.

Writing fiction, genre fiction at that, for me, is about talking to people directly about the way they see their worlds, about the way that we construct theories of reality, about complex systems, and maybe about all of the little thoughts and ideas that sound too foolish or too simple to justify saying plainly but are nevertheless important to say.

Fiction is a luxury, and perhaps there is why it is so crucial.

Saying, “I want to have a career as a fiction writer,” seems not very grounded in reality, and potentially even more pretentious. So here I am. I write professionally (documentation,) and I write science fiction because it’s the best way to say things that I think need saying. I’ll probably even get around to publishing fiction “professionally,” if I can, in the next couple of years, but I don’t think I’ll ever really give up the professional writing either. Writing fiction, I think, is a piece of a much larger puzzle. So there.

Two fiction writing related updates:

1. I wrote fiction Tuesday morning on the train. I’ve been working on other things for a few weeks/months, and hadn’t been able to squeeze any fiction writing in for a long time. It was good to get back to it, good to realize that I’d finished a section that had been nagging me, and could start fresh, and good to remember than I’m well into the final little stretch of this book. Just need to finish now. I’ll probably be posting more about this soon.

2. I’m really amazed by how much my “work writing” has effected and improved my fiction writing. I don’t think I was ever, exactly, a bad writer, but I’ve certainly become a better and more efficient writer, and despite the fact that fiction writing is probably the thing that has “slipped” the most for me in the last couple of years, it feels good to know that I’m still getting better at it.

Writing about Technical Writing

Fri, 10 Aug 2012 00:00:00 +0000

I posted something new to Critical Futures today. It’s a piece about technical writing, as part of the latest installment in which has become something of a series on technical writing.

I’ve been playing, and watching this new tool, called dexy, which is really cool in its own right, but is also really cool insofar as it says: technical writing, documentation, is important and deserves great documentation tooling.

For those of you who don’t write documentation and who aren’t very involved in the mechanics of publishing lots of text to websites probably don’t find this very exciting. But it is.

In any case, in recognition of the fact that this is now a theme in my blogging and writing, I’ve created a manually generated archive for this “series” of posts. See: Posts about technical writing at Critical Futures.

There are some other modifications that I’ve made, and am making to Critical Futures. Namely: the archives page now displays a full list of all the content on the site. Hopefully this will help Google find older posts, and make them more accessible. Despite how I feel about these old posts, I think it’s good to prevent them from falling into oblivion too soon.

More changes to come soon, of course. Also real content? Hopefully!

Now: off to sing sacred harp and eat dinner with friends!

Documentation Rhetoric

Fri, 03 Aug 2012 00:00:00 +0000

Other than shortening sentences, inserting lists, and using document structure, there are a couple of “easy edits” that I make to most documents that other send to me for review:

Remove all first person, both singular and plural.

2. Remove all passive sentences, typically by making the sentences more imperative.

In practice these changes are often related.

Expunge the First Person

Removing the first person is important less because it’s “more formal” to avoid the first person and more because it’s always unclear in documentation: Who are “we,” and who is “I”? Should I read “I” as “me” or as the author of the documentation? What if my experiences and environment isn’t like “ours?” While we can resolve these confusion points pretty quickly it gives users another set of information that they must track. And given that technical subjects can be difficult without confusing language, there’s no reason to make things more confusing.

People tend to think that this makes their documentation “friendlier,” “personable,” or “intimate.” People used to interacting directly with users (i.e. people doing user support) are particularly susceptible to first person problems. In support cases, that little bit of personal touch is incredibly valuable and goes a long way toward making people feel comfortable.

Those people are wrong. Don’t do it. Speak simply. Write about the product and the processes you’re documenting, not yourself. Convey facts, not opinions. Provide context, not perspective. If you’re writing the official documentation for a product, your perspective is obvious to readers; if you’re not writing the official documentation, that’s also apparent and probably not your job to disclaim.

Use Good Verbs

Passive sentences and weak verbs are a huge problem. Huge. People with science and engineering back rounds seem to prefer passive sentences because they think that passive sentences convey objectivity, and that this objectivity is desirable.

Passive sentences do convey a sense of objectivity, and there are some cases where there’s no way to avoid describing a property of a thing except passively. That doesn’t make the passive voice generally acceptable. Related to the reason above, passive voice tends to provide a level of “syntatic indirection,” and means that complicated sentences become unnecessarily difficult to comprehend.

In documentation, unlike some other forms, it’s possible (and desirable!) to use imperative verbs, which provides some relief. One of the main projects of documentation is to inculcate “best practices” (i.e. values and conventions,) in users. Imperative verbs are great for this purpose.

In short: Do it!

ThinkPad x220 Review

Thu, 19 Jul 2012 00:00:00 +0000

My Decision

Throughout this spring I’ve been eagerly waiting for the announcement and arrival of the new X-series laptops from Lenovo. I’ve been incredibly happy with every Thinkpad I’ve ever had, and while my existing laptop--a very swell T510--has been great, it was time:

I needed a system with a bit more power. The power of my existing system was being to frustrate me. Things took too long to compile, I was having some annoying networking processing issues, and to make matters worse…
The thing was huge. I think 15 inch laptops are a great size for doing actual work, and I’m not getting rid of this one, but it’s not the kind of thing I want to lug on my back. Which I was doing a lot.
I needed more redundancy. Most of my work in the world--writing, hacking, communicating--happens with a computer. While my data is backed up (never well enough, of course, but it’s there,) I worry more about the case where I’m stranded for a period of time without a working system.

This facilitates not only piece of mind, but also makes it possible to do things like: upgrade the T510 from 32 to 64 bits. (Don’t ask.)
In the long run, the older laptop might need to go to R. who’s personal system bit the dust a few months ago.

What Happened

But, when the new x230s came out and I found myself unimpressed. The revision got a different keyboard and I adore the old keyboard. To make matters worse the screen on the new model wasn’t any better than the one on the old: the pixel density is somewhat crappy.

In light of this, and mostly for the older keyboard, I decided to buy the older model. In short: it’s great.

I bought the RAM and hard drive aftermarket, and replaced them before booting the first time. Having 16 gigs of RAM is pretty much amazing, and I’m sold on the notion that SSDs are now a must for most common personal computing work.

Incidentally I discovered that this computer is about the same weight as the 13 inch Macbook Air (and I have the larger battery), for those of you keeping score at home. And way beefier. Thicker obviously, but still…

Point by Point

Pros:

The keyboard is the same great Thinkpad keyboard we’ve always had. I’m sure eventually I’ll give in and learn to enjoy the new keyboard, but for now, I’m going to stick with the old.
It’s way fast. Because, the speed of my old computer defined “the speed of computers,” in my mind, it was kind of nifty to learn that computers had actually gotten faster.
It’s way small. Turns out, if I’m lugging a sub-3 pound laptop around, I can totally use my awesome shoulder bag. I also don’t feel like my wrist is going to give out if I need to walk 30 feet holding the laptop in one hand.

Cons:

The screen could be so much better than it is, and there’s really no excuse. It’s not enough of a deal breaker for me, but…
That’s really it. I think 12 inch wide screen laptops don’t have quite enough wrist-rest area on them, but that’s really an unavoidable problem: if you have a wide secreen (and thus a full keyboard,) the wrist area is short and narrow. If you have a more square screen and a squished keyboard, then you have enough wrist area. One adjusts.

Emacs Thoughts + Some Lisp

Fri, 13 Jul 2012 00:00:00 +0000

In no particular order:

Org Mode Guilt and a Lisp Function

I have some guilt about having mostly forsaken org-mode,¹ in particular because I was watching Sacha Chua’s chat with John Wiegley, and I think both are such nifty hackers, and have done so many things that are pretty darn nifty.

I liked what I heard about johnw’s org mode setup so much that I might give it a try again. But in the mean time, I wanted to make my “recompile my tasklist function” to be a bit more clean. The result is follows:

(defun tychoish-todo-compile ()
   (interactive)
   (if (get-buffer "*todo-compile*")
       (progn
          (switch-to-buffer-other-window (get-buffer "*todo-compile*"))
          (recompile))
       (progn
          (compile "make -j -k -C ~/wiki")
          (switch-to-buffer-other-window "*compilation*")
          (rename-buffer "*todo-compile*")))
       (revbufs))

Notables:

This is the first time I’ve used progn which is somewhat embarrassing, but it’s a great thing to have in the toolkit now. Link: progn
I hadn’t realized until now that there wasn’t an else-if form in emacs lisp. Weird, but it makes sense.
Compilation Mode is pretty much my current favorite thing in emacs.
revbufs is this amazing thing that reverts buffers if there aren’t local modifications, and also reports to you if a buffer has changed outside of emacs and there are local modifications. So basically “does everything you want without destroying anything and then tells you what you need to do manually.” Smart. Simple. Perfect.

I might need to “macro-ize” this, as I have a lot of little compile processes for which I’d like to be able to trigger/maintain unique compile buffers. That’s a project for another day.

Emacs Thoughts

I’m even thinking about putting together a post about how, although I’m a diehard emacs user, and I’ve spent a fair bit of time learning how to do really great things with emacs, there are a lot of vim-ish things in my workflow:

I read email with mutt and I’ve tried to get into GNUS, and I try it again every now and then, but I always find it so unbelievably gnarly. At least the transition. Same with Notmuch, which I like a lot more (in theory,) but I find the fact that Notmuch and mutt have this fundamental misunderstanding about what constitutes a “read” email, to be tragic.
I use a crazy ikiwiki + deft + makefile setup for task tracking. As (obliquely) referenced above.

I might give org another shot, and I’ve been looking at task warrior, but the sad truth is that what I have works incredibly well for in most cases, and switching is hard.
I tend jump to a shell window to do version control and other things, even though I’m familiar with magit and dired, my use of these tools is somewhat spotty.

It’s not that I think org-mode sucks, or anything. Far from it, but how I was using org-mode was fundamentally not working for me. I’m thinking about giving it a try again, but we’ll see. ↩︎

Analog Editing

Wed, 04 Jul 2012 00:00:00 +0000

After doing the first pass of editing on my technical book, “Systems Administration for Cyborgs” on a screen and feeling utterly buried by it, (See: /posts/the-editing-hole,) and I’m considering different approaches for the next book. Specifically, for this novel I have, I’m thinking about getting the novel printed somehow and then editing it “analog style.”

I’d love to hear feedback from anyone who has done this, particularly recently. Particularly from people who are very digitally savy. Here’s my pro/con list:

Pros:

It might be nice to have a different “editing context,” to help me keep focus on the project without the distractions of the internet and current writing projects.
Having marked-up pages gives me an actual marker of progress, rather than a list of commits or diffs.
It might be nice to get some practice writing longhand again. It’s embarrassing when someone hands me a pen and I’ve basically forgotten how to use it.
It gives me a start to a collection of paper ephemera that I can burden some archivist with at some point.

Cons:

Context switching, from a computer, to paper, to a tablet, or whatever is annoying and eats time/focus.
Paper would force a more linear editing process, which may get me to focus on the story and characters more closely at the possible expense of seeing “broken sentences,” and other things that may be distracting to the next readers/editors/etc.
If I edit on paper, I have to go through and apply those changes to the actual text. Which adds a step, and probably a number of additional weeks to the editing process.
I’m awful writing things out long hand and I pretty much haven’t written anything long hand in 4 or 5 years.

Reflections

There’s also another little thorny problem: I’m not sure what the future of this book is: The prologue stands alone, and I want to try shopping it around. If I can get that published as a short that might be the hook to getting the rest of the book published.

Initially my plan was to have a friend read it as a podcast, and attempt to publish the prologue as a short, and then try and for-real publish the next book. The podcasting idea, while nice, wouldn’t work out as originally planned (long story.) Besides, it really depended on having someone else to the reading (I have neither the time, technical skills, nor the real ability to do the reading.) Which leaves me without a plan, and the following thoughts about the publication of this text:

I feel like my writing career¹ is in good (enough) shape that my identity as a writer depends on publishing this novel in a particular way.
On the other hand, getting the novel (or parts of it published) would be a great thing, and validates all this time/energy/interest that I have in writing science fiction.
I’m not opposed to self publishing, except that it means more work for me for what is probably less impact (i.e. readers.)
It took me embarrassingly long to write this novel. And knowing how much better my writing has gotten in the last three or four years, means that I’m pretty worried that this is really a cold pile of shit. I recognize that this is probably more reason to get the novel out to first readers, but I also feel like I should do them the favor of at least a little editing.

Printing in the Digital Age

As an aside, this has given me some time to do research on getting things printed, which I’d like to record here and share with you:

(Given a 325-350 page manuscript)

I don’t own a printer, and have no particular interest in owning one, but a good color laser (in the 300-400 USD) or a good black and white (200-300 USD) becomes much more economical if analog editing becomes a thing.

The downside of printing things yourself is that you still need some way to bind things. And there are maintenance and supply costs not factored into the above.
The price to get printing/binding done at Staples is in the 35-40 range. Chances are that these copies are likely to be the highest quality, quickest turn around, and the web interface--though annoying--is probably the most straight forward.
You can use Lulu.com to order prints. The same book costs 15 bucks. It’s also spiral bound (which seems preferable to perfect binding for writing.) I’m not sure what “lulu standard” paper is like quality wise, but I suspect it will be ok. If you’re ok with a fussy online interface, a weird approach to covers (no really, I’d just like transparent covers,) and turn around time, the price seems unbeatable.

Also, Lulu’s cover making interface makes it really hard to get a plain cover that doesn’t look like a joke.

I did some hunting around for local copy shops (I swear it seems like I pass several on my walk to work,) but had difficult finding a shop who would be able to do a very small order, and has the digital setup to accept a PDF for printing via email or a web site.

I have a full time job writing and editing. While its not the same as writing fiction, it is rewarding and economically viable, and I’m working on the kinds of projects that I want to work on. Can’t argue with that. ↩︎

Collar Design

Wed, 27 Jun 2012 00:00:00 +0000

The success or failure of the collar of a sweater determines the success or failure of an entire sweater. This post provides an overview of my basic: “how to knit a collar” system that usually works for me and some thoughts on why collars are so important.

The hard part of collars is that on the whole, collar shaping accounts for five or ten percent of the knitting, but weeks or months worth of work hangs in the balance. The collar affects both the overall style of the sweater and has a great impact on how you will feel about the sweater when you wear it. A collar that doesn’t hang right, or is too narrow or too wide is the worst: the right collar can also make a sweater that is otherwise too light feel just warm enough and more importantly the sweater that’s too heavy not feel oppressively warm.

So how do you knit a collar?

Easy.

The fine print: These instructions assume that you’re knitting the sweater in the round.</small>

For a basic, round, crew neck…

Figure out how long your sweater will be from the top of the shoulder seam to the bottom hem. Typically this is the length measurement in the pattern and you can measure it easily yourself.

Figure out how many stitches around your collar opening needs to be at the very end, but the width of the hole at the top seam. Because you’re shaping the collar, the number of stitches on the collar as you knit it will be slightly higher because of the angles/sloped edges. Typically this is 13-16 inches for most people.

Elizabeth Zimmerman would figure 1/3rd of K, which is a good starting point, but if your sweater has too much ease or if your making a sweater in excess of 40 inches (which isn’t too big,) I think you’ll end up with neck openings that are a little too wide.

Before you start, compute the following values:

Half of the total collar width. This is the total number of stitches that you need to set aside or decrease by the time you get to the shoulder seam on the front and the back.
About half of this number or a quarter of the total (or a bit less, round down here, if need be) to set aside at the base of the neck.
The total number of stitches minus the actual number of stitches set aside at the bottom of the neck, if your math is fuzzy. This is the number of stitches that you have to decrease on either side of the neck. This number must be even.
The distance between the bottom of the neck and the top of the shoulder. This is almost always within a half an inch of 3 inches. Also record the number of rows.

With these numbers in hand, do the following.

Three (3) inches before this length, begin the collar shaping. (Change depth as needed.)

Set aside stitches at the bottom of the neck.

Decrease on either side of the neck opening (possibly using a steek,) every row until you have decreased half of the number of stitches that you need to decrease. Typically this should take about an inch and a half of knitting, or half of the total collar depth to accomplish.

After you’ve knitted half of the total collar depth, figure out how many stitches you’ve decreased at this point. Set this number of stitches aside in the middle of the back of the sweater.

Meanwhile: Create a new steak or begin decreasing at the back of the neck at the same rate as you decrease at the front.

Now decrease at half the rate (e.g. every other row) for the remainder of the depth of your collar opening. Bind off. You’ve made a sweater.

If you time it right, and your stitches are not too short and wide, this basically works out to: set aside stitches at the front, decrease every row for an inch and a half, set aside stitches at the back, decrease every other row for an inch and a half. Bind off for shoulders.

Variants:

Knit more on the front so that the shoulder seam is actually at the top of the back, and the front of the sweater extends over the top of the shoulder.
Use shoulder saddles or straps to increase the depth of the neck. This is a perpendicular strip of knitting that starts at the side of the neck and extends across the shoulders and forms the top part of the sleeve.

In this case, subtract half of the width of the saddle/strap from the depth of the collar opening, and twice the width from the opening. Adjust accordingly: typically the best thing to do is figure out how much additional depth you need to decrease, figure out how many rows that will be, and plan to decrease on every row and set aside the remainder. Depending on the depth you may need to “fill in” or shape some of the depth on the back of the neck.
If your shoulders slope downward, consider short rows across one or both of the sides, to make the top of the sweater a bit more conical.
I often make Henly-style sweaters by setting aside a single stitch (and knitting a steek) 3-5 inches bellow the bottom of the collar. This keeps sweaters from becoming too warm.

Lazy Sunday

Sun, 24 Jun 2012 00:00:00 +0000

I’ve had a nice quiet weekend, the first such weekend in quite a while. It’s nice to be able to relax, work on projects without deadlines, and avoid all of the editing that I ought to be doing.

Some notable accomplishments, current projects, and other events in the last few weeks:

If you ever visit tychoish.com in your web browser (as opposed to by way of its aggregation,) you’ll note that the design has changed somewhat.

This is the design that I’ve been using for my personal wiki for months, and so I’m quite used to it, but feedback is welcome.

The design change has inspired a bit of introspection, hence this post, and perhaps some of the posts that will follow. Please bear with me.
Months ago a friend of mine said “some of us might like to know how you’re doing every now and then, and your blog is just stuff about obscure technology.”

Guilty as charged. Recently, I’ve been much more interested in using this blog as a scratch space for projects that I don’t have quite enough time to pursue in appropriate depth.

Having said that, I think (or hoped,) that I’ve calmed down a bit in recent months and years: My career/professional identity seems a bit more stable. I’m doing a better job at focusing big projects, which means some shorter posts and more personal posts may be in order.
I got a new computer last week. I’m working on a post that addresses this in a bit more depth. In short it’s great. So my largely unnecessary justification is:
- A smaller machine, which is better on my back when I am walking around.
- Beefier system, which means I can compile things quicker (and I’ve been doing more of this recently.)

The ability to dedicate the older system to some stay at home things: having a working desk at home, playing music, running some buildbot stuff, and the like.

I’ve realized that even though there are little things that I might like to change about how my computer works, and things that I’d like to setup and get working for the most part, things just work. And that’s really great.

There’s a host of stuff that I’m working on that probably isn’t apparent to the internet:
- My day job. I’m doing awesome documentation things for a neat New York City database software company/open source project that you’ve probably heard of (if you’re into this kind of thing.) It’s rewarding, interesting, and it means I can spend all most of my time day making things.
- I’ve been working on submitting documentation patches to a couple of open source projects: buildbot and MediaGoblin. I need to do more of this work.
- The editing pile. Currently on tap: a mess of blog posts, the prologue of my most recently drafted novel, and some of the last little pieces of the Cyborg Institute launch. Speaking of which that should happen in the next week or so.
Next up? More of the novel and suggestions from frist readers of the systems administration book (see next item.)
Never to be deterred, I’m hard at work on the outline for another novel. The plan is to have something I can start drafting in earnest by the end of the summer. I feel pretty good about the project, although as I was working on an outline last night, I changed the last third of the book. Oops.
The systems administration book. Available via git today, With general release following shortly. All feedback as well as pull or merge requests with comments and suggestions are all welcome. See the following for git repositories
- administration on git.cyborginstitute.net
- administration on github.com
(Both are identical.)

The cyborg institute listserv would be a good place for bug/issue tracking at least for now.
On the topic of editing, I’ve recently discovered the one clause per-line formatting style.

I’ve long attempted to keep lines short to promote cleaner diffs, but in truth, if you end up reflowing paragraphs, the resulting diffs are basically useless. I’ve encountered one-sentence-per-line tactics, which seems like a good idea, except that sentences often exceed 80 characters.

I’m not yet decided on the subject, especially for writing longer sections of text, but it does make editing easier.

Onward and Upward!

Knitting Patterns

Fri, 22 Jun 2012 00:00:00 +0000

In the course of writing a computer program, engineers typically face the same kinds problems again and again. As a result programmers have developed a way of thinking about different kinds of solutions and situations as “patterns,” which provide generalized ways of talking about common problems and strategies.

> See: Portland Pattern Repository for an > example catalog of programming patterns.

When I opened the editor to write this post, I wanted to write something to connect this idea of a “pattern” with a knitting pattern, which I think is (or could be) a very related concept.

Rather thank think about knitting designs as these static instructions for constructing a single kind of garment, I think it might be cool to think of knitting patterns in the sense that programmers use the word: as a set of generalized responses to various states and situations.

This isn’t revolutionary. If you’ve been knitting seriously for more than a couple of years, you probably already think about knitting in this way. At the same time, knitting publishers organize information in other ways: knitting “content” is either design and object centric (how to knit a “thing,") or it’s technique centric (how to make a specific kind of “stitch” or “how to execute a specific kind of operation.")

Patterns are good because they are guides to become more creative and more resourceful knitters. Patterns help us understand how to effectively use resources, to execute the objects that we want to without dithering, and how to fix mistakes in our knitting when we make them. Focusing on knitting “patterns,” is not only important because knitters will find them useful, but because it will advance the state of the craft.

Patterns also help resolve an ongoing “problem” in the knitting world: an online, free culture repository of information about knitting.

In the decade or so, I’ve watched a number of knitters attempt to create online information resources that serve as a free culture repository for knitting. Nothing has been particularly successful, and there’s not a lot of “knitting free culture” out there.

While the lack of knitting free culture is due to a large number of factors, the fact that knitting content has always centered on “objects” and “techniques” is almost certainly a factor:

objects are difficult to design, represent professional designer’s only real way of generating a reputation and income, and it’s difficult to divide the work of writing instructions for creating objects.
there are a few (3-5) really, really good knitting techniques compendiums, and while new techniques emerge every now and then the books from 1938 (Mary Thomas') are as good as any of the more recent examples.

Patterns in this sense might be the best way to build an online knitting resource. Rather than store indexes upon indexes of cast on methods, full patterns for garments, and stitch patterns, we could create an index of knitting patterns, generated and indexed by situation and purpose. Example patterns might include:

Heels for Light Weight socks.
Heels for thick socks.
Crew neck collars.
How to knit sweaters in the round when you don’t want to steek.
How to secure a steek if the yarn isn’t wool.
Knitting socks for high arches.

And so forth…

I’m going to use my (admittedly sporadic) knitting blogging to begin capturing some of these knitting patterns. Then, if there’s interest, we can convert these into a more robust form.

Anyone interested?

In Favor of Fast Builds

Thu, 07 Jun 2012 00:00:00 +0000

This is an entry in my loose series of posts about build systems.

I’ve been thinking recently about why I’ve come to think that build systems are so important, and this post is mostly just me thinking aloud about this issue and related questions.

Making Builds Efficient

Writing a build systems for a project is often relatively trivial, once you capture the process, and figure out the base dependencies, you can write scripts and make files to automate this process. The problem is that the most rudimentary build systems aren’t terribly efficient, for two main reasons:

1. It’s difficult to stumble into a build process that is easy to parallelize, so these rudimentary solutions often depend on a series of step happening in a specific order.

2. It’s easier to write a build system that rebuilds too much rather than too little for subsequent builds. From the perspective of build tool designers, this is the correct behavior; but it means that it takes more work to ensure that you only rebuild what you need to.

As a corollary, you need to test build systems and approaches with significantly large systems, where “rebuilding too much,” can be detectable.

Making a build system efficient isn’t too hard, but it does require some amount of testing and experimentation, and often it centers on having explicit dependencies, so that the build tool (i.e. Make, SCons, Ninja, etc.) can build output files in the correct order and only build when a dependency changes.¹

The Benefits of a Fast Build

Fast builds increase overall personal productivity.

You don’t have to wait for a build to complete, and you’re not tempted to context switch during the build, so you stay focused on your work.
Fast builds increase quality.

If your build system (and to a similar extent, your test system,) run efficiently, it’s possible to detect errors earlier in the development process, which will prevent errors and defects. A tighter feedback loop on the code you write is helpful.
Fast builds democratize the development process.

If builds are easy to run, and require minimal cajoling and intervention, it becomes much more likely that many people

This is obviously most prevalent in open source communities and projects, this is probably true of all development teams.
Fast builds promote freshness.

If the build process is frustrating, then anyone who might run the build will avoid it and run the build less frequently, and on the whole the development effort looses important feedback and data.

Continuous integration systems help with this, but they require significant resources, are clumsy solutions, and above all, CI attempts to solve a slightly different problem.

Optimizing Builds

Steps you can take to optimizing builds:

(Note: I’m by no means an expert in this, so feel free to add or edit these suggestions.)

A large number of smaller jobs that can complete independently of other tools, are easy to run in parallel. If the jobs that create a product take longer and are more difficult to split into components, then the build will be slower, particularly on more powerful hardware.
Incremental builds are a huge win, particularly for larger processes. Most of the reasons why you want “fast builds,” only require fast rebuilds and partial builds, not necessarily the full “clean builds.” While fast initial builds are not unimportant, they account for a small percentage of use.
Manage complexity.

There are a lot of things you can do to make builds smarter, which should theoretically make builds faster.

Examples of this kind of complexity include storing dependency information in a database, or using hashing rather than “mtime” to detect staleness, or integrating the build automation with other parts of the development tool chain, or using a more limited method to specify build processes.

The problem, or the potential problem is that you lose simplicity, and it’s possible that something in this “smarter and more complex” system can break or slow down under certain pressures, or can have enough overhead to render them unproductive optimizations.

It’s too easy to use wild-cards so that the system must rebuild a given output if any of a number of input files change. Some of this is unavoidable, and generally there are more input files than output files, but particularly with builds that have intermediate stages, or more complex relationships between files it’s important to attend to these files. ↩︎

On Build Processes

Tue, 05 Jun 2012 00:00:00 +0000

I’ve found myself writing a fair number of Makefiles in the last few weeks: In part because it was a tool, hell a class of tools, that I didn’t really understand and I’m a big sucker for learning new things, and in part because I had a lot of build process-related tasks to automate. But I think my interest is a bit deeper than that.

Make and related tools provide a good metaphor for thinking about certain kinds of tasks and processes. Build systems are less about making something more efficient (though often it does do that,) and more about making processes reproducible and consistent. In some respects I think it’s appropriate to think of build tools as.

I’ve written here before about the merits of /technical-writing/compilation for documentation, and I think that still holds true: build processes add necessary procedural structure. Indirectly, having formalized build process, also makes it very easy to extend and develop processes as needs change. There’s some up-front work, but it nearly always pays off.

While I want to avoid thinking that everything is a Makefile-shaped nail, I think it’s also probably true that there are a lot of common tasks in general purpose computing that are make shaped: format conversion, extracting and importing data, typesetting (and all sorts of publication related tasks,) archiving, system configuration, etc. Perhaps, more generic build tools need to be part of basic computer literacy. That’s another topic for a much larger discussion.

Finally, I want to raise (or re-raise) the question, that another function of build systems is reduce friction on common tasks and increase the likelihood that tasks will get done, and that people will need less technical background to do fundamentally mundane tasks. Build systems are absolutely essential for producing output from any really complex process because it’s hard to reliably produce builds without them; for less complex processes they’re essential because no one (or fewer people) do those tasks without some kind of support.

Rough thoughts as always.

Computing Literacy Project

Thu, 31 May 2012 00:00:00 +0000

I’m working on the final touches of a treatise on Systems Administration that I’ve mentioned in passing here before. I hope to have this project up on a web-server near you (near me?) in a few months. Because I’ve had a lot of fun working on this project, I decided that it would be cool to do a similar project on another topic dear to my technical interest: Computing Literacy.

Computing literacy? Isn’t that a little *early nineties* for you and or everyone?

Well yes, I suppose, but don’t think that the improvements that have aided the adoption of technology in the last 20 years have necessarily raised the amount of computing literacy. In fact, it would not surprise me if all of the abstraction and friendly user interfaces make it more difficult for users to understand how their computers actually work.

Because their computers play such an important role in everyone’s life and work, I suspect that there are a lot of folks who don’t know how these things work but want to learn. And the truth is that there aren’t a lot of resources around for people who aren’t already tinkerers and hackers.

This is where I fit in: I can write a book-like object that provides useful information in an easy to understand way that technical people can provide their friends and family say things like “what’s a database?” or “what do you mean compiled?” or “what’s a server?” or “what is this file system thing?”

Are you so arrogant as to think that you can add something new to this subject?

I’m not going to develop new technology here, or suddenly make things easier to understand, but I think there’s a niche for computer users between people like me, who download putty on other people’s computers to ssh into their VPS server which hosts a VPN that their laptops are connected to, to ssh into their laptops to kill misbehaving X11 sessions (no really, I just did that,)* and most people who just know how to open MS word and send emails, and browse the web.

And I think that giving folks who are technically creative, intuitive, and curious a way to learn about their computers and technology would be great. Particularly in a way that doesn’t assume too much prior knowledge, or an interest in complicated math.

Ok, so you’re convinced that it’s a good idea, what’s going to be special about this?

First, while the material will not be so complicated or so novel, I think the presentation may be. Additionally, I envision this document as a useful reference for describing and defining basic computing concepts, to support more technical blog posts and articles.

Finally , I think it would be fun to do this book-like object in a more iterative style, relative to the Systems Administration book: I’ll sketch out the basics, put a disclaimer at the top about links breaking and then publish it, and publish changes and make the whole thing accessible by git..

Sound cool to anyone?

So there you have it.

Expect to hear more in the late summer or fall…

The Editing Hole

Tue, 22 May 2012 00:00:00 +0000

I’m stuck in an editing hole, and not only am I not editing the things I need to edit, I’m not getting anything done.

I’m at a point where I have about 25 things on my personal task list, and 16 of them are editing related tasks: edit the article in this file, edit this fiction, edit this documentation, edit these would-be-blog posts, and so forth. It seems like I went on something of a six month writing bender, and while I did a little bit of editing during this period, I have clearly fallen behind.

There are a number of factors:

1. I’ve been making a point of putting editing tasks on the todo list because I want to make sure that I actually finish projects rather than just abandon them. I’ve not been particularly good with follow through in the past few years, so that’s been a big personal improvement project.

The sad part is that my editing queue is probably 10-20 times larger, but I’ve got some projects on the less-actionable back burner.

I’m not a very good editor.

I’m awful at copy (or otherwise) editing my own work, and while I know that I’ve become better at this in the past few years. I still know that it’s not perfect and so it seems sort of futile, which makes it hard to get inspired to do editing.

3. I find writing to be rewarding, and given the choice I will probably always choose to write new stuff. While this is clearly a learned response to the kind of work, this doesn’t make the effect any less real.

I find editing to be really difficult work.

This is probably related to #2, but editing wears me out. I find it difficult to spend long periods of time editing, which makes it difficult to make any really substantial progress on the pile of editing tasks.

As a result, I take a long time to edit things, I’m most effective at editing in short bursts. I often want to break up longer editing tasks with other kinds of work just to keep a clean mind set. After a week or so of this, I have almost everything else done on the list, leaving me with a big pile of editing that looks even bigger for the lack of other things on the list.

So now, I’m trying the following:

1. I’m working on making editing tasks smaller, which will turn into more editing tasks, but it’ll be possible to face editing tasks in units less than 10 or 20 pages.

Make more tasks for other projects.

There are two ways to get stuff done: 1) You can be really focused and work on one project at a time until it’s finished, or 2) you can be working on a lot of projects in parallel and when you start to loose focus, you switch to another kind of project. The idea is that you end up getting more done because you’re being productive more of the time. I subscribe to the second theory.

Here’s hoping it works!

Onward and Upward!

Git Feature Requests

Thu, 17 May 2012 00:00:00 +0000

The ability to mark a branch “diverged,” to prevent (or warn) on attempted merges from master (for example) into a maintenance branch.
The ability to create and track dedicated topic branches, and complementary tooling to encourage rebasing commits in these sorts of branches. We might call them “patch sets” or “sets” rather than “branches.” Also, it might be useful to think about using/displaying these commits, when published, in a different way.
Represent merge commits as hyperlinks to the user, when possible. I think GitHub’s “network graph” and similar visualizations are great for showing how commits and branches interact and relate to each other.

This would probably require some additional or modifies output from “git log”.
Named stashes.
Branched stashes (perhaps this is closer to what I’m thinking about for the request regarding topic branches.)
The ability to checkout “working copies,” of different points/branches currently from a single repository at the same time, using “native” git utilities.

Related, “shelf” functionality is scriptable, but this too needs to be easier and more well supported.

I think legit is a step in the right direction, but it’s weird and probably makes it more difficult to understand what’s happening with git conceptually as opposed to the above features which would provide more appropriate conceptual metaphors for the work that would-be-git-users need.

Limitiations of GitHub Forks

Thu, 10 May 2012 00:00:00 +0000

Assumption:

git is pretty awesome, but it’s conceptually complex. As a result using git demands a preexisting familiarity with git itself or some sort of wrapper to minimize the conceptual overhead.
The collaboration methods (i.e. hosting) provided by git, which are simple by design to allow maximum flexibility, do not provide enough structure to be practically useful. As a result providers like GitHub (and BitBucket and gitorious) offer a valuable service that makes it easier--or even possible--for people to use git.

Caveats:

there are problems with using centralized repository services controlled by third parties, particularly for open source/free software projects.

There are ways that GitHub succeeds an fails in this regard. but this dynamic is too complex to fully investigate within the scope of this post.
If you use GitHub as designed, and the way that most projects use nGitHub, then you have a very specific and particular view of how Git works.

While this isn’t a bad thing, it’s less easy to use git in some more distributed workflows as a result. This isn’t GitHub’s fault so much as it is an artifact of people not really knowing how git itself works.

Assertions:

GitHub’s “fork” model[^fork] disincentives people from working in “topic” branches.
By making it really easy for people to publish their branches, GitHub disincentives the most productive use of the “git rebase” command that leads to clean and clear histories.
There’s no distinction between a “soft fork” where you create a fork for the purpose of submitting a patch (i.e. a “pull request”) and a “hard fork,” where you actually want to break the relationship with the original project.

This is mostly meaningful in context of the other features that GitHub provides, notably the “Network” chart, and the issue tracker. In a soft-fork that I would intend to merge back in, I’d like the issues to “come with,” the repository, or at least connect in some way to the “parent.” For hard forks, it might make sense to leave the old issues behind. The same with the network chart, which is incredibly powerful, but it’s not great at guessing how your repository relates to the rest of its “social network.”

The solution: keep innovating, keep fighting lock-in, and don’t let GitHub dictate how you work.

Making Things Easier

Tue, 08 May 2012 00:00:00 +0000

I spent a lot of time in the past few months thinking about “automation,” as a project to take things that take a long time and require a lot of human intervention into things that just do themselves, and I think this is the wrong approach.

While total automation is an admirable, it’s difficult, both because it requires more complex software to deal with edge cases, but also because it’s hard to iterate into a fully automated solution.

Let’s back up for a moment and talk about automation in general.

Computers are great at automating things. When you figure out how exactly to accomplish something digitally (i.e. polling an information source for an update, transforming data, testing a system or tool,) writing a program to perform this function is a great idea: not only does it reduce the workload on actual people (i.e. you.) I think the difference between people who are “good with computers,” and people who are “great with computers,” is the ability to spot opportunities for these kinds of automations, and potentially implement them..

To my mind the most important reason to automate tasks is to ensure consistency and to make it more likely that tedious tasks get done.

Having said this, rather than develop complete task automations for common functions, the better solution is probably to approach automation on the bottom up: instead of automating a complete process, automate smaller pieces particularly the most repetitive and invariable parts, and then provide a way for people to trigger the (now simplified) task.

The end result, is a system that’s more flexible easier to write, and less prone to failure under weird edge cases. Perhaps this is a manifestation of “worse is better” also.

Thoughts?

Onward and Upward!

Distributed Bug Tracking

Mon, 30 Apr 2012 00:00:00 +0000

The free software/open source/software development world needs a distributed bug tracking story. Because the current one sucks.

::: {.contents} :::

The State of the Art

There are a number of tools written between 2006 and 2010 or so that provide partial or incomplete solutions to the problem. Almost isn’t quite good enough. The “resources” section of this post, contains an overview of the most important (my judgment,) representatives of the current work in the area with a bit of editorializing.

In general these solutions are good starts, and I think they allow us (or me) a good starting point for thinking about what distributed bug tracking could be like. Someday.

Bug tracking needs are diverse, which creates a signifigant design challenge for any system in this space. There are many existing solutions, that everyone hates, and I suspect most would-be developers and innovators in the space would like to avoid opening this can of worms.

Another factor is that, while most people have come to the conclusion that distributed source control tools are the “serious” contemporary tool for managing source code the benefits of distributed bug tracking hasn’t yet propogated in the same way. Many folks have begun to come to terms with the fact that some amount of tactical centralization is inevitable, required, and even desirable¹ in the context of a issue tracking systems.

Add to this the frequent requirement that non-developer users often need to track and create issues, and the result is that we’ve arrived at something of an impasse.

Requirements

A distributed bug tracking system would need:

A good way to provide short, unique identifiers for individual issues and comments so that users can discuss issues canonically.
An interface contained in a single application, script, or binary, that you could distribute with the application.
A simple/lightweight web-based interface so that users can (at least) review, search, and reference issues from a web browser.

Write access would also be good, but is less critical. Also, it might be more practical (both from a design and a workflow perspective,) to have users submit bugs on the web into a read-only “staging queue,” that developers/administrators would then formally import into the project. This formalizes a certain type of triage approach that many projects may find useful.
To be separable from the source code history, either by using a branch, or by using pre-commit hooks to ensure that you never commit changes to code/content and the bugs at the same time.
To be editable, and to interact with commonly accessible tools that users already use. Email, command line tools, the version control systems, potentially documentation systems, build systems, testing frameworks and so forth.
Built on reliable tools.²
To provide an easy way to customize your “views” on bugs for a particular team or project. In other words, each team can freely decide which extra fields get attached to their bugs, along with which fields are visible by default, which are required, and so on--without interfering with other projects.

The Future of the Art

We (all) need to work on building new and better tools to help solve the distributed issue tracking problem. This will involve:
- learning from the existing attempts,
- continuing to develop and solidify the above requirements,
- (potentially) test and develop a standard (yaml/json?) based data storage format that is easy to parse, and easily merged that multiple tools can use.
- Develop some simple prototype tools, potentially as a suite of related utilities (a la early versions of git.) that facilitate interaction with the git database. With an eye towards flexibility and extensible.
While there are implications for free software hosting as well as vendor independence and network service autonomy (a la `Franklin Street Statement <http://autonomo.us/2008/07/franklin-street-statement/>`_.) I think the primary reason to pursue distributed bug tracking has more to do with productivity and better engineering practices, and less with the policy. In summary:
- Bug database systems that run locally and are fast³ and always available.
- Tools that permit offline interaction with issue database.
- Tools that allow users to connect issues to branches.
- Tools that make it possible to component-ize bug databases in parallel with software

Resources

(With commentary,)

dist-bugs mailing list

This is the canonical source for discussion around distributed bug tracking.
Bugs Everywhere

This is among the most well developed solution speaking holistically. “be” is written in Python, can generate output for the web. It uses its own data format, and has a pretty good command line tool. The HTML output generate is probably not very fast at scale (none are,) but I have not tested it.
Ditz

Ditz is a very well developed solution. Ditz: implemented in Ruby, has a web interface, has a command line tool, uses a basic YAML data format, and stores data in branch. Current development is slow, getting it up and running is non-trivial, and my sense is that there isn’t a very active community of contributors. There are reasons for this, likely but they are beyond the scope of this overview.
pitz

Pitz is a Python re-implementation of Ditz, and while the developer(s?) have produced a “release,” the “interface” is a Python shell, and to interact with the database you have to, basically write commands in Python syntax. From a data perspective, however, Pitz, like Ditz is quite developed. Pitz while it stores data in-tree, I think it’s important source of ideas/examples/scaffolding.
Artemis

This is a really clever solution that uses Maildirs to store issues. As a result you can interact with and integrate Artimis issues with your existing email client. Pull down changes, and see new bugs in your email, without any complicated email and list server setups.

The huge caveat is that it’s implemented as a plugin for Mercurial, and so can’t be used with git projects. Also, all data resides in the tree.
git-issues

In most ways, git-issues is my favorite: it’s two Python files, 1700 lines of code, stores issues outside of the source branch, and has a good command line interface. On the downside, it uses XML (which shouldn’t matter, but I think probably does, at least in terms of attracting developers,) and doesn’t have a web-based interface. It’s also currently un-maintained.
Prophet/sd

SD, which is based on a distributed database named Prophet, is a great solution. The primary issue is that it’s currently unmentioned and is not as feature complete as it should be. Also a lot of SD focuses on synchronizing with existing centralized issue trackers, potentially at the expense of developing other tools.

It seems that you want centralized issue databases, or at least the fact that centralized issue databases appear canonical is a major selling point for issue tracking software in general. Otherwise, everyone would have their own text file with a bunch of issues, and that would suck. ↩︎
Because I don’t program (much) and it’s easy to criticize architectural decisions from afar, I don’t want to explicitly say “we need to write this in Python for portability reasons” or something that would be similarly unfounded. At the same time, adoption and ease of use is crucial here, both for developers and users. Java and Ruby (and maybe Perl,) for various reasons, add friction to the adoption possibilities. ↩︎
“Is Jira/Bugzilla/etc. slow for you today?” ↩︎

2012 New Haven Singing

Fri, 20 Apr 2012 00:00:00 +0000

I went to a fabulous all day Sacred Harp singing in New Haven Connecticut last weekend. It was great. Thoughts:

Size and Space

This wasn’t a huge singing. There plenty of singings in the Northeast that have higher attendance, but that doesn’t matter, there was something nice about getting to sing with the people assembled.

It helped that the room was great for singing, and it was the perfect size for the crowd. Sacred Harp singers are often big folklore geeks or big music geeks, but I think deep down, we’re all really huge room geeks. Because a room that just sounds and feels good, makes all the difference in the world.

Note to self: Go to more awesome regional singings in great rooms in the future.

Heat

It was a pleasantly warm spring day, and with 80 or so people in a room, it got warm, and while this did subtract something from the comfort level, it also sets the mood somehow, and changes the tone of the day. Also, when the air is a bit more humid (but it only needs to be a bit,) and it’s not as drafty and cold, its easier to keep your voice warmed up. The end result: I (or one) will sing better between April and September.

My Voice Part

In an unusual move for me, I spent 3 out of 4 sessions singing Tenor. While I don’t have a “super bass” voice, I’m defiantly on bass side of baritone. In Sacred Harp, theoretically everyone can sing tenor, and it’s fun to mix things up a bit and songs sound different in different parts. I am also finding that having a sense of another part makes it possible to have a more rich sense of the music. Highly recommended.

Also, every time I sing tenor it takes me 25 minutes to remember (or remind my body) how to do it, so it was particularly nice to have a good long spell of singing to both figure out how to sing the part but also to get more comfortable with it.

My local singing community has been a bit bereft of basses lately, so I haven’t had much opportunity to actually sing tenor as much as I might like recently, so it was a particularly good change of pace.

Song Selection

I had something of an epiphany about leading and choosing your song at a singing.

While choosing a song that you like and enjoy hearing is obviously a part of the processes, I think song choice is more about choosing the right song for the moment, and figuring out what will sound best next, given the previous few songs.

I used to obsess a great deal about what song I would lead, and study it (at least some) before the singing even began. This weekend, I came with a few songs that I’d been thinking about but changed at the last moment when I thought that the song I had picked out wouldn’t fit very well.

The Origins of Morris Dancing

Mon, 09 Apr 2012 00:00:00 +0000

When you’re out in the world Morris Dancing, everyone asks what you’re doing. Actually, more typically people ask “What country is this from?” but I don’t know what that means. In any case, this past weekend I witnessed the following exchange between elderly spectator to the foreman of a certain New York City Men’s Team as the tour was moving between stands:

Spectator: What is this and where is it from?

J.D.: It’s English Morris Dancing, do you know what that is?

Spectator: No.

J.D.: Do you care?

Spectator (pauses, unsure of how to continue) Yes, is it like cricket or football?

J.D. Cricket.

(At this point the spectator continued about his way satisfied and the tour continued to the next stand.)

This exchange, as these usually go, was pretty good. And the cricket part is totally right. There’s actually, as I understand it, a lot of connection between the history of cricket and Morris Dancing: village’s teams would play cricket and dance Morris, Morris Kits were often cricket uniforms with ribbons and bells, and before cricket, most Morris dancers wore black pants.

Still.

Lies About Documentation...

Thu, 05 Apr 2012 00:00:00 +0000

.. that developers tell.

All the documentation you’d need is in the test cases.
My comments are really clear and detailed.

3. I’m really interested and committed to having really good documentation.

This code is easy to read because its so procedural.
This doesn’t really need documentation.

6. I’ve developed a really powerful way to extract documentation from this code.

The documentation is up to date.
We’ve tested this and nothing’s changed.
This behavior hasn’t changed, and wouldn’t affect users anyway.
The error message is clear.

11. This entire document needs to be rewritten to account for this change.

You can document this structure with a pretty clear table.

Often this is true, more often these kinds of comments assume that it’s possible to convey 3-5 dimension matrixes clearly on paper/computer screens.

I can do that.
I will do that.
No one should need to understand.

NYC Subway Strategy

Thu, 29 Mar 2012 00:00:00 +0000

I have a question for game theory/urban planning/transit geeks, in part for practical reasons, and in part for a story I’m developing:

Is there some sort of resource that explains “most efferent” rapid transit rider strategies, perhaps from a game theory perspective?

I’ve been living in NYC for almost a year, with frequent visits for about 6 months before that and I’ve learned things like:

The physical layouts of a number of station complexes and transfer points, to facilitate quick/easy transfers.
A faltering sense of when to take an express and when to take a local, and when it makes sense to switch.
A decent sense of which route will be more direct/quicker in a given situation.
An acceptable sense of which part of the train you need to be on.

I’m interested in knowing if there is any work aimed at a general audience that addresses any of these questions, in particular:
The express/local decision making logic, particularly given situations like:
When does it make sense to walk ~5-10 blocks to an express stop (possibly in the wrong direction,) rather than walk a shorter distance to a local stop that would require a transfer. (If your destination is an express stop?)
At what point in a journey does it make the most sense to transfer between trains?
How does time-of-day affect the logic.
Specific differences (if any) for navigating subway lines with multiple converging services. (e.g. the “M” with regards to the F and E, as well as the R with regards to the N in Manhattan.)
Logic for transferring between non-parallel train services that intersect at multiple points in a given journey (e.g. in Brooklyn, vs. Manhattan, if needed for inter-borough trips; E/M; 4,5,6 vs N, R.)
Logic for transferring between services that run on the same track (e.g. in NYC: N/R, E/C, F/M, etc.) with respect to how your journey overlays the route divergence and convergences.

The Million Words of Crap

Wed, 28 Mar 2012 00:00:00 +0000

There’s this saying, I traced it down once but have forgotten the source again, that before you can write anything good, you have to write a million words of crap.

Well I’ve done it. I’m guessing that most writers hit this number without much difficulty fairly early on in their lives/careers, but it’s hard keep records and do proper accounting of this data. And let’s be frank, it’s kind of a stupid thing to track. While I’m definitely a better writer these days than I was even 2 years ago, I know that I have a lot more to learn.

In any case, I have a firm account of a million words of crap that I’ve written in the last 10 years or so:

tychoish.com this wiki, mostly rhizome accounts for a bit more than 800,000 words.
I wrote a novel in 2002 and 2003 that now exists primarily as a paper volume in a zippered binder that I use as a foot rest. That was 100,000 words.
Since September 26, 2011 I’ve written a bit under 101,500 words for my work. The number is slightly inflated on account of code samples, some repetitive sections, and a few articles that colleagues wrote that I did some significant editing/rewriting.
Between June 2009 and May 2011, I wrote somewhere between 100,000 and 200,000 words for a previous job (again some measure of redundant content, other people’s work, and code samples makes this figure hard to track.)
The Knowing Mars novella that I wrote between July 2007 and ~March 2008 is about 35,000 words.
The novel that I finished the first draft of few weeks ago is about 85,000 words.
I’m 35,000 words into the first draft of a technical book-like object.
I have 10,000 words in a couple of other projects.

Which puts me way way way, over (what? 1,300,000.) I wonder when I broke the million word mark?

Some observations on writing.

I still make all kinds of writing mistakes that I find embarrassing and difficult. Particularly with fiction (which probably only accounts for 200,000-250,000 words,) I feel incredibly clumsy.
At the same time, as long as I have a clear idea of what a piece of text needs to say, I’m reasonably comfortable pulling together a draft without much fuss. Sometimes figuring out what I need to say involves a bunch of reading and long walk, but it’s possible.
I’m getting better at writing less. Writing concisely is hard work and it’s easy to get into the habit of generating “word stew” that don’t say anything useful and are impenetrable to read.
Writing is hard. I’ve always been an awful speller and I have some dyslexic moments, but--as longtime readers of the blog have surely noticed--I’m getting way better and writing cleaner copy.

Editing other peoples writing on a regular basis has also been enlightening because it allows me to put things in perspective, and be able to see value in what I can do with words. Also it’s cool to be able to help other people write.
This is totally a correlation, but I pretty much never write anything longhand, and I think I’m a better writer for it. The downside is that I’m incredibly dependent on my own computer and its setup to do anything.

Onward and Upward!

Loops and Git Automation

Thu, 15 Mar 2012 00:00:00 +0000

This post provides a few quick overviews of cool bits of shell script that I’ve written or put together recently. Nothing earth shattering, but perhaps interesting nonetheless.

Making a Bunch of Symbolic Links.

I have two collection of Maildir folders to store my mail. One in a ~/work/mail folder for my work mail, and ~/mail for my personal projects. I want the work mailboxes to be symbolically linked to the personal ones (some work email, notably from GitHub comes to a personal address, and I want to be able to refile as necessary.) I moved the work maildirs last week, so I needed to create about 15 symbolic links, and so I wrote the following little loop:

cd ~/mail
for i in `find ~/work/mail -maxdepth 1 -mindepth 1`; do
   ln -s $i work.`echo $i | cut -d / -f 6`
done

Should be useful, no?

Commit all Git Changes

For a long time, I used the following bit of code to provide the inverse operation of “git add .”. Where “git add .” adds all uncommited changes to the staging area for the next commit, the following commit automatically removes all files that are no longer present on the file-system from the staging area for the next commit.

if [ "`git ls-files -d | wc -l`" -gt "0" ]; then
  git rm --quiet `git ls-files -d`
fi

This is great if you forget to use “git mv” or you delete a file using rm, you can run this operation and pretty quickly have git catch up with the state of reality. In retrospect I’m not really sure why I put the error checking if statement in there.

There are two other implementations of this basic idea that I’m aware of:

for i in `git ls-files -d`; do
  git rm --quiet $i
done

Turns out you can do pretty much the same thing with the following statement using the xargs command and you end up with something that’s a bit more succinct:

git ls-files --deleted -z | xargs -0 git rm --quiet

I’m not sure why, I think it’s because I started being a Unix nerd after Linux dropped the argument number limit, and as a result I’ve never really gotten a chance to become familiar with xargs. While I sometimes sense that a problems is xargs shaped, I almost never run into “too many arguments” errors, and always attempt other solutions first.

A Note About `xargs`

If you’re familiar with xargs skip this section. Otherwise, it’s geeky story time.

While this isn’t currently an issue on Linux, some older UNIX systems (including older versions of Linux,) had this limitation where you could only pass a limited number of arguments to a command. If you had too many, the command would produce an error, and you had to find another way.

I’m not sure what the number was, and the specific number isn’t particularly important to the story. Generally, I understand that this problem would crop up when attempting to take the output of a command like find and piping or passing it to another command like grep or the like. I’m not sure if you can trigger “too many arguments” errors with globbing (i.e. *) but like I said this kind of thing is pretty uncommon these days.

One of the “other ways” was to use the xargs command which basically takes very long list of arguments and passes them one by one (or in batches?) to another command. My gut feeling is that xargs can do some things, like the above a bit more robustly, but that isn’t experimentally grounded. Thoughts?

Onward and Upward!

Cron is the Wrong Solution

Thu, 08 Mar 2012 00:00:00 +0000

Cron is great, right? For the uninitiated, if there are any of you left, Cron is a task scheduler that makes it possible to run various scripts and programs at specified intervals. This means that you can write programs that “do a thing” in a stateless way, set them to run regularly, without having to consider any logic regarding when to run, or any kind of state tracking. Cron is simple and the right way to do a great deal of routine automation, but there are caveats.

At times I’ve had scads of cron jobs, and while they work, from time to time I find myself going through my list of cron tasks on various systems and removing most of them or finding better ways.

The problems with cron are simple:

Its often a sledge hammer, and it’s very easy to put something in cron job that needs a little more delicacy.
While it’s possible to capture the output of cron tasks (typically via email,) the feedback from cronjobs is hard to follow. So it’s hard to detect errors, performance deterioration, inefficiencies, or bugs proactively.
Its too easy to cron something to run every minute or couple of minutes. A task that seems relatively lightweight when you run it once can end up being expensive in the aggregate when they have to run a thousand times a day.

This isn’t to say that there aren’t places where using cron isn’t absolutely the right solution, but there are better solutions. For instance:
Include simple tests and logic for the cron task to determine if it needs to run before actually running.
Make things very easy to invoke or on demand rather than automatically running them regularly.

I’ve begun to find little scripts and dmenu, or an easily called emacs-lisp function to be preferable to a cron job for a lot of tasks that I’d otherwise set in a cron job.
Write real daemons. It’s hard and you have to make sure that they don’t error out or quit unexpectedly--which requires at least primitive monitoring--but a little bit of work here can go a long way.

Onward and Upward!

Git In Practice

Tue, 21 Feb 2012 00:00:00 +0000

Most people don’t use git particularly well. It’s a capable piece of software that supports a number of different workflows, but because it doesn’t mandate any particular workflow it’s possible to use git productively for years without ever really touching some features.

And some of the features--in my experience mostly those related to more manual branching, merging, and history manipulation operations--are woefully underutilized. Part of this is because Github, which is responsible for facilitating much of git’s use, promotes a specific workflow that makes it possible to do most of the (minimal required) branch operations on the server side, with the help of a much constrained interface. Github makes git usable by making it possible to get most of the benefit of git without needing to mess with SHA1 hashes, or anything difficult on the command-line.

That’s a good thing. Mostly.

Nevertheless, there are a few operations that remain hard with git: I sometimes encounter situations that I have to try a few times before I get it right, and there are commands that I always have to check the man page to figure out how to specify the references. And even then I’m sometimes still confused. So maybe I (or we?) can spend a little bit of time and figure out what processes remain hard with git and maybe try and see if there is a way to make the process a bit more streamlined.

Here’s my list:

Reorder all commits since x commit.

This is basically: find the commit before the earliest one that you want to change, run git rebase -i <commit hash> to reorder the commits even though git sorts the commits in the order that I find most un-intuitive.
Create local branches to track remote branches or repositories.

Setup the remotes, if necessary, and then run: git branch --track <local-branch-name> <remote>/<branch-name> and git config branch.{name}.push {local-branch}:master.
Stash all local changes and switch branches.

It would also be nice if you could figure out way for git (or a helper) to see any open files in your text editor and save/close them if needed.
Pull a commit from the history of one branch into another branch without pulling anything else.

I think this is chery-pick? It might also be nice to pull a series of commits from one branch, rebase them into one commit in the destination branch, and then commit that.
Pretty much every time I’ve tried to use the merge command to get something other than what I would have expected to happen by using “pull,” it ends tragically.

Reader suggestions:

Put your process/procedural frustrations with git here.

How about we work on figuring out how to solve these problems in comments?

Assisted Editing

Thu, 09 Feb 2012 00:00:00 +0000

I learned about artbollocks-mode.el from Sacha Chua’s post, and it’s pretty freaking amazing.

Basically, it does some processing of your writing--while you work--to highlight passive sentences and affected jargon.¹ And that’s all. There are some functions for generating statistics about your writing, but I find I don’t use that functionality often. You can enable it all of the time, or just turn it on when you’re doing editing.

After a few weeks, I’ve noticed a marked improvement in the quality of my output. I leave it on all the time, but I’m pretty good at resisting the urge to edit while I’m writing. Or at least I’m pretty good at picking up again after going back to tweak a wording. In general it’s hard to keep more than a few things in an editing pass at any time.

It turns out that the instant feedback on passive sentences, even though it’s not perfect, is great for improving the quality of my content the first time out. And it’s even better for doing editing work. It’s harder to ignore a passive sentence when the editor is highlighting you see a screen full of them for you.

It’s of course important to be able to ignore its suggestions from time to time, and it’s no harder to ignore than “flyspell-mode” (the on-the-fly spell checker in emacs.)

This is perhaps the clumsiest part of the default distribution, as jargon is terribly specific to the kind of writing you’re doing, and it turns out that one of the “art critic”/post-modern words (i.e. “node”) is a word that I end up using (acceptably, I think) in a technical context when describing a clustered system. And there’s a difference between a technical lexicon and a jargon, and regular expressions aren’t terribly sensitive to this, so the actual list of words that you need to call yourself out on, varies a bit from person to person. But once you customize it, it’s great. ↩︎

Allowable Complexity

Fri, 03 Feb 2012 00:00:00 +0000

I’m not sure I’d fully realized it before, but the key problems in systems administration--at least the kind that I interact with the most--are really manifestations of a tension between complexity and reliability.

Complex systems are often more capable flexible, so goes the theory. At the same time, complexity often leads to operational failure, as a larger number of moving parts leads to more potential points of failure. I think it’s an age old engineering problem and I doubt that there are good practical answers.

I’ve been working on this writing project where I’ve been exploring a number of fundamental systems administration problem domains, so this kind of thing is on my mind. It seems, that the way to address the hard questions often come back to “what are the actual requirements, and are you willing to pay the premiums to make the complex systems reliable?”

Trade-offs around complexity also happen in software development proper: I’ve heard more than a few developers talk in the last few months weigh the complexity of using dynamic languages like Python for very large scale projects. While the quests and implications manifest differently for code, it seems like this is part of the same problem.

Rather than prattle on about various approaches, I’m just going to close out this post with a few open questions/thoughts:

What’s the process for determining requirements that accounts for actual required complexity?
How do things that had previously been complex, become less complex?

Perhaps someone just has write the code in C or C++ and let it mature for a few years before administrators accept it as stable?
Is there an corresponding level of complexity threshold in software development and within software itself? (Likely yes,) and is it related to something intrinsic to particular design patterns, or to tooling (i.e. programming language implementations, compilers, and so forth.)

Might better developer tooling allow us to programs of larger scope in dynamic languages (perhaps?)

Reader submitted questions:

Your questions here.

Answers, or attempts thereat in comments.

Persistent Emacs Daemons

Thu, 02 Feb 2012 00:00:00 +0000

I’ve been subject to a rather annoying emacs bug for months. Basically, when you start emacs with the --daemon switch, and the X11 session exits, and any emacs frames are open, then the emacs process dies. No warning. The whole point, to my mind, of the daemon mode is to allows emacs sessions to persist beyond the current X11 session.

This shouldn’t happen. I think this is the relevant bug report, but I seem to remember that the issue has something to do with the way that GTK interacts with the X11 session and emacs’s frames. It’s something of a deadlock: the GTK has no real need to fix the bug (and/or it’s a behavior that they rely on for other uses,) and it might not really be possible or feasible for emacs to work around this issue.¹

I also think that it’s probably fair to say that daemon mode represent a small minority all emacs usage.

Regardless, I’ve figured out the workaround:

Turns out, it’s totally possible to build GNU emacs without GTK, by using the “Lucid” build. Which is to say, use the windowing system kit built for Lucid Emacs (i.e. XEmacs,) rather than GTK. I was able, using the code below, to get an emacs experience with the new build that seems identical² to the one I used to get with GTK, except without the frustrating crashes every time that X11 spazzed when I decided to unplug a monitor or some such. A welcome improvement, indeed.

The following emacs-lisp covers all of the relevant configuration of the “look and feel” of my emacs session. Install the Inconsolata font if you haven’t already, you’ll be glad you did.

(setq-default inhibit-startup-message 't
              initial-scratch-message 'nil
              save-place t
              scroll-bar-mode nil
              tool-bar-mode nil
              menu-bar-mode nil
              scroll-margin 0
              indent-tabs-mode nil
              flyspell-issue-message-flag 'nil
              size-indication-mode t
              scroll-conservatively 25
              scroll-preserve-screen-position 1
              cursor-in-non-selected-windows nil)

(setq default-frame-alist '((font-backend . "xft")
                            (font . "Inconsolata-14")
                            (vertical-scroll-bars . 0)
                            (menu-bar-lines . 0)
                            (tool-bar-lines . 0)
                            (alpha 86 84)))

(tool-bar-mode -1)
(scroll-bar-mode -1)
(menu-bar-mode -1)

Hope this helps you and/or anyone else that might have run into this problem.

I’d like to add the citation and more information here, but can’t find it. ↩︎
To be fair, I mostly don’t use the GUI elements in emacs, though having emacs instances outside of the terminal is nice for displaying images when using emacs-w3m, and for having a little bit of additional display flexibility for some more rich modes. ↩︎

A Life Changing Laptop Riser

Wed, 01 Feb 2012 00:00:00 +0000

*tl;Dr>* I got one of those nifty laptop risers that puts your laptop up closer to eye level, and it has pretty much improved all of my interactions with computers a thousand fold and it’s made it possible for me to effectively use two screens. This post explores this.

One of my coworkers had a laptop stand she wasn’t using and I asked to borrow it for an afternoon, and my neck stopped hurting. I never thought my neck hurt before, but apparently it does.

Or did.

But there’s more: for years now I’ve kept an extra monitor around (and had one at work) but the truth is that I have never really felt like I’ve been able to get the most out of an external monitor.

Somehow, putting my laptop 4 inches in the air was the little change that made everything better. The laptop is generally on the left of the external monitor, and I have task lists, notes buffers, the chat window, and my status logging window on the laptop, and then three windows on the external (emacs buffer, terminal, emacs buffer) on the right. My primary focus centers between the monitors, but probably edges slightly toward the external, most of the time.

Also, I discovered that I--apparently--have a slight processing/attention defect whereby I find it painful and difficult to focus on things that are happening on the right side of the screen for any amount of time. Which is weird because my right eye has always been noticeably stronger. I’ll ponder this more later.

My virtual desktops for email and web browsing are a bit less rigid, but the same basic idea. Somehow it seems to work. I’ve done a little bit of work recently to get the layouts right, to minimize the impact of the window management of most context switching (scripting various transitions, saving layouts, etc.) In all things are going great.

It strikes me that I’ve not posted here even a little about my setup in a while. The truth is that it’s not terribly surprising and I’ve not changed very much recently. I’m back to one laptop, and as anxious as having one laptop makes me sometimes (I fear the lack of redundancy,) not having to keep it synced makes life easier. I’ve put some time into doing a little bit of polish on all of little bits of configuration/code that I have that makes my computing world go around, but mostly it’s pretty good.

It’s nice, and I’d write more about it, but I want to get back to getting things done around here. Exporting and exploring some of this stuff in greater depth is definitely on my list, so hang in there, and if there’s something you particularly want to see, be in touch.

2011 Retrospective

Thu, 26 Jan 2012 00:00:00 +0000

For the most part, I’m quite happy with everything that I was able to accomplish last year. I’ve moved cities (for the second year in a row) and last year I changed jobs twice: in both cases, I think the current will stick for a while. And I’m working on other projects, with some impressive speed. Last year wasn’t been great for finishing things, but I guess there’s room for improvement this year.

After a fair amount of professional angst I’m finally doing pretty much exactly what I want to be doing: I’m writing a substantial/total revision of a software manual for a company developing an open source database system. I’ll leave you to figure out the details, but it’s great.

A couple years ago, I said to myself, that I wanted to be a “real technical writer,” which is to say, work with engineering teams, write documentation and tutorials for a single product or group of products, and operate on a regular release schedule. I’ve done a great deal of writing for technology companies: from project proposals and journalism, to tutorials and content for distributors, to white papers, marketing, and sales materials. Delightfully, I’ve managed to get there, and in retrospect it’s both somewhat amazing, and incredibly delightful.

A while back, I had dinner with a friend who’s been doing the same thing I do for a long time (we know each other through folk dance and singing,) and by comparing our experiences it was great to learn that my experience is quite typical, both in terms of the work I’m doing and the procedural engineering practice frustrations (e.g. “What do you mean you changed the interface without telling me?!?!")

At work we have this thing where we send in an account of what we did during the day so that other people know what we’re working on, and so that we can keep our team on the same page. After all, when you’re all looking at computer screens all day, and in a few different time zones, it’s easy to loose track of what people are working on.

At the bottom of these emails, we’re prompted to ask “what are your blockers and impediments.” Often I say something clever like “Compiler issue with Spacetime interface or Library.” Or something to that effect. It feels like a good description of the last year.

Onward and Upward!

Aeron Woes

Tue, 24 Jan 2012 00:00:00 +0000

I have an Aeron chair at my desk at home. Confession.

I got it in April when I moved to New York City. The only piece of furniture that I had that I couldn’t move in my (now former) car was my desk chair. I found a good deal on an Aeron chair and I rationalized to myself that the cost of the chair was actually about the cost of movers. Savings right?

It also helped, that I was leaving a job where I had an Aeron chair in my office, and I knew that in the short term I would be working from home. While my old desk chair was (and is) quite nice, it’s not quite the same. Sit in an Aeron chair for a couple of two years, and it’s hard to go back. I’ve sat in other chairs since then, and it’s never quite the same.

Having said that, after a cleaning incident today, I would like to collect a few gripes about the Aeron chair for your consideration.

The assembly right beneath the chair collects dust and dirt in a proportion that doesn’t seem quite possible. It’s clearly an artifact of the mesh, and likely a commentary on the air circulation of my apartment.

Regardless, dusting nightmare.
The arms scuff and scratch on desks, if the bottom of the desk isn’t completely smooth. This isn’t an actual problem: the chair still works fine and is as comfortable as ever, but it’s a annoying.

I’ve never looked at the underside of a desk before seriously. With every other chair I’ve either ordered a variant sans arms, or I’ve take then arms off as soon as possible.

The Aeron arms are low enough that they’ve never bothered me, so I thought “might as well.” But it’s still annoying.

That’s all.

Documentation Emergence

Tue, 17 Jan 2012 00:00:00 +0000

I stumbled across a link somewhere along the way to a thread about the Pyramid project’s documentation planning process. It’s neat to see a community coming to what I think is the best possible technical outcome. In the course of this conversation Iain Duncan, said something that I think is worth exploring in a bit more depth. The following is directly from the list, edited only slightly:

I wonder whether some very high level tutorials on getting into Pyramid that look at the different ways you can use it would be useful? I sympathize with Chris and the other documenters because just thinking about this problem is hard: How do you introduce someone to Pyramid easily without putting blinders on them for Pyramid’s flexibility? I almost feel like there need to 2 new kinds of docs:

easy to follow beginner docs for whatever the most common full stack scaffold is turning out to be (no idea what this is!)
some mile high docs on how you can lay out pyramid apps differently and why you want to be able to do that. For example, I feel like hardly anyone coming to Pyramid from the new docs groks why the zca under the hood is so powerful and how you can tap into it.

Different sets of users have different needs from documentation. I think my “:Multi-Audience Documentation” post also addresses this issue.

I don’t think there are good answers and good processes that always work for documentation projects. Targeted users and audience changes a lot depending on the kind of technology at play. The needs of users (and thus the documentation) varies in response to the technical complexity and nature every project/product varies. I think, as the above example demonstrates, there’s additional complexity for software whose primary users are very technical adept (i.e. systems administrators) or even software developers themselves.

The impulse to have “beginner documentation,” and “functional documentation,” is a very common solution for many products and reflects two main user needs:

to understand how to use something. In other words, “getting started,” documentation and tutorials.
to understand how something works. In other words the “real” documentation.

I think it’s feasible to do both kinds of documentation within a single resource, but the struggle then revolves around making sure that the right kind of users find the content they need. That’s a problem of documentation usability and structure. But it’s not without challenges, lets think about those in the comments.

I also find myself thinking a bit about the differences between web-based documentation resources and conventional manuals in PDF or dead-tree editions. I’m not sure how to resolve these challenges, or even what the right answers are, but I think the questions are very much open.

9 Awesome Git Tricks

Fri, 06 Jan 2012 00:00:00 +0000

I’m sure that most “hacker bloggers” have probably done their own “N Git Tricks,” post at this point. But git is one of those programs that has so much functionality and everyone uses it differently that there is a never ending supply of fresh posts on this topic. My use of git changes enough that I could probably write this post annaully and come up with a different 9 things. That said here’s the best list right now.

::: {.contents} :::

See Staged Differences

The git diff command shows you the difference between the last commit and the state of the current working directory. That’s really useful and you might not use it as much as you should. The --cached option shows you just the differences that you’ve staged.

This provides a way to preview your own patch, to make sure everything is in order. Crazy useful. See below for the example:

git diff --cached

Eliminate Merge Commits

In most cases, if two or more people publish commits to a shard repository, and everyone commits to remote repositories more frequently then they publish changes, when they pull, git has to make “meta commits” that make it possible to view a branching (i.e. “tree-like”) commit history in a linear form. This is good for making sure that the tool works, but it’s kind of messy, and you get histories with these artificial events in them that you really ought to remove (but no one does.) The “--rebase” option to “git pull” does this automatically and subtally rewrites your own history in such a way as to remove the need for merge commits. It’s way clever and it works. Use the following command:

git pull --rebase

There are caveats:

You can’t have uncommitted changes in your working copy when you run this command or else it will refuse to run. Make sure everything’s committed, or use “git stash”
Sometimes the output isn’t as clear as you’d want it to be, particularly when things don’t go right. If you don’t feel comfortable rescuing yourself in a hairy git rebase, you might want to avoid this one.
If the merge isn’t clean, there has to be a merge commit anyway I believe.

Amend the Last Commit

This is a recent one for me..

If you commit something, but realized that you forgot to save one file, use the “--amend” switch (as below) and you get to add whatever changes you have staged to the previous commit.

git commit --amend

Note: if you amend a commit that you’ve published, you might have to do a forced update (i.e. git push -f) which can mess with the state of your collaborators and your remote repository.

Stage all of Current State

I’ve been using a versing of this function for years now as part of my download mail scheme. For some reason in my head, it’s called “readd.” In any case, the effect of this is simple:

If a file is deleted from the working copy of the repository, remove it (git rm) from the next commit.
Add all changes in the working copy to the next commit.

git-stage-all(){
   if [ "`git ls-files -d | wc -l`" -gt "0" ]; then; git rm --quiet `git ls-files -d`; fi
   git add .
}

So the truth of the matter is that you probably don’t want to be this blasé about commits, but it’s a great time saver if you use the rm/mv/cp commands on a git repo, and want to commit those changes, or a have a lot of small files that you want to process in one way and then snapshot the tree with git.

Editor Integration

The chances are that your text editor has some kind of git integration that makes it possible to interact with git without needing to drop into a shell.

If you use something other than emacs I leave this as an exercise for the reader. If you use emacs, get “magit,” possibly from your distribution’s repository, or from the upstream.

As an aside you probably want to add the following to your .emacs somewhere.

(setq magit-save-some-buffers nil)
(add-hook 'before-save-hook 'delete-trailing-whitespace)

Custom Git Command Aliases

In your user account’s “~/.gitconfig” file or in a per-repository “.git/config” file, it’s possible to define aliases that add bits of functionality to your git command. This is useful defining shortcuts, combinations, and for triggering arbitrary scripts. Consider the following:

[alias]
all-push  = "!git push origin master; git push secondary master"
secondary = "!git push secondary master"

Then from the command line, you can use:

git secondary
git all-push

Git Stash

“git stash” takes all of the staged changes and stores them away somewhere. This is useful if you want to break apart a number of changes into several commits, or have changes that you don’t want to get rid of (i.e. “git reset") but also don’t want to commit. “git stash” puts staged changes onto the stash and “git stash pop” applies the changes to the current working copy. It operates as a FILO stack (e.g. “First In, Last Out”) stack in the default operation.

To be honest, I’m not a git stash power user. For me it’s just a stack that I put patches on and pull them off later. Apparently it’s possible to pop things off the stash in any order you like, and I’m sure I’m missing other subtlety.

Everyone has room for growth.

Ignore Files

You can add files and directories to a .gitignore file in the top level of your repository, and git will automatically ignore these files. One “ignore pattern” per line, and it’s possible to use shell-style globing.

This is great to avoid accidentally committing temporary files, but I also sometimes put entire sub-directories if I need to nest git repositories within git-repositories. Technically, you ought to use git’s submodule support for this, but this is easier. Here’s the list of temporary files that I use:

.DS_Store
*.swp
*~
\#*#
.#*
\#*
*fasl
*aux
*log

Host Your Own Remotes

I’ve only once accidentally said “git” when I meant “github” (or vice versa) once or twice. With github providing public git-hosting services and a great compliment of additional tooling, it’s easy forget how easy it is to host your own git repositories.

The problem is that, aside from making git dependent on one vendor, this ignores the “distributed” parts of git and all of the independence and flexibility that comes with that. If you’re familiar with how Linux/GNU/Unix works, git hosting is entirely paradigmatic.

Issue the following commands to create a repository:

mkdir -p /srv/git/repo.git
cd /srv/git/repo.git
git init --bare

Edit the .git/config file in your existing repository to include a remote block that resembles the following:

[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = [username]@[hostname]:/srv/git/repo.git

If you already have a remote named origin, change the occurrence of the word remote in the above snippet with the name of your remote. (In multi-remote situations, I prefer to use descriptive identifier like “public” or machine’s hostnames.)

Then issue “git push origin master” on the local machine, and you’re good. You can us a command in the following form to clone this repository at any time.

git clone [username]@[hostname]:/srv/git/repo.git

Does anyone have git tricks that they’d like to share with the group?

6 Awesome Arch Linux Tricks

Wed, 04 Jan 2012 00:00:00 +0000

A couple of years ago I wrote “Why Arch Linux Rocks” and “Getting the most from Arch Linux.” I’ve made a number of attempts to get more involved in the Arch project and community, but mostly I’ve been too busy working and using Arch to do actual work. Then a few weeks ago when I needed to do something minor with my system--I forget what--and I found myself thinking “this Arch thing is pretty swell, really.”

This post is a collection of the clever little things that make Arch great.

::: {.contents} :::

abs

I’m using abs as a macro for all of the things about the package build system that I enjoy.

Arch packages are easy to build for users: you download a few files read a bash script in the PKGBUILD file and run the makepkg command. Done. Arch packages are also easy to specify for developers: just specify a “build()” function and some variables int eh PKGBUILD file.

Arch may not have as many packages as Debian, but I think it’s clear that you don’t need comprehensive package coverage when making packages is trivially easy.

If you use Arch and you don’t frequent that AUR, or if you ever find yourself doing “./configure; make; make install” then you’re wasting your time or jeopardizing the stability of your server.

yaourt

The default package management tool for Arch Linux, pacman, is a completely sufficient utility. This puts pacman ahead of a number of other similar tools, but to be honest I’m not terribly wild about it. Having said that, I think that yaourt is a great thing. It provides a wrapper around all of pacman’s functionality and adds support for AUR/ABS packages in a completely idiomatic manner. The reduction in cost of installing this software is quite welcome.

It’s not “official” or supported, because it’s theoretically possible to really screw up your system with yaourt but if you’re cautious, you should be good.

yaourt -G

The main yaourt functions that I use regularly are the “-Ss” which provides a search of the AUR, and the -G option. -G just downloads the tarball with the package specification (e.g. the PKGBUILD and associated files) from the AUR and untars the archive into the current directory.

With that accomplished, it’s trivial to build and install the package, but you get to keep a record of the build files for future reference and possible tweaking. So basically, you this is the way to take away the tedium of getting packages from the AUR, while giving you more control and oversight of package installation.

rc.conf

If you’ve installed Arch, then you’re already familiar with the rc.conf file. In case you didn’t catch how it works, rc.conf is bash script that defines certain global configuration values, which in turn controls certain aspects of the boot process and process initialization.

I like that it’s centralized, that you can do all kinds of wild network configuration in the script, and I like that everything is in one place.

netcfg

In point of fact, one of primary reasons I switched to Arch Linux full time, was because of the network configuration tool, netcfg. Like the rc.conf setup, netcfg works by having a network configuration files which define a number of variables which are sourced by netcfg when imitating a network connection.

It’s all in bash, of course, and it works incredibly well. I like having network management easy to configure, and setup in a way that doesn’t require a management daemon.

Init System

Previous points have touched on this, but the “BSD-style” init system is perfect. It works quickly, and boot ups are stunningly fast: even without an SSD I got to a prompt in less than a minute, and probably not much more than 30 seconds. With an SSD: it’s even better great. The points that you should know:

Daemon control scripts, (i.e. init scripts) are located in /etc/rc.d. There’s a pretty useful “library” of shell functions in /etc/rc.d/function and a good template file in``/etc/rc.d/skel` for use when building your own control scripts. The convention is to have clear and useful output and easy to understand scripts, and with the provided material this is pretty easy.
In /etc/rc.conf there’s a DAEMON variable that holds an array. Place names, corresponding to the /etc/rc.d file name, of daemons in this array to start them at boot time. Daemons are started synchronously by default (i.e. order of items in this array matters and the control script must exit before running the next script.) However, if a daemon’s name is prefixed by an @ sign, the process is started in the background and the init process moves to the next item in the array without waiting.

Start-up dependency issues are yours to address, but using order and background start-up this is trivial to implement. Background start ups lead to fast boot times.

Task Updates

Sat, 31 Dec 2011 00:00:00 +0000

Life has been incredibly busy and full lately and that’s been a great thing. I’ve also been focusing my time on big projects recently rather than posting updates here and updating the wiki. And then I have this day job which basically counts as a big project. While I like the opportunity to focus deeply on some subjects, I also miss the blog.

tycho is conflicted about something. Shocking.

In any case, I want to do something useful with this space more regularly. So here I am and expect me more around these parts.

I’ve been working on a total refresh of my Cyborg Institute project. I want it to be an umbrella for cool projects, nifty examples, great documentation, and smart people¹ working on cool projects. If that’s ever going to happen, I need to get something together myself. The first release will contain:

A book-like object, that provides an introduction to the basic principals of Systems Administration for developers, “web people,” and other people who find themselves in charge of systems, without any real introduction to systems administration. (Status: 70% finished, with a couple more sections to draft and some editing left.)
A Makefile based tasklist aggregator, inspired by org-mode but largely tool agnostic. (Status: 95% finished, with documentation editing and some final testing remaining.)
A logging system for writers. I use it daily, and I think it’s a vast improvement over some previous attempts at script writing, and I did a pretty good job of documenting it, but it’s virtually impossible to manage/maintain. Having said that, I always wanted to rewrite it in Python (as a learning exercise,) so that might be a cool next step (Status: Finished save editing and an eventual rewrite.)
Emacs and StumpWM config files, packaged as “starter-kits” for new users. I have good build processes for both of these. I don’t think that I need to document them fully, but I need to write some READMEs. Since there’s a lot of redistribution of others code, I need to figure out the most compatible/appropriate license. (Status: Finished except for the work of free afternoon.)

Probably, all of these Cyborg Institute projects will get released at about the same time. The blockers will be finishing/editing the book and editing everything else. I might make the release a thing, we’ll see.

Other than that, I:

Updated /technical-writing/compilation.
Finished the first draft of this novel. Editing will commence in June. I’ve also started planning a fiction project, for a draft to begin in the fall?
Wrote a few paragraphs on the ISD page, but I’m starting to think that as my time becomes more limited, that the critical-futures wiki project, as such, will probably be the first thing to fall on the floor, unless someone else is really interested in making that be a thing.

Onward and Upward!

My intention for the Cyborg Institute has always been (and shall remain,) as a sort of virtual think tank for cool projects put up by myself and others. You all, dearest readers, count in this group. ↩︎

Update Pending

Tue, 27 Dec 2011 00:00:00 +0000

It’s been a while since I’ve written one of these “clip posts,” but there’s no time like the present to get started with that. I hope everyone out there in internet-land is having a good end of the year. I’ll try and get a retrospective/new years out in the next few days, and avoid belaboring the point here.

As I said last friday it’s my intent to focus here on shorter/quicker thoughts, and focus my free writing/project time for work on longer projects (fiction, non-fiction, perhaps some programming.) So far so good.

Other Cool Things on the Internet

Longer Forms

Fri, 23 Dec 2011 00:00:00 +0000

A friend asked me a question (several weeks ago by publication) on a technical topic and I spent most of the next few days writing a missive on database administration strategy. That seemed like a normal response. I was delighted to find that: I liked the voice, I enjoyed writing the longer document, and there are a dozen or so other related topics that I wanted to explore. So, apparently, I’m writing a book. This is exactly what I need: more projects. Not.

But it’s a good thing: I find the writing inspiring and invigorating. I have a perspective and collection of knowledge that hasn’t been collected and presented in a single place. I like long form writing. The larger piece might also be a good contribution to my portfolio (such as it is.)

I think this kind of writing suits my attention span.

This has left me without a lot of spare time for blogging, and (as I’m prone to do every so often,) rethinking the future of my efforts on tychoish.com and as a blogger. This is boring for all of you, but I’ll give some higher level stuff here and we can follow up in comments:

Blogging is fun, and even though I’ve not been posting regularly, I’m always writing blog posts. Sometimes I find myself writing posts in emails to friends, but I’m never really going to stop writing blog posts.
The general explosion of blog publishing that we saw a few years ago has declined. Audience fragmentation happened, readership got entrenched. I feel like I weathered the storm pretty well and I’m really happy with the site and readers I have, but I’m also pretty confident that blogging isn’t going to be the means by which I “level up."¹
eBooks have finally happened. For the last decade most people have been saying that ebooks are great for reference material (given search-ability,) and for providing an introduction to a text that people will eventually buy in a paper edition. That may be true, but I think it’s changing rapidly, and with kindles and tablets and smart-phones, I think eBooks have effectively won, such as it is.

In another ten years, perhaps, we’ll just call them books.
I’m pretty clear that keeping a blog, and perhaps most of the writing I do in my spare time is for my own enjoyment and betterment and helps to develop a personal portfolio and account of my work. I have no (real) interest in using my writing on tychoish.com or any other side that I maintain, as a way of supporting myself to any greater or lesser extent.

I want to be in the business of writing things and working with technology and ideas and people, not the business of publishing. While the line is not always clear between “writing projects that you publish yourself online,” and “new media publisher,” I want to stay away from the later as much as possible.

So I think this means that most of my “tychoish,” writing time will go to writing this book project, and to fiction, and once my blog post backlog is fully depleted (heh,) most of my postings will either be announcements/progress-reports or a bunch of shorter more off-the-cuff notes.

Here’s hoping at least.

I can’t really believe that I just used “level up” in this context. ↩︎

Whiteness and Diversity

Thu, 22 Dec 2011 00:00:00 +0000

This post is a follow up to my earlier post on diversity and representation In short, while I think it’s great that we’re beginning to talk and write about race and representation in our fiction and field, I think we¹ need to expand our analysis of whiteness.

Whiteness in Science Fiction

I’m still working on figuring out what this means, and I’m sorry that I haven’t developed my thinking sufficiently to be more clear on this. In light of that here are a collection of my thoughts on representation:

Whiteness is multiple and I think it’s possible (and important) to depict whiteness and white characters critically and without recapitulating normalization. At the same time, it’s important to avoid falling victim to a lot of the normalization to which uncritical representations of racial diversity often fall pray.
The theory around race and representation must deal with issues around assimilation. More diversity is useful, but to move forward on issues of representation, the field needs to better understand the process of assimilation. I want to see stories that help us unpack assimilation.
Whiteness is complex and a major problem with stories that “don’t do race well,” is not just that the characters aren’t explicitly of color, but that whiteness isn’t portrayed very well. This is part of the struggle of privilege, but not only does science fiction need to be better about diversity and representation of non-white characters, but we the thinking on whiteness needs to continue to evolve apace.

Diversity and Quotas

Discussions about diversity and representation in fiction often lead the under-informed to ask “So what, do you want to impose some sort of quota system? Does that mean diversity is more important than quality?”

The answer is almost always no.

I’d also like to point out that this is one of those cases where whiteness and systematic bias conspire to define “quality,” in unuseful ways. But this is another argument for another time.

The canonical answer is: there’s a great deal of amazing work written by people of color and a lot of great fiction that incorporates and addresses the experiences of people of color. This is great, and if we’ve learned anything in the last couple of years, it’s that if you look for this work it’s there. The real challenge revolves around cultivating that work so that there’s more of it, and promoting² that work so that there’s a large audience.

The science fiction writing/reading/editing community. ↩︎
Promoting and marketing literature is by no means a solved problem under any conditions. ↩︎

Technical Writing Fiction

Wed, 21 Dec 2011 00:00:00 +0000

On Outer Alliance Podcast #8, David Levine talked about having worked as a technical writer for some 15 years and then said something to the effect of “It’s a point of great personal pride that I’ve never put a bulleted list in a piece of fiction.”

I laughed out loud. Perhaps frightening a woman walking her dog nearby.

In most ways, the kind of writing that I do for work, API references, tutorials, administration overviews, best-practice descriptions, is very different from the kinds of things I write away from work, or at least I like to think so.

The truth is that I’ve learned a bunch about writing and about communicating in general from writing documentation. While my “professional background,” doesn’t include formal technological training, I definitely “broke in” because I was familiar with technology and could write, rather than being a particularly skilled or trained writer. Any more (just 2.5 years on,) I think the inverse is more true, but that’s conjecture.

Technical writing has definitely shaped the evolution of my taste: a couple years ago, I found myself most drawn to complex tightly constructed prose in fiction. These days I mostly go for sparse clear concise prose that isn’t particularly ornamented. Perhaps it’s only really possible to tune the internal editor for one kind of style at a time.

Having said that, I will confess to feeling--and resisting--the urge to put a bulleted list or some other structured convention of software manuals in fiction.

It’s the little things, really.

Xen and KVM: Failing Differently Together

Tue, 20 Dec 2011 00:00:00 +0000

When I bought what is now my primary laptop, I had intended to use the extra flexibility to learn the prevailing (industrial-grade) virtualization technology. While that project would have been edifying on its own, I also hoped to use the extra flexibility to some more consistent testing and development work.

This project spurned a xen laptop project, but the truth is that Xen is incredibly difficult to get working, and eventually the “new laptop” just became the “every day laptop,” and I let go of the laptop Xen project. In fact, until very recently I’d pretty much given up on doing virtualization things entirely, but for various reasons beyond the scope of this post I’ve been inspired to begin tinkering with virtualization solutions again.

As a matter of course, I found myself trying KVM in a serious way for the first time. This experience both generated a new list of annoyances and reminded me about all the things I didn’t like about Xen. I’ve collected these annoyances and thoughts into the following post. I hope that these thoughts will be helpful for people thinking about virtualization pragmatically, and also help identify some of the larger to pain points with the current solution.

Xen Hardships: It’s all about the Kernel

Xen is, without a doubt, the more elegant solution from a design perspective and it has a history of being the more robust and usable tool. Performance is great, Xen hosts can have up-times in excess of a year or two.

The problem is that dom0 support has, for the past 2-3 years, been in shambles, and the situation isn’t improving very rapidly. For years, the only way to run a Xen box was to use an ancient kernel with a set of patches that was frightening, or a more recent kernel with ancient patches forward ported. Or you could use cutting edge kernel builds, with reasonably unstable Xen support.

A mess in other words.

Now that Debian Squeeze (6.0) has a pv-ops dom0 kernel, things might look up, but other than that kernel (which I’ve not had any success with, but that may be me,) basically the only way to run Xen is to pay Citrix¹ or build your own kernel from scratch, again results will be mixed (particularly given the non-existent documentation,) maintenance costs are high, and a lot of energy will be duplicated.

What to do? Write documentation and work with the distributions so that if someone says “I want to try using Xen,” they’ll be able to get something that works.

KVM Struggles: It’s all about the User Experience

The great thing about KVM is that it just works. “sudo modprobe kvm kvm-intel” is basically the only thing between most people and a KVM host. No reboot required. To be completely frank, the prospect of doing industrial-scale virtualization on-top of nothing but the Linux kernel and with a wild module in it, gives me the willies is inelegant as hell. For now, it’s pretty much the best we have.

The problem is that it really only half works, which is to say that while you can have hypervisor functionality and a booted virtual machine, with a few commands, it’s not incredibly functional in practical systems. There aren’t really good management tools, and getting even basic networking configured off the bat, and qemu as the “front end” for KVM leaves me writhing in anger and frustration.²

Xen is also subject to these concerns, particularly around netowrking. At the same time, Xen’s basic administrative tools make more sense, and domU’s can be configured outside of interminable non-paradigmatic command line switches.

The core of this problem is that KVM isn’t very Unix-like, and it’s a problem that is rooted in it’s core and pervades the entire tool, and it’s probably rooted in the history of its development.

What to do? First, KVM does a wretched job of anticipating actual real-world use cases, and it needs to do better at that. For instances it sets up networking in a way that’s pretty much only good for software testing and GUI interfaces but sticking the Kernel on the inside of the VM makes it horrible for Kernel testing. Sort out the use cases, and there ought to be associated tooling that makes common networking configurations easy.

Second, KVM needs to at least pretend to be Unix-like. I want config files with sane configurations, and I want otherwise mountable disk images that can be easily mounted by the host.

Easy right?

The commercial vendor behind Xen, under whose stewardship the project seems to have mostly stalled. And I suspect that the commercial distribution is Red Hat 5-based, which is pretty dead-end. Citrix doesn’t seem to be very keen on using “open source,” to generate a sales channel, and also seems somewhat hesitant to put energy into making Xen easier to run for existing Linux/Unix users. ↩︎
The libvirtd and Virt Manager works pretty well, though it’s not particularly flexible, and it’s not a simple command line interface and a configuration file system. ↩︎

Erstwhile Programmer

Thu, 15 Dec 2011 00:00:00 +0000

This is the story of how I occasionally realize I exist on the continuum of “programmers,” rather than just being an eccentric sort of writer type.

::: {.contents} :::

Evidence

download-mail

I have this somewhat peculiar method of downloading email that I think works great. A few weeks ago, however, I was trying to compress things in “hot storage,” and realized that I had a problem.

For a year or so, I had been automating commits to the git repository that held all my mail. In order to effectively archive and compress some mail, I needed to do some serious rebasing to not only remove a bunch of messages from the current repository but also pull that content from the history and flatten the history somewhat.

The problem was that I had 50,000 commits and there’s simply no effective way to rebase that many commits in a reasonable amount of time, particularly given I/O limitations. So I gave up, started from (relative) scratch, and rewrote the scripts to be a little bit more smart… You know in an afternoon.

See the revised code here: download mail

ikiwiki-tasklist

I’ve written about this before in my post on my new personal organization stuff, but it’s no great announcement that I’m moving away from working in emacs' org-mode and doing more work with ikiwiki and some hand-rolled scripts. I think org-mode is great, it just ended up getting in my way a bit and I think I can get more of what I need to get done in other ways.

I have learned a great deal from org-mode. I made the biggest leap away from org-mode when I wrote ikiwiki tasklist, which does all of the things I had been using org-mode’s agenda for. It’s not a complicated at all: look in some files for some lines that begin with specific strings and put them into a page that is the perfect task list.

See the code here: ikiwiki tasklist.

Common Lisp Weenie

“What Window Manager is that,” he asked.

“StumpWM, it’s written in Common Lisp,” I said, launching into a 30 second pitch for Stump.

My pitch about stump is pretty basic: the Common Lisp interface allows you to evaluate code during run-time without restarting the window manager or loosing state; it’s functionally like screen, which is very intuitive for window management; and it has emacs-like key-bindings, which I think work pretty well.

“So you’re a Common Lisp programmer?”

“Well not really, I mean, I know enough to get by.”

“Right.”

Conclusion

In several (technical writing) job interviews recently, people asked me about my programming experience, and my answer varied a lot.

I know how computer programs work, I know how people write computer programs, I understand how software testing and debugging works, I understand the kinds of designs that lead to good programs and the kinds that lead to bad software. I don’t write code--really--but I can sort of hack things together in shell scripts when I need to.

The answer to the question, these days, is “I’m a programmer in the way that most people are writers: most people are comfortable writing a quick email or a short blurb, but get stuck and have trouble really writing longer or more complicated kinds of text. Reasonably capable but not skilled.”

The above code examples work: they do what I need them to do, and particularly in the case of the mail script, they work much better than the previous iteration. I need to do more work, and I feel like I’m reaching the boundaries of what can be comfortably done in shell scripting. My next big programming project is to go through these two scripts and port them to Python and see if I can add just a little bit of additional functionality in the process.

I’m sure I’ll report to you on this as my work progresses.

Ikiwiki Tasklist Update

Mon, 12 Dec 2011 00:00:00 +0000

I added a few lines to a script that I use to build my task list, and for the first time ever, I opened a file with code in it, added a feature, tested it, and it worked. Here’s the code with enough context so it makes sense (explained later if you don’t want to spend the time parsing it:)

ARG=`echo "$@" | sed -r 's/\s*\-[c|p|s]\s*//g'`
WIKI_DIR="`echo $ARG | cut -d " " -f 1`"
if [ "`echo $ARG | cut -d " " -f 2 | grep -c /`" = 1 ]; then
   TODO_PAGE="`echo $ARG | cut -d " " -f 2`"
elif [ "`echo $ARG | cut -d " " -f 2 | grep -c $EXT`" = 1 ]; then
   TODO_PAGE="$WIKI_DIR/`echo $ARG | cut -d " " -f 2`"
else
   TODO_PAGE="$WIKI_DIR/`echo $ARG | cut -d " " -f 2`.$EXT"
fi

This is from the section of the script that processes the arguments and options on the command line. Previously, commands were issued such that:

ikiwiki-tasklist [-c -p -s] [DIR_TO_CRAWL] [OUTPUT TODO FILE NAME]

My goal with the options was to have something that “felt like” a normal command with option switches and had a lot of flexibility. The two fields that followed: however, I didn’t provide as much initial flexibility. The directory to crawl for tasks (i.e. “[DIR_TO_CRAWL]") was specified the way it is now, but the output file was 1) assumed to have an extension specified in a variable at the top of the script, 2) automatically placed the output file in the top level of the destination directory.

It worked pretty well, but with the advent of a new job I realized that I needed some compartmentalization. I needed to fully use the tasklist system for personal and professional tasks without getting one set of items mixed in with the other. Being able to have better control of the output is key to having full control over this.

The modification detects if the output file looks like a path rather than a file name. If it’s senses a path, it creates the task list in the path specified, with no added extension. If a file name specifies the extension, then you won’t get “.ext.ext” files. And the original behavior is preserved.

I’m a hacker by inclination: I take code that I find and figure out how to use it. Sometimes I end up writing or writing code, but I’m not really a programmer. My own code, at least until recently has tended to be somewhat haphazard and until now (more or less) I’ve not felt like I could write code from scratch that was worth maintaining and enhancing in any meaningful way.

Apparently some of that’s changed.

I’ve made a few additional changes to the scripts, but most of these feel more trivial and can be described as “I learned how to write slightly tighter shell scripts. so if you’re using it you might want to update: the ikiwiki tasklist page is up to date.

Representation and Race Futurism

Thu, 08 Dec 2011 00:00:00 +0000

I had an item on my list of blog posts to write for a couple of years to write something reflecting on “RaceFail,” and finally a gave up, because I didn’t want to write a book, I didn’t know what to say, and I was more interested in the actual discourse itself than finding the “side of right,” in a conversation that was both way too simple and way too complex all at once.

So rather than reboot the conversation, which has ended in some senses and continues on in others, I want to start writing a bit here about race and representation in fiction, but also discussing the way that conversations transpire online. Here’s part one. I’ll figure out some way to index them all together once they’re posted and assembled.

I wrote this scene a while back where a character who grew up on a small¹ outpost visits a space ship. Given relativistic space travel, from the character’s perspective, the crew of the space ship are 750 years old or so, despite being in their subjective early forties. That means the character’s 31st-great-grandparents (roughly) were cousins of the people he’s looking at.

He notices a few things: the people on the ship are all taller than he is and also taller than everyone from the outpost. He also notices that there’s more more skin tone variation amongst the people on the ship than there is among the people in the outpost.

There are a bunches of problems with this story. Including the fact that its not finished and that there are parts of the execution that I think need a lot work. But this part, I quite like. For this story (and I think in general,) I’ve drawn the following conclusion:

Race is temporally constrained. We understand racial difference and our own racial experiences in terms of our current reality. This changes.
The aspects of race which are the result of lineage (skin color, bone structure,) are likely to change over time as lineages continue. We can assume that these kinds of changes will be pronounced in smaller populations over longer periods of time.

To a large extent the tension between the “outpost people” and the “ship people” is the core of the conflict in this story. I’ve been thinking in this story about the impact of colonialism (and race as a result) on societies and political outlook. It’s almost certainly not perfect, but I enjoy the possibilities, the story has its moments, and I’m finding the theory building productive.

I’m circling around a point: in-story diversity, particularly, diversity that reflects late 20th/early 21st century notions of difference alone cannot further thought race and racism. In other words, diversity is not criticism. There are many ways to productively further the discussion of difference in (genre) fiction, lets not stop with representation.

I’ll be writing more about this in the future. Comments are very welcome!

Under a billion people. ↩︎

The Future of File Organization and Security

Tue, 06 Dec 2011 00:00:00 +0000

I was having a conversation with a (now former) coworker (a while ago) about the future of shared file systems, unstructured organization and management, and access control. What follows are a collection of notes and thoughts on the subject that have stuck with me.

Let’s start with some general assumptions, premises, ideas:

File system hierarchies are dead or dying. To have a useful file system hierarchy the following qualities are essential:
Every piece of data needs to belong in one location and only one location.
Every container (e.g. directory or folder) needs to hold at least two objects.
Hierarchy depth ought to be minimized. Every system can use two levels. After the second level, each additional level should only be added a very large number of additional objects are added to the system. If you have 3 functional levels and less than 1000 objects, you might be in trouble.

As you might imagine, this is very difficult to achieve, and the difficulty is compounded by huge amounts of legacy systems, and the fact that “good enough is good enough,” particularly given that file organization is secondary to most people’s core work.

While there are right ways to build hierarchical structure for file system data, less structure is better than more structure, and I think that groups will tend toward less over time.
Access control is a lost cause. Legacy data and legacy practices will keep complex ACL-based systems for access control in place for a long time, but I think it’s pretty clear that for any sort of complex system, access control isn’t an effective paradigm. In some ways, access control is the last really good use of file system hierarchies. Which is to say, by now the main use of strong containers (as opposed to tags) is access control.

I don’t think that “enterprise content management”-style tools are there, yet. I suspect that the eventual solution to “how do I control access to content” will either: be based on a an cryptography key system which will control access and file integrity, or there will be a class of application, a la ECMS, with some sort of more advanced abstracted file system interface that’s actually use-able.

I’m betting on encryption.

Tagging and search are the ways forward. In many cases, the location of files in hierarchy help determine the contents of those files. If there are no hierarchies then you need something more useful and more flexible to provide this level of insight.
Great search is a necessity. Luckily it’s also easy. Apache Solr/Lucene, Xapian, and hell Google Search Appliances make great search really easy.
Some sort of tagging system. In general, only administrators should be able to create tags, and I think single tag-per object (i.e. categories) versus multiple tags per object should be configurable on a collection-by-collection.

Tag systems would be great for creating virtualized file system interfaces, obviating the need for user-facing links, and leveraging existing usage patterns and interfaces. It’s theoretically possible to hang access control off of tag systems but that’s significantly more complicated.

One of the biggest challenges with tag systems is avoiding recapitulating the problems with hierarchical organization.

The most difficult (and most interesting!) problem in this space is probably the access control problems. The organizational practices will vary a lot and there aren’t right and wrong answers. This isn’t true in the access control space.

Using public key infrastructure to encrypt data may be an effective access control method. It’s hard replicate contemporary access control in encryption schemes. Replicating these schemes may not be desirable either. Here are some ideas:

By default all files will be encrypted such that only the creator can read it. All data can then be “world readable,” as far as the storage medium and underlying file systems are concerned.
The creator can choose to re-encrypt objects such that other users and groups of users can access the data. For organizations this might mean a tightly controlled central certificate authority-based system. For the public internet, this will either mean a lot of duplicated encrypted data, or a lot of key chains.
We’ll need to give up on using public keys as a method of identity testing and verification. Key signing is cool, but at present it’s complex, difficult to administer, and presents a significant barrier to entry. Keys need to be revocable, particularly group keys within organizations.

For the public internet, a some sort of social capital or network analysis based certification system will probably emerge to supplement for strict-web-of-trust based identity testing.
If all data is sufficiently encrypted, VPNs become obsolete, at least as methods for securing file repositories. Network security is less of a concern when content is actually secure. Encryption overhead, for processing isn’t a serious concern on contemporary hardware.

Thoughts?

Multi-Audience Documentation

Mon, 05 Dec 2011 00:00:00 +0000

I’ve written before about different types of documentation, and the different purposes and goals that each type services. Rather than rehash what documentation is, I’m interested in using this post to think about ways of managing and organizing the documentation process to produce better documentation more easily, with the end goal of being able to increase both maintainability and usability of documentation resources.

Different groups of users--say: administrators, end-users, and developers--interact with technology in overlapping but distinct ways. For some technologies, the differences between the classes of users is not significant and one set of documentation is probably good do every one, plus or minus a very small set. In most other cases, multiple resources are required to be able to address the unique needs of different user groups. Figuring out effective ways to address the different kinds of questions that various groups of users ask, but in a way that makes sense to those users is often the primary challenge in writing documentation.

Having said that, writing different sets of documentation for different users is a lot of work, but given time its not insurmountable. The problem is after six months or more (say,) or a couple of new releases when its time to update the documentation, there are three manuals to update instead of one. This is pretty much horrible. Not only is it more work, but the chances for errors skyrockets, and it’s just a mess.

The solution, to my mind, is to figure out ways to only ever have to write one set of documentation. While it might make theoretical sense to split the material into multiple groups, do everything you can to avoid splitting the documentation. Typically, a well indexed text can be used by multiple audiences if its easy enough for users to skip to read only the material they need.

The second class of solutions revolves around taking a more atomic approach to writing documentation. In my own work this manifests in two ways:

Setting yourself up for success: understanding how software is going to be updated, or how use is likely to change over time allows you to construct documents that are organized in a way that makes them easy to update. For example: Separate processes from reference material, and split up long processes into logical chunks that you can interlink to remove redundancies.

Unfortunately, in many cases, it’s necessary to learn enough about a project and the different use patterns before you have the background needed to predict what the best structure of the documentation ought to be.
Separate structure from content: This is a publishing system requirement, at the core, but using this kind of functionality must be part of the writer’s approach. Writers need to build documentation so that the organization (order, hierarchy, etc.) is not implicit in the text, but can be rearranged and reformed as needed. This means writing documentation “atoms” in a structurally generic way. Typically this also leads to better content. As a matter of implementation, documentation resource would require a layer of “meta files” that would provide organization that would be added at build time.

In effect this approach follows what most systems are doing anyway, but in practice we need another processing layer. Sphinx is pretty close in many ways but most document formats and build systems don’t really have support for this kind of project organization (or they require enough XML tinkering to render them unfeasible.) Once everything’s in place and once all of the atoms exist, producing documents for a different audience is just a matter of compiling a new organization layer and defining an additional output target.

This also produces problems for organization. If content is segregated into hundreds of files for a standard-book-length manual (rather than dozens, say) keeping the files organized is a challenge. If nothing else, build tools will need to do a lot more error checking and hopefully documentation writers will develop standard file organizations and practices that will keep things under control.

Thoughts? Onward and Upward!

Knitting in Three Dimensions

Wed, 23 Nov 2011 00:00:00 +0000

It’s relatively straight forward to think about knitting in terms of creating two dimensional shapes. Most of us start by knitting something “easy”¹ like a scarf. From there it’s easy enough to teach knitters to create a never ending variety of polygons. This, however, misses what I think of as the really cool part of knitting. I think the way to understand how knitting works, to be able to knit things that more closely resemble what you want, and to have the most fun knitting is to always think about knitting as three dimensional.

This isn’t an elaborate argument in favor of circular knitting: that argument has been fairly well made and I’ll recount my favorite points on request, but circular knitting is a great technique and knitting in three dimensions is an entire practice.

Knitting Gestalts / Knitting Shapes

I’ve written about this before but one of the best parts about sweater knitting is thinking about how the sweater--the whole object--comes together into a garment. Rather than knitting a collection of flat pieces that can be sewn into a garment (tailoring) knitting lets you build and shape garments with various seamless and nearly-seamless methods.

I sometimes describe this kind of knitting as “architectural,” but the key (for me) is thinking about the entire object as a whole. There’s something that’s nearly magical that happens when you can take a few rows curled up on a circular needle and see in your mind fits into the object that you’re knitting. The process of using knitting stitches, increases and decreases to get from the former to the later is relatively trivial if have can think about the entire object (a “knitting gestalt”) in three dimensions in your mind.

Knitting Mechanics

If “knitting gestalts” provide a top-down perspective on knitting, I think there’s a “bottom up” three dimensional perspective that is important when thinking about how stitches fit together. While a big part of knitting has to do wit the shapes and forms, the textures, drape and “hand” of the fabric all have a lot to do with the final evaluation of the object. To understand drape and texture, it’s important to consider the properties of individual knitting stitches and the effects of yarn weight/texture, needle size, and personality of the knitter. The second part (yarn type, needle size, knitting style) is pretty common, the first (knitting stitch) is less so.

I have a favorite example of this kind of thinking. I’m not sure where I learned this but it’s suck with me:

Knitted fabric typically curls. This happens because the “purl side” of the knitted stitch has a greater surface area than the “knit side,” which causes unaltered stocking stitch to roll up. At the same time, the “knit” side of the stitch is a little bit wider than the “purl” side of the stitch, so the edges will curl in. The way to counteract this, is to mix knit-and-purl stitches on the same row to balance the surface areas out and thus counteract the effects. Think about ribbing and seed stitch… Think about knit and purl patterns and how they change the tendency of the fabric to roll. Think about the path of the yarn through a knitting stitch.

See? Isn’t is cool?

In Conclusion

Whatever kind of knitting you want to do is fine with me: I don’t care to tell anyone that the way they knit is wrong. At the same time, I don’t think there’s any sense in being afraid of your knitting: knitting is great fun and I think once you know the basics most knitters can knit just about everything. So my goal in this post, and in all of my knitting posts, is to share my own process and encourage you (all) to branch out in your own work.

Have fun!

Onward and Upward!

Scarf knitting seems so easy and mechanically it is: knit the same number of stitches row after row after row. But there are issues. First, garter stitch to the uninitiated doesn’t look like “knitting,” and with a high rows per inch ratio these scarves take forever to knit. Such projects are always discouraging. ↩︎

Intellectual Audience

Tue, 22 Nov 2011 00:00:00 +0000

My friend Jo wrote a post a while ago that addressed the subject of building an audience for your scholarly work. You can read the post on her blog, here.

One of the things that I think Jo is really great at is thinking practically about academic careers and trajectories in light of the current academic job market. While people working in traditional academic spaces and on a traditional academic course have a different set of challenges than folks like me, her points still resonate.

How do you build networks and audiences? Two things:

You talk to people.

Audiences are built on relationships. While we might like to think that writers and scholars are able to attract audiences purely on the basis of their work, in practice additional work is required.

You make sure you have something to show for yourself.

Everyone’s got ideas, and projects that they’d like to work on. People love to talk about their ideas. Success, I think, comes when you have something to show for yourself and your projects, and give people some level of confidence that your can make good on your ideas.

In sort, write more, publish more. While quality matters some, being more than someone to talks well at parties is really important.

I think this approach is useful for people doing any kind of creative or intellectual work that engages an audience, but I’m interested in your thoughts.

Minimalism Versus Simplicity

Mon, 21 Nov 2011 00:00:00 +0000

A couple of people, cwebber and Rodrigo have (comparatively recently) switched to using StumpWM as their primary window managers. Perhaps there are more outside of the circle of people I watch but it’s happened enough to get me to think about what constitutes software minimalism.

While StumpWM is a minimal program in terms of design and function; however, in terms of ram usage or binary size, it’s not particularly lightweight. Because of the way Common Lisp works, “binaries” and RAM footprint is in the range of 30-40 megs. Not big by contemporary standards, but the really lightweight window managers can get by with far less RAM.

In some senses this is entirely theoretical: even a few years ago, it wasn’t uncommon for desktop systems to have only a gig of ram, so the differences would hardly have been noticeable. Now? Much less so. Until 2006 or so, RAM was the most performance effecting limited resource on desktop system, since then, even laptops have more than enough for all uses. Although Firefox challenges this daily.

Regardless, while there may be some link between binary size and minimalism, I think it’s probably harmful to reduce minimalism and simplicity to what amounts to an implementation detail. Let’s think about minimalism more complexly. For example:

Write a simple (enough) script in Python/Perl and C. It should scan a file system and change the permissions of files such that they match the permissions of the enclosing folder, but not change the permissions of a folder if it’s different from it’s parent. Think of it as “chmod -R” except from the bottom up. This is a conceptually simple task and it wouldn’t be too hard to implement, but I’m not aware of any tool that does this and it’s not exactly trivial (to implement or in terms of its resource requirements.)

While the C program will be much more “lightweight,” and use less RAM during while running, the chances are that the Python/Perl version will be easier to understand and use much more straightforward logic. The Python/Perl version will probably take longer to run and there will be some greater overhead for the Python/Perl runtime. Is the C version more minimal because it uses more RAM? Is the Perl/Python program more minimal because it’s interface and design is more streamlined, simple and easier to use?

I’m not sure what the answer is, but lets add the following factor to our analysis: does the “internal” design and architecture of software affect the minimalism or maximalism of the software?

I think the answer is clearly yes, qualified by “it depends” and “probably not as much as you’d think initially.” As a corollary as computing power increases the importance of minimalist implementations matters less generally, but more in cases of extremely large scale which are always already edge cases.

Returning for a moment to the question of the window manager, in this case I think it’s pretty clear: StumpWM is among the most minimal window managers around, even though it’s RAM footprint is pretty big. But I’d love to hear your thoughts on this specifically, or technological minimalism generally.

Back to Basics Tasklist and Organization

Mon, 14 Nov 2011 00:00:00 +0000

I’m a huge fan of emacs' org-mode on so many levels: as an IDE for knowledge workers, as a task management system, as a note taking system, and as the ideal basic mode for so many tasks. However, I’ve been bucking against org-for a number of tasks recently. The end result is that I’m becoming less org-dependent. This post is a reflection on how I’ve changed the way I work, and how my thinking has changed regarding org-mode.

Fair warning: this is a really geeky post that has a somewhat specialized context. If you’re lost or bored. check back later in the week.

The Perils of Org

The problem I keep running into with org is that I really don’t prefer to work in org-mode.¹ Org is great and very flexible, but I don’t like that it means that all text-based work is dependent on emacs. My brain is already wired for Markdown and reStructured Text from years of blogging and work projects respectively.

And then there’s this organization problem. There are two ways you can organize content in org-mode. The first is to just dump every thing in one org-mode file and use the hierarchical outlining to impose organization to organize everything. The second is to have every project inside of it’s own file and use outlining incidentally as the project needs it. Content aggregation happens in the agenda.

The problem with the “large files” approach is that you end up with a small handful of files with thousands of lines and imposing useful organization is difficult (too many levels and things get buried; not enough and inevitably your headings aren’t descriptive enough and you get confused. Furthermore, I end up living in clone-indirect-buffer-other-window’d and org-narrow-to-subtree’d buffers, which is operationally the same as having multiple files it just takes longer to set up.

The problem with the other approach, having lots of different files, is that I have a hard time remembering what is in each file, or in logically splitting big projects into multiple files. The agenda does help with this, but the truth is that the kinds of org-headings for organization and tasks are not always the same kinds of headings that make sense for the project itself. I often need more tasks than organizational divides in a project. I tried this approach a couple of times, and ended up with useless mush in my files.

Typically, I can never make the “lots of file approach” really work, and the big files problem lead me to general avoidance of everything. Not good. The key to success here is good aggregation tools.

Hodgepodge

In response, I’ve made a couple of tweaks to how I’m doing… pretty much everything. That is:

I’ve moved most of my open projects into a locally ruining and compiling ikiwiki instance. Both laptops have this setup, and there’s a central remote to keep both (all?) machines in sync.
I’m using ikiwiki tasklist to basically replicate the functions of org-agenda. Basically this crawls the entire wiki looking for lines that begin with certain keywords and generates a “todo” page based on these notes. Really simple, incredibly useful and it solves much of my aggregation needs.
I still have some stuff in org-mode: notes for the nearly-finished novel, lots of random old (legacy) data, 12 various open tasks, and org-capture. I’m thinking of pointing various org-capture templates at files in the wiki but haven’t gotten there yet.
I’ve basically taken the “lots of little files,” approach to my writing and work. I’ve not over-leaded the system yet. Each major project gets a page in the root level of the wiki for overview and planing, and then sub-pages for all related project files (if/as needed)
It turns out that the markdown-mode for emacs has gotten a few improvements since the last time I downloaded the file, including better support for wiki-links that are mostly compatible with ikiwiki. Also from the same developer deft which implements a pretty nifty incremental search for text files in a given directory. So between these tools, ikiwiki, and the ikiwiki-tasklist there’s support for the most important things.
In terms of publishing, beyond ikiwiki for tychoish.com and the personal organization instance, I have a couple of other smaller wikis (also ikiwiki powered,) and I’ve been playing with Sphinx as publishing for more structured documents and resources (i.e. documentation, novels, and collections,) particularly those that need multiple formats and presentations.

I’m sure there will be more shifts in the future, I’m sure. I think this is a good start. Thoughts?

This has pretty much always been the case. I think of it as a personal quirk. ↩︎

Cyberpunk Sunset

Thu, 10 Nov 2011 00:00:00 +0000

I’m not sure where I picked up the link to this post on the current state of cyberpunk, but I find myself returning to it frequently and becoming incredibly frustrated with the presentation.

In essence the author argues that while the originators of the cyberpunk genre (i.e. Gibson and Sterling, the “White Men”) have pronounced cyberpunk “over,” the genre is in fact quite vibrant and a prime location for non-mainstream (“other”) voices and per perspectives. Also, somehow, the author argues that by denying that cyberpunk continues to be relevant and active we’re impinging the diversity that’s actively occurring in the space.

My thoughts are pretty simple:

This is old news. People have been pronouncing cyberpunk dead since 1992 or thereabouts. And they’ve largely been right. Cyberpunk died, because the technological horizon 1980s (e.g. BBSs) developed in a particular way. In someways the cyberpunks got it right (there is a digital reality, there are digital natives, and unique digital social conventions.) In many ways no one got it right: more people are using the internet per-capita than anyone thought in 1984 and no one predicted that the internet would be as commercial as it is.

In light of this the kinds of things that the people active in technology and in cyberpunk are thinking about and addressing have changed a lot. In many ways, Cory Doctorow is a pretty fitting heir to the cyberpunk lineage, but I think it’s also true that the cyberpunk tradition has shifted it’s focus into other issues and ideas.

That interest in the present and the near future has always been a significant defining characteristic of cyberpunk, at least as relevant as the DIY and outsider aspect. In this respect, cyberpunk’s critique was accepted and quite transformative for the genre.

At the same time, the “hackers,” and “cyberpunks,” grew out of academia (e.g. Free Software) and not the punk movement.
The cyberpunks, even when (white) men were the front men for the (sub)genre, have always been outsiders. In the 80s were the “Young Turks” of the science fiction world. Samuel Delany’s Nova is often cited a key cyberpunk-precursor, and there are some pretty important precursors in Stars in My Pocket, Dhalgren, and The Einstein Intervention.
I want to be sure to not forget about Melissa Scott while we’re at it. Trouble and her Friends is a great example of using cyberpunk to explore subcultures and experiences of people (queers, PoC, etc.) on the margins. While Trouble is almost on the late end for “original” cyberpunk I think it counts. The blogger seems to think that only queers and PoC and others have only recently taken up cyberpunk, and that seems particularly shortsighted, and not particularly true.
One of the most troubling aspects of the argument is the assumption that if “cyberpunk” is over than no one can write cyberpunk anymore and that to declare such would be to silence all of the would be *punks.

This is absurd.

Not only is this not true, but it’s also not how literature works. I’m also pretty sure that this is not consistent with the origins of cyberpunk, or the way the genre memes play out.

What I think happened when cyberpunk stopped being on the cutting edge and we realized that a critique of the present required different science fictional method (I think that resurgence in “New Space Opera” in the 90s is part of this, as well as a hard-SF turn in the form of Beggers in Spain and a turn toward alternate histories.) As a result, what’s happening cyberpunk has become something closer to fantasy.

The division (and implications) of the difference between “fantasy” and/or “super soft science fiction” and the science fiction mainstream is at play and probably out side of the scope of this post.

So I’m not that sure where we’re left? Am I missing something? Lets hear it out in comments!

Arranging Patterns for Sweater Design

Tue, 08 Nov 2011 00:00:00 +0000

There are two major problems in sweater making (design.) First, figure out how to make the shape you want out of knitting, and second to place some sort of ornamental feature (pattern) in the knitting without disrupting the shape.

Shaping isn’t easy, but it’s solvable. Once you figure out how to make the shapes you want, it’s just a matter of implementing a known process. Shaping becomes trivial.

The second problem, the design, is the really clever part.

Fitting patterns and embellishments onto a shape, just isn’t solvable, and never becomes trivial. There are tricks, and practices makes it easier, but the possibilities are truly endless and there’s no reason to make any two sweaters exactly the same.¹

The patterns and embellishments can be pretty broadly defined: cables, colorowork, other texture patterns, and so forth.

In my own knitting, I have taken to focusing almost entirely on the second problem. I have a basic sweater form that works really well for me, and each sweater explores a different combination of patterns.

My approach is as follows:

Divide the sweater into quadrants and plan a single quadrant of the sweater. Repeat this pattern over the entire sweater. This automatically centers the pattern on the sweater.
Be flexible with the number of stitches but not too flexible. Also remember to account for a “middle” and “end” stitch which may not be repeated on every quarter.
Think of birds eye view of the design. This means thinking about sweater design as a collection of pattern columns.
Use patterns at either side of the sweater both for a nice effect and to “bound” the patterns. This can be helpful in controlling the number of stitches.
Unless you plan to knit your sweater horizontally, plan your sweater virtually even if you have horizontal patterns. It’s crucial to center patterns that run horizontally across the sweater. Thinking vertically is the easiest way to do this visually for all kinds of patterns.

If you’re using only one pattern that’s fewer than 10-15 stitches, you may be able to just make sure that your pattern divides in the total number of stitches in the sweater, but that’s a much less common problem.

Beyond that, it’s all trial, error, and practice. Onward and Upward!

There may be some exceptions, but generally. ↩︎

Constraints for Mobile Software

Mon, 07 Nov 2011 00:00:00 +0000

This post is mostly just an overview of Epistle by Matteo Villa, which is--to my mind--the best Android note taking application ever. By the time you read this I will have an Android Tablet, but it’s still in transit while you read this and that’s a topic that dissevers it’s own post.

Epistle is a simple notes application with two features that sealed the deal:

1. It knows markdown, and by default provides a compiled rich text view of notes before providing a simple notes editing interface. While syntax highlighting would be nice, we’ll take what we can get.

2. It’s a nice, simple application. There’s nothing clever or fancy going on. This simplicity means that the interface is clean and it just edits text.

For those on the other side there’s Paragraft that seems similar. While in my heart of hearts I’m probably still holding out for the tablet equivalent¹ of emacs. In the mean time, I think developing a text editing application that provide a number of paradigmatic text editing features and advances for the touch screen would be an incredibly welcome development.

In the end there’s much work to be done, and the tools are good enough to get started.

I want to be clear to say equivalent and not replacement, because while I’d like to be able to use emacs and have that kind of slipstream writing experience on an embeded device, what I really want is something that is flexible and can be customized and lets me do all the work that I need to do, without hopping between programs, without breaking focus, that makes inputting and manipulating text a joy. And an application that we can trust (i.e. open source, by a reputable developer,) in a format we can trust (i.e. plain text.) Doesn’t need to be emacs and doesn’t need lisp, but I wouldn’t complain about the lisp. ↩︎

Update Irregular

Fri, 04 Nov 2011 00:00:00 +0000

I’m sorry that I’ve not posted here very much recently and also that the links in this post will be pretty unadorned. I’ll make up for it with table of contents:

::: {.contents} :::

Ok, so it’s not much, but lets get started.

New Posts

While my posting volume has gone down, I have posted something since the last update post.

Rest assured that there’s more stuff in the pipeline.

Lost Posts

For some reason, that I haven’t figured out and don’t really care to, for a number of months, my posts from July of 2009 went missing. Usually this wouldn’t even be worth mentioning, except I July of 2009 was a big month for me writing wise--I’d just moved to the east coast, I had my first real tech job and my mind was full and I felt on fire. I consider a couple of these posts to be “tychoish classics.” The good news is I’ve found them, so here they are:

They seem to be arranged alphabetically rather than sequentially, Sorry about that!

Contra Dancing Feature

A contra dancing friend of mine solicited an email from me a few weeks ago about a couple of contra related topics, which have worked their way into posts that you can read below.

The Internet is A Cool Place: Tablet/Cloud Computing

I found the following link on twitter from a few of my awesome (former) coworkers. It’s a blog post about a programmer who is using an iPad, a remote server, and a computer to do all of his work. read more

It’s an interesting possibility, frankly and I could probably make the shift easily enough if I wanted. Having said that, I feel like I’m a little too sensitive to TCP/SSH hiccups and I feel weird throwing all of my (potential?) productivity into something totally network dependent.

An Emacs Tip Interlude

I picked up the following little bit of emacs configuration, that I think is wicked cool. It removes some of the limitations on m mini-buffers, which gives them a lot of pretty cool features. It seemed like the kind of configuration that I should have known about and didn’t, so maybe some of you don’t know either.

(setq shell-command-default-error-buffer t)
(setq enable-recursive-minibuffers t)

Git, not Hypertext, Is the tool by which we experience becoming Nomad

I’m not entirely sure why I’m following Stephen Ramsay on twitter, but I am. The other day I saw the following exchange, and I feel like it’s worth recording:

<sramsay> Back in the day, it was fashionable to say that hypertext
  enacted certain theories associated with postmodernism (Deleuze, etc.)
<sramsay> We had it totally wrong.  *Git* is the tool
  by which we experience becoming nomad.

I love this idea, and I’ve been saying variants of this for a while, but it’s nice to get reinforcements. At the same I think it’s possible to easy to loose sight of how git is actually used of git when focusing on its transformative aspects. Which makes the theory read a little more hollow.

Onward and Upward!

Remote Accessibility/Reverse Tunneling/Super Dynamic DNS

Wed, 02 Nov 2011 00:00:00 +0000

I have a question for my system administrator readers. And maybe the rest of you as well.

I run a web server on my laptop that hosts about 8 test sites. Nothing special: mostly test and development sites for various public sites, but from time to time I think, “shit, wouldn’t it be nice if I could just give someone a link to this.” My solution is generally to copy whatever it is that I’m working on up to the server that runs this website, and while that generally work just fine, it could be better.

So here’s what I’m thinking:

I’d like to be able to hook my laptop up to the internet and be able to let people access (some) of the content running on this web server. I don’t want it to be automatic, or open my entire machine to the world (though… I could secure it, I suppose.) The options I’ve considered.

I set up a VPN that I can connect to the public server (that hosts this website) from the laptop, and I have a virtual host (or set of virtual hosts) that proxy requests to the laptop. Wherever I am, it works. I’m not worried about the bandwidth or the strain on the server given the usage pattern I’m expecting.
- Pros: Simple, Secure, works even if I’m on a weird local network. Potentially useful for other kinds of nifty hacking including tunneling all traffic through the VPN on insecure connections.
- Cons: Way complex, and I’m not sure if it will work. I’ll need to set up VPN software. And it’s total overkill.
Some sort of scripted dynamic DNS solution, probably involving running my own DNS server.
- Pros: less proxy madness. Pretty simple.
- Cons: running a DNS server. Won’t work on some (most) local networks.
There has to be some sort of alternate approach using a minimalist tunneling solution. There are a few of them, I think they’re nifty, and it would probably be perfect. I’m just not sure what it is.

…and then half the night later, I finished deploying the VPN. I have to say that I’m really pleased with it:
It can (and has) replaced my ssh tunnels for sending email. That’s pretty great.
The web server stuff works, though I don’t have anything really up there yet. I feel like I need some sort of access restriction method, but I don’t really like any of the options. HTTP Auth is annoying rather than protective, SSL is terribly imperfect and fussy, host based control isn’t very tight.
I think I will be able to finally sacrifice a laptop to the “homeserver” because aside from dis/re-enabling the “sleep on laptop close” function. If needed it’ll be dead simple to convert a sever laptop to a mobile laptop.

Thoughts?

Teaching Writing Skills

Tue, 25 Oct 2011 00:00:00 +0000

All of my friends who have taught composition are appalled when they hear me say that I want to teach writing. But it’s true: I would be interested in having the opportunity to give people the kind of writing education that I never got to have. I’ve even collected a few of these ideas on a very rough “pedagogy” page. This post, by contrast, will be a list of “things I wish I could have learned before I got a job writing.”

How to write in long form. The skils and process for writing something that’s a hundred pages is fundamentally different from the process for writing pieces that are a few hundred words or a few pages. Project management, planning, and organization are totally different skills.
Working with editors. In school, the editing process is very conversational. Editors, comment and ask you to make changes if agree with their judgment. Writers need to learn how to gather requirements, write the best possible content, and then hand it over to an editor who will modify the text without comment. Not only is it important to learn how to “get over this,” but also in how to learn from this kind of editing
How to revise work. While I’ve learned to avoid making a number of mistakes to which I’m particularly prone, and spot those errors when the slip through, it’s really the process of applying for jobs that has taught me how to revise my own writing. Revision is probably the hardest writing skill, and I think there are probably better ways to teach revision than some sort of idealized “drafting process.”
How to write at volume, even when you’re not feeling inspired. We’re pretty good at teaching people to write when they’re inspired or have done a lot of research. But writing Writing needs to be as instinctive as speech and the kind of thing that you don’t need to be inspired to be able to do. Not because anyone writes that much, but it’s a comfort thing.
How to document things. Which is to say, how to record a practice, wprocedure, or interface, to tell people (and your future self,) how to do something. I had to figure this out on my own, and I think people would be much better writers for being 10% worse at writing essays and 10% better at writing processes.

That would do it! I’ve included some work in this direction in the pedagogy page, but comments, are always valuable.

Is Dropbox the Mobile File System Standard

Tue, 18 Oct 2011 00:00:00 +0000

I’ve started using Dropbox on my Android devices recently (and my laptop as a result,¹) and I’m incredibly impressed with the software and with the way that this service is a perfect example of the kind of web services that we need to see more of. While I have some fairly uninteresting concerns about data security and relying on a service that I’m not administrating personally, I think it’s too easy to get caught up the implications of where the data lives and forget what the implications of having “just works,” file syncing between every computer.

I used to think that the thing that kept mobile devices from being “real” was the fact that they couldn’t sell “post-file system” computer use. I’m not sure that we’re ready to do away with the file system metaphor yet. I think Dropbox is largely successful because it brings files back and makes them available in a way that makes sense for mobile devices.

The caveat is that it provides a file system in a way that makes sense in the context for these kinds of “file systemless” platforms. Dropbox provides access to files, but in a way that doesn’t require applications (or users) to have a firm awareness of “real files. Best of all, Dropbox (or similar) can handle all of the synchronization, so that every application doesn’t need to have its own system.

This might mean that Dropbox is the first functionally Unix-like mobile application. I think (and hope) that Dropbox’s success will prove to be an indicator for future development. Not that there will be more file syncing services, but that mobile applications and platforms will have applications that “do one thing well,” and provide a functionality upon which other applications can build awesome features.

This isn’t to say that there aren’t other important issues with Dropbox. Where your data lives does matter, who controls the servers that your data lives on is important. Fundamentally, Dropbox isn’t doing anything technologically complicated. When I started writing the post, I said “oh, it wouldn’t be too hard to get something similar set up,” and while Dropbox does seem like the relative leader, it looks like there is a fair amount of competition. That’s probably a good thing.

So despite the concerns about relying on a proprietary vendor and about trusting your data on someone else’s server, data has to go somewhere. As long as users have choices and options, and there are open ways of achieving the same ends, I think that these issues are less important than many others.

To be fair, I’m using it to synchronize files to the Android devices, and not really to synchronize files between machines: I have a server for simple file sharing, and git repositories for the more complex work. So it’s not terribly useful for desktop-to-desktop sharing, But for mobile devices? Amazing. ↩︎

Why you Don't Want Programers to Write Your Documentation

Wed, 12 Oct 2011 00:00:00 +0000

So the documentation sucks. Hire someone to make the documentation suck less.

Simple enough, right?

Right.

Just don’t hire a programmer to write documentation, even though this seems to be a pretty common impulse. There are a lot of reasons, but here are some of the most important from my perspective:

Programmers focus on the code they write, or might write, to be able to describe and document entire projects. It’s really hard to get programmers to approach documentation from the biggest possible frame.
Programmers have a hard time organizing larger scale documentation resources, because they approach it as a database problem rather than a cognitive/use problem.
Programmers solve problems by writing code, not by documenting it. You can push programmers to write notes and you can push the best programmers who can write to work on documentation; but unless you dedicate an engineer to writing documentation full time (which is a peculiar management decision) documentation will always come second.
I’d wager that every organization large enough to have documentation that sucks is probably large enough to have enough documentation for a full time technical writer.
Engineers, particularly those who are familiar with a piece of technology, do this really interesting thing where they explain phenomena from the most basic assumptions prompted to describe something, but regularly skip crucial steps in processes and parts explanations if they think they’re obvious.

Interesting cognitive phenomena do not make for good documentation.

What am I missing?

Overdue Update

Wed, 05 Oct 2011 00:00:00 +0000

I went through a few days ago to collect all of the updates and work that I’d done since the last time I did one of these posts. Sometimes just looking through an activity log is all you need to remember that you’re actually doing something. Here’s what I’ve been working on:

isd" is a developing page of material for my ongoing story project. I’ve written it in my “Wiki Fiction” post. This particular page is basically sysadminning-punk, if I may be so bold. The story as a whole is largely a future history of human’s expansion into the solar system and beyond, using Kim Stanley Robinson’s idea of Accelerando. This piece in particular is all about the emergence of an asynchronous database network that eventually replaces the Internet as we know it, because interstellar distances render trans-planetary TCP (in any form) unworkable. I’ve got an outline and I’m slowly working to fill in the blanks. Editing, ideas, and any amount of text are all welcome.
I added a page to the technical writing section on “Why the Documentation Sucks,” and I’ve done some additional editing and revising of this section of the wiki. The /technical-writing/documentation-sucks page has also served as the foundation of my post about documentation publishing systems.
I’ve added a number of new posts that I hope won’t be forgotten too quickly. They are:
There has also been some discussions of new and old rhizomes on the discourse pages.
A few weeks ago I posted a few new scripts in the code section. Tell me what you think:
- ikiwiki tasklist
- download mail
My post, 9 Awesome SSH Tricks from March 2011, spent a number of hours on the front page of hacker new. You can read the comments over there, and probably on some other sites as well.

The coolest part about this is that some of you have helped to build a page of ssh tricks on the wiki that go above and beyond the little tricks that I use.

And finally,I ’d like to welcome Kevin Grande, who made a new folk page recently. I’m also very sorry that I haven’t been updating more frequently. I started a new job on September 26, and between that and my usual gallivanting around for singing and dancing my blogging habit has. One the other hand I’m writing about 1200 words a day, and life is pretty good so no complaints there.

Onward and Upward!

Writing Software Beyond Emacs

Tue, 27 Sep 2011 00:00:00 +0000

The ideal writing application is emacs, at least for me. In the absence of emacs (as on a tablet,) I’ve been thinking about what features I actually need in a writing application. While I’ve grown to admire the power of a full Lisp machine in my text editor, I accept that it’s not, strictly speaking required. Here’s a first stab at the list of requirements. Feel free to comment or submit a patch to this page.

Simple, primarily full screen editing.
The ability edit very large files, 100kbs should present no issue.
Some sort of syntax highlight during the editing process, preferably support for LaTeX, Markdown, and org-mode.
Word count generation for the entire files and for current selections.
Auto-save as crash protection.
Undo/Redo last typing action.

Nice to have (but not crucial) features:

The ability to edit one file and reference another (or potentially edit) at the same time. Bonus points for being able to switch between to parts of a single file at once.
The ability to hide or collapse some sections of a file.
Optional spell checking.
Parenthetical and double-quote matching.
Soft and hard line/word wrapping.

Other than that I don’t think there’s anything that I really need to have to get writing done 90% of the time. How about you?

Android Tablets and the Workstations of the Future

Tue, 20 Sep 2011 00:00:00 +0000

I’ve only had the tablet for a few weeks, but I’m pretty sure the tablet incarnation of Android is probably 80% of what most users need in a workstation. I’m not most users, but I figure: hook up a big screen and a real keyboard. Create some key bindings to replace most of the gestures, and write a few pieces of software to handle document production, presentations, and spreadsheets in a slightly more robust manner, and you’re basically there. I wouldn’t give up my laptop today for a tablet, and I think the platforms still have a ways yet to go, but that’s not insurmountable.

Prediction: in the next decade, we’ll see embeded tablet-like devices begin to replace desktops computers for some classes of use and users. General web surfing, reading, quick email, and watching videos on YouTube seem like the obvious niche for now. I started to explore this in “Is Android the Future of Linux,” but it’s not abusrd to suggest that Android or iOS like devices might begin to address more “general purpose desktop computing.”

I want to be clear: we’re not there yet. These systems aren’t versatile and fully featured enough to keep up with full time use on an extended basis. This is mostly an application/software problem. As applications evolve and as more functionality moves to remote systems anyway (this is the “cloud,” we’ve heard so much about,) tablet operating systems will seem much more capable for general purpose work. Better mobile productivity software will help as well. Eventually, I think Android and similar platforms will have a shot at the desktop market for most usage because:

IT departments will get a lot more control over intra-organization information flow, which could save a lot of money for various IT categories: administration, support, and data protection costs.
Behind the firewall dropbox-like services, and creating some sort of centralized workstation configuration management (which makes sense for a flash based device,) means backups can happen automatically, and if devices need to be re-imaged or are lost or damaged, it only takes a few minutes to get someone back to work after a technology failure.
Limited multi-tasking ability will probably increase productivity.
Disconnecting keyboards from the screen will probably lead to better ergonomic possibilities.
Eventually, it will be easier to integrate Android-like devices with various workflow management/content management systems.

The technology needs to mature and workers and IT departments need to become more comfortable with tablets, without question. Also, there are some fundamental developments in the technology that need to transpire before “desktop tablets” happen, including:
More power user-type interface features.
Split screen operation. There are enough “common tasks” that require looking at two different pieces of information at the same time that I think tablets will eventually have to give up “full screen everywhere,” operation. Conventional windowing is unnecessary, and I don’t think anyone would go for that, but displaying two different and distinct pieces of information at once is essential.
Better “Office” software for spreadsheets, presentations and document preparation. A necessary evil.
Behind the firewall (preferably open source) solutions to replace services like Dropbox/Box.net and whatever other services emerge as essential parts of the “tablet/smartphone” stack.
VPN clients and shared file system clients that are de ad simple to use. I think these are features for operating system vendors to develop.

Thoughts? Onward and upward!

Publishing System Requirements

Mon, 19 Sep 2011 00:00:00 +0000

Like issue tracking systems, documentation publication systems are never quite perfect. There are dozens of options, and most of them are horrible and difficult to use for one reason or another. Rather than outline why these systems are less than ideal, I want to provide a list of basic requirements that I think every documentation publishing system¹ should have.

Requirements

Tag System. You have to be able to identify and link different pieces of content together in unique and potentially dynamic ways across a number of dimensions. Tagging systems, particularly those that can access and create lists of “other posts with similar tags,” are essential for providing some much needed organization to projects that are probably quite complex. Tagging systems should provide some way of supporting multiple tag namespaces. Also operations affecting tags need to be really efficient, from the users and software’s perspective, or else it won’t work at realistic scales.
Static Generation. Content needs to be statically generated. There are so many good reasons to have static repositories. It allows you to plan releases (which is good if you need to coordinate documentation releases with software releases) most documentation changes infrequently. The truth is this feature alone isn’t so important, but static generation makes the next several features possible or easier.
Development Builds. As you work on documentation, it’s important to be able to see what the entire resource will look like when published. This is a mass-preview mode, if you will. The issue here, is that unlike some kinds of web-based publications, documentation often needs to be updated in batches, and it’s useful to be able to see those changes all at once because the can all interact in peculiar ways. These test builds need to be possible locally, so that work isn’t dependent on a network connection, or shared infrastructure.
Verification and Testing. While building “self-testing” documentation is really quite difficult (see also /technical-writing/dexy,) I think publication systems should be able to do “run tests” against documents and provide reports, even if the tests are just to make sure that new of software versions haven’t been released, or that links still work. It’s probably also a good idea to be able to verify that certain conventions are followed in terms of formatting: trailing white space, optional link formats, tagging conventions, required metadata, and so forth.
Iteration Support. Documents need to be released and revised on various schedules as new versions and products are developed. Compounding this problem, old documentation (sometimes,) needs to hang around for backwards compatibility and legacy support. Document systems need to have flexible ways to tag documents as out of date, or establish pointers that say “the new version of this document is located here.” It’s possible to build this off of the tag system, but it’s probably better for it to be a separate piece of data.
Version Control. These systems are great for storing content, facilitating easy collaboration, and supporting parallel work. Diffs are a great way to provide feedback for writers, and having history is useful for being able to recreate and trace your past thinking when you have to revisit a document or decision weeks and months later.
Lightweight Markup. It’s dumb to make people write pure XML in pretty much every case. With rst, markdown, and pandoc the like there’s no reason to write XML. Ever. of story End.
Renaming and Reorganization. As document repositories grow and develop, it seems inevitable that the initial sketch of the organization for the body of work change. Documents will need to be moved, URLs will need to be redirected or rewritten, and links will need to be updated. The software needs to support this directly.
Workflow Support. Documentation systems need to be able to facilitate editorial workflows and reviews. This should grow out of some combination of a private tag name space and a reporting feature for contributions, which can generate lists of pages to help groups distribute labor and effort.

This might just be a quirk of my approach, but I tend approach documentation, terms of process and tooling, as if it were programming and writing software. They aren’t identical tasks, of course, but there are a lot of functional similarities And definitely enough to take advantage of the tooling and advances (i.e. make, git, etc.) that programmers have been able to build for themselves. Am I missing something or totally off base?

Think knowledge bases, documentation sites, and online manuals. I’m generally of the opinion that one should be able to publish all of these materials using the same tool. ↩︎

Org Mode and Mobile Writing

Thu, 15 Sep 2011 00:00:00 +0000

This post is adapted from a post I made to the org-mode email list a few weeks ago. I proposed an application to compliment MobileOrg for writing. Where MobileOrg collects the core bits of org-mode’s task planning functionality in a form that makes sense for smart phone users, the parts of org-mode functionality that people use to for writing and organizing the content of larger form projects isn’t particularly accessible.

I spend (or should spend) 70% or more of my time in front of a computer writing or editing something in org-mode. Most of my org files have tens of thousands of words of blog posts, notes, drafts of articles, and so forth. While I can store that data on an android device with only minor problems using a little script that I put together, and I can capture content into my org-files using email and some nifty filters, and there are text editors that can let me edit these files: it could be better.

The proposal is simple. Can we build something like Epistle for org-mode? It might just render org-mode text to HTML, and frankly that would be enough for me. If the editing interface had an org-indent-mode equivalent, org-syntax highlighting, and even collapsing trees or org-narrow-to-subtree, that’d be kind of like heaven.

I’m not a mobile developer, so I can’t promise to start making an app this instant if there’s interest but if anyone’s bored and thinks this might be a good idea (or knows of something that might work better for this.) I’d love to hear about it. If someone wants to start work on this, I’ll do whatever I can to help make this a reality.

Onward and Upward!

Tablet Interfaces and Intuition

Tue, 13 Sep 2011 00:00:00 +0000

I’ve been using FBReaderJ to read .epub files on my tablet recently, and I discovered a nitfty feature: you can adjust the screen’s brightness by dragging your finger up or down the left side of the screen. Immediately this felt like discovering a new keybinding or a new function in emacs that I’d been wishing for a while time. Why, I thought, aren’t there more tricks like this?

The iPhone (and the iPad by extension) as well as Android make two major advances over previous iterations of mobile technology. First, they’re robust enough to run “real” programs written in conventional programming environment. Better development tools make for better applications and more eager developers (which also makes for better applications.) Second, the interfaces are designed to be used with fingers rather than stylus (thanks to capacitive touch screens) and the design aesthetic generally reflects minimalist values and simplicity. The mobile applications of today’s app stores would not work if they were visually complex and had multi-tiered menus, and hard to activate buttons.

The tension between these two features in these platforms makes it difficult to slip nifty features into applications. Furthermore, th economy of application market places does not create incentives for developers to build tools with enduring functionality. The .epub reader I mentioned above is actually free software.¹ I write a couple of posts a while back on innovation (one and two) that address the relationship between free software and technological development but that’s beside the point.

Given this, there are two major directions that I see tablet interfaces moving toward:

1. Tablet interfaces will slowly begin to acquire a more complete gestural shorthand and cross-app vocabulary that will allow us to become more effective users of this technology. Things like Sywpe are part of this, but I think there are more.

2. There will be general purpose systems for tablets that partially or wholly expect a keyboard, and then some sort of key-command system will emerge. This follows from my thoughts in the “Is Android the Future of Linux?” post.

I fully expect that both lines of development can expand in parallel.

I also found the base configuration of FBReader (for the tablet, at least) to be horrible, but with some tweaking, it’s a great app. ↩︎

Issue Tracking and the Health of Open Source Software

Mon, 12 Sep 2011 00:00:00 +0000

I read something recently that suggested that the health of an open source project and its community could be largely assessed by reviewing the status of the bug tracker. I’m still trying to track down the citation for this remark. This basically says that vital active projects have regularly updated bugs that are clearly described and that bugs be easy to search and easy to submit.

I’m not sure that free software communities and projects can be so easily assessed or that conventional project management practices are the only meaningful way to judge a project’s health. While we’re at it, I don’t know that it’s terribly useful to focus too much attention or importance on project management. Having said that, the emergence of organizational structure is incredibly fascinating, and could probably tolerate more investigation.

As a starting point, I’d like to offer two conjectures:

First, that transparent issue tracking is a reasonably effective means of “customer service,” or user support. If the bug tracking contains answers to questions that people encounter during use, and provide a way to resolve issues with the software that’s productive and helps with support self-service. Obviously some users and groups of users are better at this than others.
Second, issue tracking is perhaps the best way to do bottom-up project/product management and planning in the open, particularly since these kinds or projects lack formal procedures and designated roles to do this kind of organizational work.

While the overriding goal of personal task management is to break things into the smallest manageable work units, the overriding goal of issue tracking systems is to track the most intellectually discrete issues within a single project through the development process. Thus, issue tracking systems have requirements that are either much less important in personal systems or actively counter-intuitive for other uses. They are:

Task assignment, so that specific issues can be assigned different team members. Ideally this gets a specific developer can “own” a specific portion of the project and actually be able to work and coordinate efforts on the project.
Task prioritization, so that less important or crucial issues get attention before “nice to have,” items are addressed.
Issue comments and additional attached information, to track progress and support information sharing among teams, particularly over long periods of time with asynchronous elements.

While it’s nice to be able to integrate tasks and notes (this is really the core of org-mode’s strength) issue tracking systems need to be able to accommodate error output and discussion from a team on the best solution, as well as discussion about the ideal solution.

The truth is that a lot of projects don’t do a very good job of using issue tracking systems, despite how necessary and important bug trackers. The prefabricated systems can be frustrating and difficult to use, and most of the minimalist systems¹ are hard to use in groups.² The first person to write a fully featured, lightweight, and easy to use issue tracking system will be incredibly successful. Feel free to submit a patch to this post, if you’re aware of a viable systems along these lines.

I’m thinking about using ikiwiki or org-mode to track issues, but ditz suffers from the same core problem. ↩︎
Basically, they either sacrifice structure or concurrency features or both. Less structured systems rely on a group of people to capture the same sort of information in a regular way (unlikely) or they capture less information, neither option is tenable. Without concurrency (because they store things in single flat files) people can’t use them to manage collaboration, which make them awkward personal task tracking systems. ↩︎

Is Android the Future of Linux?

Tue, 06 Sep 2011 00:00:00 +0000

By now, several weeks ago, in correspondence Matt Lundin that he thought Android was probably future of Linux," mostly as a throw away line. This feels like a really bold statement,¹ and I’ve enjoyed thinking about Android and “the future of Linux."²

On the face of it, Android is the future of Linux. Android is the Linux that most people will interact with before all others in a concrete manner. In all likelihood The future of Linux is probably mostly in running web servers, virtualization hosts, and any other server that matters. At this point, Linux’s platform support and use cases is far less interesting than its prevalence: the ubiquity of Linux, GNU, and BusyBox, is more import an that the fact that Linux runs everywhere in hundreds of different usage profiles.

And really, “desktop Linux” or even “Linux for end-users,” is something of a distraction. We don’t all have to use Linux on the machines beneath our fingers for Linux to be successful. I’m a desktop Linux user because it’s the right system for the work I do, and I can’t work the way I need to with any other kind of system. But I use my systems in a very peculiar way and the thing that makes Linux ideal for me (and the people who are good at building Linux systems,) is not necessarily the qualities that make the best Linux distributions for most users.

As someone who cares about Linux adoption and the use of free software, I don’t want my argument to lead to the very common “let non-technical users use Macs” argument. Although it’s true that OS X can be a convincing introduction to power and use of having a full UNIX-like system on your lap: this was my root (as it were.) Rather, I think that the way to encourage Linux adoption is to increase computer literacy until users respect and value and power that Linux-based systems offer.

Easier said than done, of course.

If this is the case, then Android isn’t a very good introduction to Linux-based operating systems. Not because it’s bad software, but because the kernel is pretty irrelevant to the overall user experience, or the interface that most users have.

Regardless, while madalu is probably right, I don’t think it matters. Android is largely orthogonal to the adoption of Linux. The bigger questions are:

Does Microsoft have a tablet strategy? Really? The last time Linux made headway into users' hands (i.e. netbooks,) Microsoft changed strategies, and not only pushed Linux-based systems out of the market, but they also basically killed the device class. Netbooks really aren’t a thing anymore.
How close are we to tablet-like (or tablet-derived) devices from replacing general purpose computers for some classes of day to day activity. I suspect corporate machines will be the first to fall (more constrained/specific use cases; tablet systems give IT administrators more control, and increasingly work happens in web apps.)

If corporate fleets are the first to fall, the first question becomes much more important. In any case, stay tuned, I’m working on collecting the rest of my thoughts on these questions. In the mean time, I’d look forward to hearing from you.

Onward and Upward!

I would like to fully apologize ahead of the time if I’m characterizing the argument unfairly. ↩︎
Though mostly ceremonial to mark the 20th anniversary, and because there have been 39 releases of the 2.6.x series kernel which is absurd to keep track of after a while, Linux is getting a version boost to version 3.x. ↩︎

Advice for Blogging Successfully

Wed, 31 Aug 2011 00:00:00 +0000

Although I forget it, sometimes, the following video is probably the single best piece of advice for better blogging. Watch the video:

Cory Docotorow - How to be an Uber Blogger

Attention and time are scarce while content is plentiful. “If you write it they’ll read” cannot, thus, be true of blogging. Interesting and important content is interesting, but success lies in managing the attention economy and focusing on output that is easy to read and easy to skim. Its not glamorous, and it requires giving up a bunch of pride, but writers must write with readers in mind.

I’ll skip the meandering analysis and get to a couple of key questions that I think remain open, even 4 years after this video was posted:

What about non-newsreel blogs? Blogs that are more analysis and less regurgitation’s of boingboing/metafilter/slashdot?
What about non-blog content? Are books and articles, subject to the same overload (yes?) but are the solutions always “write easy to process, easy to skip,” content? (Maybe?)
The connection between the “attention” economy, evolving search engine use patterns, and mobile technology change the way that we interact with and compensate for overload?

Discuss!

Update Rhythm

Tue, 30 Aug 2011 00:00:00 +0000

I wonder if, at some point, this constant state of overload and flux in my world will begin to seem normal and I’ll just adjust to that normal. In the mean time, exciting things are happening and I’m not quite sure of the best way to write about them. Perhaps soon. For now, I’m trying to get better about updating more regularly and I have a bunch of links of stuff that have happened on the wiki in the past couple of weeks that I’d like to share. Here we go:

Discussion of Rhizomes

jfm and I had a good exchange about an old post, /posts/ideology-and-systems-administration. Basically the posts says, “systems administrators have a unique approach to solving technological problems,” and discussed the implications of systems administrators background on technology development. I think our clarifications were useful.

There are a couple of comments on my recent series on a productivity. First, I wrote a post about task planning and creating task items, and Matt posted a comment. Second, a number of us had an ongoing conversation on mobile productivity in response to the “Mobile Productivity Challenges” post that touched on emacs (of course!) input, and context switching.

Site Tweaks

This is a pretty minor point, but I’ve been subtly tweaking the design a little in the site. There are now links to the tags page and the site map in the upper right hand corner. I’ve also made links to as-of-yet-uncreated wiki pages red (according to wiki-convention.) I think (and hope) that red links are easier to spot when they’re red. Feedback on the design would be most welcome. My goal is to make the site welcoming, easy to use, and to minimize the amount of “fussiness.” It might be time for a full refresh, but feedback on the subject might be good.

Critical Futures and Wiki Fiction

Eventually the story will move to the Critical Futures domain, but that’s a bit down the road. Right now I’d rather focus my time/energy on writing some stories, for now (on this wiki.) Infrastructure can come next.

I hope to work on a series of posts that explore collaborative fiction organizing over the next few weeks. If people are interested, that is.

External Links

I came across a blog from the comment on the make emacs better post that I wanted to offer as a link The Babbage Files is a great cyborg/emacs/free software blog.

This is probably not news to anyone, but from John Wiegley I learned about the following two emacs gems that merit mention:

paredit which makes handling all of the parentheses in Lisp coding much easier.
redshank, which basically adds a number of tempting systems and associated tools for

And in another direction, I’ve been playing with pylookup. This emacs add on makes it possible to access python documentation from within emacs, from a local copy. The interface is a little bit fiddly but it’s pretty much heaven. More things should work like this.

Onward and Upward!

Wiki Fiction and Critical Futures

Mon, 29 Aug 2011 00:00:00 +0000

I’m starting preliminary development on a wiki fiction project that will eventually take over the criticalfutures.com domain. This post is a discussion of that history, my idea, and what I hope to accomplish.

My friend Julia and I have been corresponding on topics related to the future of publishing and genre fiction for a few weeks. When the topic turned to wikis, I spouted off the things I usually say about bootstrapping wiki participation (it’s hard and pretty lonely,) then I had an idea.

I read wikis, mostly wikipeida, a lot. For fun. I’m sure a lot of people do this, as “getting lost in wikipedia,” is a thing that happens. You say, “I’m interested in public transit in Iran,” and you get lost clicking through various pages related to rapid transit systems in the middle east, and then an hour or two is past and you really ought to finish that blog post. The same thing happens on the c2 wiki for me.

While I wish I were less compulsive about it, reading wikis is a pleasurable reading experience, and since the format seems the web, why not run with it? The question becomes: why are we spending so much time figuring out the most ideal way to publish novels and short stories--forms that developed with the physicality of the book--in the digital age?

To be fair, I think there’s a place for digital distribution of paper-centric forms (periodicals and monographs,) but I doubt that in 50 years “digital fiction,” will mean eBook editions of novels. People have been making a similar point for some time about video games for a while. Interactive fiction is definitely a part of digital fiction, but I don’t think it’s the full story.

Meanwhile back at the point…

Here’s the idea. We use wiki software to construct a website that is written as a light-hearted encyclopedia. In the vein of TV Tropes meets Wikipedia except with fictional content. But there needs to be more than just page after page of exposition and condensed blather: my current plan is to have a “dialogue” section, which will be bits of dialogue and scenes published with some contextual metadata (when it happened, who was present, where it happened.) The dialogues can then be linked to as quasi-citations in the more conventional expository wiki pages.

So basically I’m proposing a couple of things here. First, I want to splitting up all content into small self contained pages. This makes it better for multiple people to edit, because editing and writing can happen in a more parallel manner, and you don’t need to agree to an outline, or write things in any sort of sequence. Second, shorter pages with more segmented content is easier to read for the attention limited.

Having said that, I’m not sure that collaborative, for all that will editing is really the way to go. The truth is that so few people edit wikis relative to the number of people who could edit wikis, that you might be better off having some sort of more select editorial community, just in terms of establishing buy-in from contributors and avoiding diffusion of responsibility. I’m undecided.

Along a similar line of thought, I’m considering releasing release updates and new content on a regular basis (e.g. bi-weekly or monthly?) rather than every time an edit is made. This will require some sort of closed-development process. At the same time new wiki projects often fail because there’s little incentive to return to a wiki to “see what’s changed. Blogs, contrast are good at securing return visits.

Thoughts? Anyone interested in being on the editorial board?

And Then I Broke Down and Got a Tablet

Thu, 25 Aug 2011 00:00:00 +0000

Ok, I caved and got a tablet. This is a post about my experiences with the tablet and some general thoughts on the format.

I opted for the Motorola Xoom. It’s an Android device, I appreciate the Motarola build quality, and I’m very pleased with my choice. First impressions first:

Reading on the tablet is great. I have a Kindle, and while I respect how lightweight the Kindle is itself. Despite the extra weight, the slightly larger screen and the back light is very very nice and very welcome.
I don’t expect that I’ll be doing a lot of writing on the tablet, a laptop is never really going to be that far away, but I’m really surprised by how easy it is to (almost) touch type on the tablet. A number of very simple and probably straightforward innovations to the keyboard could make things so much better.
I think all devices need some sort of “don’t auto rotate” hardware switch. In fact, I think apple’s whole “lets get rid of hardware buttons,” movement to be really annoying. Buttons should be overloaded, sure, but I hate having to hunt through menus to modify basic behavior. Having said that, the “software control bar” at the bottom of Android 3.0 is brilliant and a good move (given screen rotation.)
I lament not having a Google voice widget for the tablet. Makes sense that they wouldn’t want this for tablets that had data plans, but I just have a wifi tablet.
The Kindle app doesn’t let you bookmark your place in periodicals. Which might make sense if you were reading the Times, but doesn’t make a lot of sense when reading fiction magazines with articles in the rage of 10k words.
I’m in love with the calendar application, except for the “full month view,” in which you scroll by weeks, not by months. Even with this glitch, I’m curious as to why there aren’t (stand alone) calendar applications of this quality for desktops.
I’ve tended to use the tablet for situations where I want to have a distraction free experience (usually for reading,) or where I want to do “computer things” in a situation where I might need to interact with other people. Having a tablet in your lap is more social than a laptop. As such, I don’t think it would ever be able to replace a “real” computer for very long, but that doesn’t make it less useful.

I’ll be writing more about the tablet experience and some cyborg features of tablet use and usability.

Onward and Upward!

New Sweater

Tue, 23 Aug 2011 00:00:00 +0000

This post comes in two parts: an update on the current knitting project and an introduction to a new sweater that I’ve started recently. See the Ideal Sweater and Sweater Stories posts for more information on my current thinking about knitting and writing.

Current Sweater: Gray and Black

I’ve been trying to write an update on my current knitting, but to be honest I’d rather be knitting than writing little updates and taking picture of the thing that I’m knitting. Also, while there’s something engaging and captivating about the long slog from the hem to the shoulder during the knitting, even though progress is always apparent, there’s not a lot to talk about for weeks and weeks while the piece grows.

I think it’s going to be a great sweater, and while it’s not the first thing I’ve really knit recently, I think it’s the first that I care to finish. There are a lot of great things about this sweater: the pattern is fun to knit, it’s the perfect size, it matches the cats, and it’s visually interesting without being busy. Also, I used Shetland yarn, and it’s really impossible to say enough about how much little things like that matter.

I’m not yet done with the sweater. As I draft this post, I’ve knit the body and the collar, and just have sleeves left to do. I knit sleeves from the top-down, and am three inches or so past the shoulder. The sleeves will be pretty straight forward and are just a matter of spending some time.

The problem with knitting sleeves from the top down is that you have to have a full sweater on your lap. In the summer this means overheating with a pound of wool on your lap or knitting at a table. Neither of which is terribly ideal. As a result I’ve started the next sweater. I’ll try and post something about the sweater as a whole when its done but in the mean time I’d like to collect a few thoughts and lessons learned so far.

I think the neck opening is a bit too wide. I have a crew-neck formula that I’ve been using for a while that might need to be tweaked.

Basically three inches before the shoulder, decrease every row on both sides of the neck for an inch and a half, decrease every other row for the last inch and a half. Create a steek over the back of the neck after the first inch and a half and decrease both the front and the back at the same rate. Start at the bottom of the front steek with half the number of stitches you’ll eventually decrease (from the front.)

In EPS terms, the total neck opening should consume a third of the total number of stitches and one sixth of the total number of stitches three inches before the shoulder seam. But divide everything in half for the front and back to get a usable number.
I think my gauge has changed noticeably in the last 4 years, but only the row gauge. I for one find this strange, though there’s not much I can do with it except deal with it.
This sweater has a number of turned hems, and I’ve realized a two important facts about knitting turned hems:
- The conventional instructions say: knit a facing, knit a round of Purl stitches to “turn the hem” and then knit on. (Can be done in reverse, depending on which direction your going.) Don’t. Knit two rounds of purl stitches. The turn is much more sharp.
- Knit the hem facing on two needle sizes smaller than the actual knitting. If you can knit the facing on 80-90% of the number of stitches as well (in some situations this isn’t feasible.)
- Knit the purl stitches with the smaller needle rather than switching to the smaller needle to knit the facing.
- Knit one more round before joining the hem in than you think you need.

If you’re knitting with Shetland and not planning to treat your steaks to secure them, steam the steek before cutting. Also, an extra couple of stitches wouldn’t hurt. Tragedy was averted, but it was closer than I’d like.

I’ll get more notes out (and perhaps elaborations of these points? If there’s interest.) after I finish.

New Sweater: Blue and Blue

I started a new sweater because the existing sweater was a bit to heavy and too warm. And I had yarn in the closet that was begging to be knit. This won’t be the first sweater I started knitting in August for this reason.

Also, I had the plan for the new sweater all developed and I wanted to get started: I’m an adult and I can do that.

The yarn is Shetland. Harrisville Designs “Midnight,” is the darker color, and I’m using some light blue-gray that I got from Webs a few years ago for the contrasting color. Probably the last great mill end from webs. The lighter color came in a 3 pound cone, and I’ve already made a sweater (a flop) out of the yarn and didn’t seem to make much of a dent.

The pattern itself is built around the same snowflake pattern that I’ve been using (this makes sweater number 4 with the same pattern,) but is the most reminiscent of the first sweater in the series, with some improvements for greater knitability. The effect, I hope, will be reminiscent of cables.

At the time of drafting, I’ve not yet joined the hem facing and the lower edge. I’ll probably post again about this one again after it starts looking like a sweater but before the long slog starts.

Onward and Upward!

Science Fiction Reading Progress

Mon, 15 Aug 2011 00:00:00 +0000

I’ve been mentioning what I’ve been reading as part of my weekly “accomplishment” posts, but I wanted to take the opportunity to write a slightly longer review and reflection of some recent reading.

After delaying for far too long, I’ve finally gotten through the April/May issue of Asimov’s. There are probably all sorts of reasons why double issues make a lot of sense for publishers, but I have to say that I find them a bit grueling to read. Maybe it was just this particular issue, but I found that the balance between novellas and short stories wasn’t terribly good, but maybe it was these stories, and my own tastes rather than anything wrong with the editing itself. There were some great stories: I loved Kristen Kathryn Rusch’s story, and I thought the cover story was fun but weird in that way that I don’t think Steampunk always works as well as it seems like it should. Michael Swanwick’s and Mike Resnick’s stories were high on the poignant-factor and low on the larger meaning but they worked.

I’ve still not read the June or August issues, and I’m going to try and start reading Clarkesworld. Here’s hoping I still have time to do other things after periodical reading obligations.

I also read (in about two weeks) Excision by Iain M. Banks. As I was moving to the east coast I made the decision to start going through all of the late 80s and 90s era space opera that I had totally missed, and I can’t quite recall why I chose Banks. I think there’s something about the grandiosity of The Culture that I really quite like. I found the first two really hard to grok, and now they mostly make sense. I think if I could do it again¹ I’d read Use of Weapons, maybe Excision, I’d make Player of Games optional but definitely 3rd if anything and then Consider Phlebas. I think Phlebas is among the best, but without the context of the others its a bit too odd.

I’ve also been listening to podcasts: The Outer Alliance Podcast, FLOSS Weekly, FaiF, and Escape Pod. Good content, great pacing for my now daily walks, and it’s good to stay in touch with all of that content. Podcasts were something I’d listened to a lot when I was exercising or driving alone. I’ve not driven very much in the last year, and my exercise routine has only recently started to become regular. So it’s nice to get back in that habit.

What are you all reading?²

And if I weren’t such an ardent traditionalist about reading series of books in the order of their publication. ↩︎
And, admittedly, listening to. ↩︎

Micro-Entrepreneurship, Good Enough, and Crowd Sourcing

Tue, 09 Aug 2011 00:00:00 +0000

I read this post by one of the partners in one of the coolest web services around, you should open that in a new window and then come back.

Back? Great!

Lars, proposes crowd-funding as a way to support free software development. Basically, run a “sponsor me to develop stuff” program, but rather than fund free software as a start-up around a single project or work for a big vendor.

It’s a nifty idea, and it’s got me thinking about micro-entrepreneurship. This would be where you make or do things, but not on a big scale. The businesses you create are small, and probably aren’t completely full-time equivalent, but in aggregate it’s good enough. While this is not the most prominent form of entrepreneurship on the internet, my sense is that it’s way bigger than most people think.

We’re too used to seeing multi-million dollar venture capital fund raising, IPOs, big acquisition deals, to realize the multitude of people who are making a few to several tens of thousands of dollars doing much smaller amounts of work.

I suppose I could write a whole post on good enough economics in the vein of this post on patronage from JamesGovernor but I’ll just leave a place holder link to a wiki page, in case someone else wants to fill things in.

Thoughts:

Service-businesses don’t scale particularly well, any individual work can only produce so much work, and it’s hard to make individuals any more productive. In light of that, large service-based firms are unlikely to form.
Most people have pretty specialized skills and abilities. Self-employment, particularly full time employment makes it difficult for people to spend most of their time doing what their best at. Specialization and differing skills is also what creates a market for service-based endeavors.
Lacking health care and other benefits of traditional employment, it’s hard for people to be more self-employed and less conventionally-employed. Given this, doing entrepreneurial projects on a smaller scale makes more sense.
Some kinds of entrepreneurial activities are attractive because, while they may not produce the same level of income as a salaried position, they allow more freedom and flexibility. This is the conventional justification for self-employment, and also the reason that most aligns with the “good enough” policy.

The problem with these kinds of “little businesses,” is that it’s too easy to focus on income earning work (e.g. freelance, and client work,) at the expense of doing basic work (e.g. developing core free software, doing basic research, writing fiction.) While the crowd sourcing notion makes a lot of sense, it requires a lot of faith in the crowd. I’m also unsure of how sustainable it is: while individuals can justify small amounts of money for such purposes, organizations cannot. Without organizational support, revenue is much lower, and it probably puts the larger financial burden on the smaller users, relatively speaking.

Lately Review

Fri, 05 Aug 2011 00:00:00 +0000

It’s Friday and I have a bunch of links, notes, and accomplishments to share.

First up, jfm and I have been continuing the discussion we had about task lists in a new discussion of the /posts/mobile-productivity-challenges. I’ve also imported some conversation from facebook (to a discourse page, since removed) following up on the :Cyborg Analysis and Technology Policy post that I made this last week. I’m really really proud of the extent to which the comments and edits that I’ve gotten have made my writing and thinking clearer on these subjects.

Also, a thanks to the people who have done things like fix links and correct stupid typos. Sorry to have caused the trouble, and I’m eternally grateful for the helping hand.

Next up, I wrote a tutorial for a reader who commented in the Make Emacs Better thread. The question addressed how to load optional functionality and “contributed” lisp code in emacs, and I wrote a little tutorial on how to load .el files in emacs. I think of this as a very basic and straightforward piece of customizing emacs; but it’s sufficiently complicated and counter-intuitive enough that I think a little bit of documentation is in order.

The above also marks the debut of a documentation section within the wiki, like the code section, that I hope to update every now and then as I write tutorials and reference material that I think someone may be able to use. No promises, and feel free stash content here as well. It’s all gravy.

Speaking of the code section, I wrote a little script that I use as dbl, that I describe in the Epistle Linker. Basically this little function goes through a directory and creates symbolic links to that directory in a specified directory and mangles the names of the file (prepends a few charters and changes the extension.) You an read the code, but it makes it possible to use a service like Dropbox without disrupting your local git setup and file organization. There’s a known issue with Dropbox that makes it slightly less than ideal, but what can you do.

When I was posting the epistle-linker, I realized that I had probably forgotten to mention the fact that I have this nifty little bit of glue that uses a procmail filter (you do use procmail, don’t you?) to deposit note to a particular email address (configurable) into an org-mode file for filtering. This is ideal for emailing your brain (i.e. org-mode) an item from your phone or tablet, say.

And finally: I have an external link. I think this follows nicely from the “how to work and ‘live’ in the mobile world.” Apparently ecl, an embeded Common Lisp implementation, has been built to run on Android and iOS. How awesome is that?

That’s all I have. I (finally) finished the April/May issue of Asimov’s, I subscribed to Clarkesworld, as if I needed more short fiction to read and distract me from everything else. been I’ve reading Iain M. Banks' Excision, which has been a great deal of fun.

Other than that. It’s been a pretty quiet couple of weeks.

Managing Emacs Configuration

Fri, 05 Aug 2011 00:00:00 +0000

This document outlines the use of emacs' require and provide functions to help new users understand how to better configure the text editor. While there are a number of different strategies for organizing emacs configuration files and lisp systems and there is no single dominant best practice, consider this document if you find your .emacs or [init.el]{.title-ref}` file growing out of control.

Background and Overview

After using emacs for any period of time, one begins to develop a rather extensive emacs configuration. Emacs comes with very little default configuration and large number of configuration possibilities. Because writers, programmers, and researchers of all persuasions and backgrounds use emacs for a larger array of tasks and work profiles, the need for customization is often quite high. n Rather than have a massive emacs configuration with thousands of lines, I’ve broken the configuration into a number of files that are easier to manage, easier to troubleshoot, and easier to make sense of. These files are then linked together and loaded using emacs' native require function. This document explains that organizational principal and provides the code needed to duplicate my configuration.

I store all of my emacs configuration in a folder that I will refer to as ~/emacs/, in actuality this is a sub-folder within a git repository that I use to store all of my configuration folders, and you should modify this location to suit your own needs. Additionally, I have the habit of prepending the characters tycho- to every function and emacs file name that are my own writing. This namespace trick helps keep my customization separate from emacs' own functions or the functions of loaded packages and prevents unintended consequences in most cases. You might want to consider a similar practice.

Configuring .emacs

My .emacs file is really a symbolic link to the ~/emacs/config/$HOSTNAME.el file. This allows the contents of .emacs to be in version control and if you have your emacs configuration on multiple machines to use the same basic configuration on multiple machines with whatever machine specific configuration you require. To create this symlink, issue the following command: :

ln -s ~/emacs/config/$HOSTNAME.el ~/.emacs

Make sure that all required files and directories exist. My .emacs file is, regardless of it’s actual location, is very minimal because the meat of the configuration is in ~/emacs/tycho-init.el. Take the following skeleton for ~/.emacs: :

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Startup and Behavior Controls
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(setq load-path (cons "~/emacs" load-path))

(setq custom-file "~/emacs/custom.el")
(add-to-list 'load-path "~/emacs/snippet/")
(add-to-list 'load-path "/usr/share/emacs/site-lisp/slime/")

(require 'tycho-display)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Machine Specific Configuration Section
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(tycho-font-medium)
(setq bookmark-default-file "~/garen/emacs/bookmarks/arendt"
  w3m-session-file "~/garen/emacs/bookmarks/w3m-session-arendt"
  bookmark-save-flag 1)

(if (file-directory-p "~/garen/emacs/backup")
(setq backup-directory-alist '(("." . "~/garen/emacs/backup")))
  (message "Directory does not exist: ~/garen/emacs/backup"))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
;; Load the real init
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(require 'tycho-init)

(menu-bar-mode -1)

The first seq defines the load path. Like other configuration paths, this is the directory that emacs will look for files to load when you use require later. load-path does not crawl a directory hierarchy, so if you store emacs lisp within ~/emacs/, you’ll need to add those directories here. To see the value of the load-path use “C-h v” in emacs. I then define “custom.el” as it’s own file to prevent customize from saving configuration in my init file. Then I use require to load a number of display-related functions (from the file ~/emacs/tycho-display.el,) including the tycho-font-medium function.

Then I have a number of machine-specific configuration opens set, mostly to keep multiple machines from overwriting state files.

Finally, I load the file with the real configuration with the (require 'tycho-init) sexp. The configuration is located in the ~/emacs/tycho-init.el file. The file closes with the (menu-bar-mode -1) sexp, which is the last part of the configuration to evaluate and ensures that there isn’t a menu-bar at all.

Require and Provide

require, however, does not simply load .el files in the load path. Rather, the file needs to be announced to emacs. Accomplish this with provide functions in the file. For ~/emacs/tycho-display.el the relevant parts are as follows: :

(provide 'tycho-display)

(defun tycho-font-medium ()
  (interactive)
  (setq default-frame-alist '((font-backend . "xft")
              (font . "Inconsolata-13")
              (vertical-scroll-bars . 0)
              (menu-bar-lines . 0)
              (tool-bar-lines . 0)
              (alpha 86 84)))
  (tool-bar-mode -1)
  (scroll-bar-mode -1))

(global-set-key (kbd "C-c f m") 'tychoish-font-medium)

(setq-default inhibit-startup-message 't
      initial-scratch-message 'nil
      save-place t
      scroll-bar-mode nil
      tool-bar-mode nil
      menu-bar-mode nil
      scroll-margin 0
      indent-tabs-mode nil
      flyspell-issue-message-flag 'nil
      size-indication-mode t
      scroll-conservatively 25
      scroll-preserve-screen-position 1
      cursor-in-non-selected-windows nil)

The provide call, identifies this file as the location of the tycho-display functionality. tycho-font-medium describes the font and display parameters that I called in the .emacs file. And the file ends with a keybiding to call that function and a number of default settings.

Init and Conclusion

While the tycho-init.el file holds all of the interesting configuration options, functions and settings, it’s mostly beyond the scope of this file. When you download contributed emacs.lisp files from emacswiki, put them in ~/emacs/ and put the require call in tycho-init.el. By convention provide names map to file names but be sure to check files to ensure that this is the case.

Using this setup as a framework, you can create--without confusion--a number of configuration files to properly collect and organize your settings, emacs modes, and other emacs code and functions that you’ve gotten from other users. Good luck!

You may also be interested in a couple other tutorials I’ve collected on emacs, notably:

Interfaces in Enterprise Software

Wed, 03 Aug 2011 00:00:00 +0000

This post is a continuation of my human solution to IT and IT policy issues series. This post discusses a couple of ideas about “enterprise” software, and its importance the kind of overall analysis of technology that this posts (and others on this site) engage in. In many ways this is a different angle on some of the same questions addressed in my “Caring about Java” post: boring technologies are important, if not outright interesting.

There are two likely truths about software that make sense upon reflection, but are a bit weird when you think about it:

The majority of software is used by a small minority of users. This includes software that’s written for and used by other software developers, infrastructure, and the applications which are written for “internal use.” This includes various database, CRM, administrative tools, and other portals and tools that enterprise uses.
Beautiful and intuitive interfaces are only worth constructing if your software has a large prospective userbase or if you’re writing software where a couple of competing products share a set of common features. Otherwise there’s no real point to designing a really swanky user interface.

I’m pretty sure that these theories hold up pretty well, and are reasonably logical. The following conclusions are, I think, particularly interesting:

People, even non-technical users, adjust to really horrible user interfaces that are non-intuitive all the time.
We think that graphical user interfaces are required for technological intelligibility, while the people who design software use GUIs as minimally as possible, and for the vast majority of software the user interface is the weakest point.

The obvious questions then, is: why don’t we trust non-technical users with command lines? Thoughts?

Create Better Task Items

Mon, 01 Aug 2011 00:00:00 +0000

I was paging through a list of things that I made for myself during a call I was in a few weeks ago, and was utterly dismayed by how useless the items were on the list. I wasn’t sure what needed to be done, I couldn’t remember what things meant, and I was left with the sinking suspicion that I had forgotten something crucial. I write this, in part, as a lesson to my past self on how to write good task list items.

Hopefully you’ll find it useful.

Task items must be actionable. You need to be able to read the subject or summary and know: what the project is, what kind of work it is, what needs to be done, and what very next thing you need to do is.

Tasks cannot be open ended. It’s really tempting to write tasks in the form of “work on a project” or “make progress on email backlog,” but don’t. How do you know if you’ve done the task? Is all progress the same? Is the actual work activity plainly obvious from an open ended task description?

Tasks need to concise. I’m a big fan of including some sort of status information and some sort of instruction and context with your tasks, but you need to be able to look at a task list and triage what to do next without thinking very much and without spending more than a few seconds deciphering messages from your past self. Write good summaries.

Try to organize your projects and tasks so that most of your task items are not dependent upon other items. Sometime dependencies are unavoidable, but I find if you’re clever, you can chop things up into parallel tasks that are easier to work on but that accomplish the same goal. In some cases, long strings of dependent tasks can be just as troublesome as large open tasks, because in the moment they amount to clutter.

Also your feedback and suggestions from your own experience may be of interest to all of us! I look forward to hearing from you!

Cyborg Analysis and Technology Policy

Wed, 27 Jul 2011 00:00:00 +0000

I want to put together a series of posts about cyborg perspectives on industrial IT practices. This post introduces that series.

The “cyborg analysis,” makes difficult to ignore that most problems with technology, even seemingly outright “technological problems” are better as understood as problems at the intersection of humans and technology. Ignoring either people or technology, leads to imperfect analysis. While, false divides between “problems with users,” and “problems with technology” aren’t helpful either, this might be a conceptually useful exercise.

I’ve been thinking that many of the seemingly technological problems of IT policy are really not technological problems at all, and are better thought of as “people problems.” I’m thinking of “problems” like file permissions, access control, group and user management, desktop management, and data organization. Obviously, some problems, like workflow and tool optimization, are seen human problems while others are often targeted as technological problems.

I think of human solutions as those responses to technology issues that address misunderstanding by providing training and education to users. There’s also a class of automations that make these human solutions easier to provide: for example, test harnesses of various sorts, code/text validation, and some template systems. Technological solutions, by contrast are those that, through the use of code, enforce best practices and business objectives: web filters, access management systems, and anything that are designed to “break” if business policy are not satisfied.

The divides between solutions are not natural or clear, just as problem domains in IT are rarely neatly or easily “silo’d.” There is often a fine line between writing code to automate a process to make users' work easier and writing code to control users and save users from themselves.

This post, and the series that follows it, are thus to explore this tension, and what I suspect are under-realized opportunities for human-solutions. I would like to think that problem domains in the larger IT world, particularly with thorny issues that require lots of code, may be better addressed with more human-centric solutions. Current thoughts revolve around:

The practice and method of software design.
The development and use of software developed for “internal use,” rather than as a software product.
Refraining the discussion about digital security to address it from a human angle rather than from a purely technological angle.

As always, I’d like to invite comments and discussion.

Mobile Productivity Challenges

Mon, 25 Jul 2011 00:00:00 +0000

I’ve never really figured out how to do work with anything less than a full computer. I’ve tried everything: Palm Pilots and Pocket PCs in the early 2000s, laptops, and eventually I just settled on just dragging a (smaller) laptop almost everywhere circa 2005. I get the feeling that, most people who have been thinking about mobile technology and productivity assume that the only impediments to mobile productivity are better hardware and software. Contemporary (multi)touch screen devices are the current embodiment of this theory.

I’m convinced that the theory is wrong.

Having better and more powerful technology doesn’t hurt anything. I’m a huge fan of the smaller, integrated, and more powerful devices. Software written specifically with mobile users in mind does improve the potential for productivity. These technological improvements, however, make the underlying problem more apparent.

The challenge of getting things done when mobile has little to do with the capabilities of the mobile platform, and more with the way people think about and plan work when mobile. Not only is this hugely frustrating to users, but technological capability that people can’t use threatens the ongoing development and adoption of new technology.

Using Mobile Technology More Effectively

The solution, here, I think is two-fold:

Fully Integrated Applications

Let’s develop integrated applications, not just integrated devices with different applications. Just as we didn’t need separate devices for every mobile function: telephony, music playing, book reading, mobile internet, and so forth. We don’t need different applications for every function: calendaring, messaging, email, contact management, notes, reading, and so forth.

At the very least applications need to be highly interoperable, so that users can send data between application functions easily, and synchronize data back to desktop and web portals seamlessly.

Task Planning Strategies for Mobile Productivity

I don’t think that the “user stories” for mobile technology are really fully developed, and as a result any interaction with a mobile device that isn’t responsive (i.e. there’s an alert of a new event, and people respond to it,) is either “twiddling nobs” (i.e. non productive,) or entertainment focused (i.e. playing music, video, or opening a book.) Perhaps that’s enough for some uses, but these this kind of workflow covers a small percentage of what people do with computers.

If mobile technology is going to replace a general purpose laptop, ever, even in limited situations, we need to figure out how to work in different ways. I know that I am loosing a great deal of time, when I’m using my phone switching between the notes app, the reader, the task list, and the calender. This task switching gets in the way of doing things to a much larger extent than similar behavior does when using a conventional computer. I would even posit that, the cost of context switching is inversely related to the size of the interface.

Better application integration will help this, but I think the real solution is providing a method for people to organize their mobile time more effectively. The task list that we build for ourselves when we’re doing “conventional” work (i.e. things that we need to remember to do, open projects, open issues,) aren’t particularly useful or usable when we’re looking at a tablet or a phone. If we don’t know what we ought to be doing, it doesn’t matter what the device or software is capable of in theory.

There are probably a dozen or more solutions to this problem, but here’s my first stab at it. What if there was a way to “forward tasks” to ourselves when we’re on the run, but have a few moments? We all loose time waiting in queues, or waiting for trains, and these seem like ideal phone times. If we had a way to queue things for ourselves, so we could spend the time doing something. Even better, would be software that would not only collect and display the queue but would also connect with the application where whatever needed to be done was and then record the results and send the back to our desktops when we were done.

Thoughts?

Links, Reviews, and Updates

Fri, 22 Jul 2011 00:00:00 +0000

While this week flew by in many respects and I only got a couple of posts out, there is much change and progress afoot. This post is an attempt to catalog some of the work I (and others) have been doing that hasn’t made it onto the blog:

Discussion of the “Better Task List” post by jfm`. Including spoilers for posts that I hope to have ready next week.
Further discussion of the make emacs better post. I’m thinking that it’s probably nearly time to split that into a few pages. There’s a lot of great content there and people have added a lot. I’m a huge fan.
Not a link, except to say that I did some fairly substantial tweaks of the site’s design, which is probably only worth mentioning because I suspect most people read the site on RSS. Different fonts in the headers, and I rearranged the masthead to be a little more clean, and changed the links a bit.
I’m in the slow process of cleaning up the Cyborg Institute site which I’ve neglected for far too long. I’m importing a lot of the content that I wrote over there, notably sygn and tubmle-manager. Next up, some straggling blog posts, and a clean up of the existing content to match my current projects and work.
The knitting posts, which is collected separately from rhizome posts is in full swing, and I hope to be able to post a few things there every now and then.
There’s now a real tag index and a tag cloud that looks like something. I’d avoided putting together a page like this for some time, because there were a lot of junk tags and enough really big tags that the cloud didn’t really work. I’ve mostly cleaned that up, leaving the wiki with a rather awesome tag cloud

I’ve also found a few things on the web that I think you might enjoy on the web:
A new blog called observatory. I’ve been talking to the author a bit. I realized that there aren’t very many blogs that are so verbose. I suppose ByteBaker is another example, but there aren’t many of them around.
undo-tree-mode is a nifty little emacs hack that makes undoing and redoing much less complicated and weird. (From that make emacs better discussion.) Though I have to admit that I no longer have a problem with the default behavior, even if I know it’s a bit counter intuitive.
I’ve been reading Strange Horizons more than I have in the past, thanks mostly to instapaer and InstaFetch for Android. I was particularly found of Genevieve Valentine’s column/review of a glorious mess of a movie trope.

That’s all I have for this week.

Comments Undo-tree also allows for undoing based on time (see here), apparently a feature that vim has.

Ideal Sweater

Tue, 19 Jul 2011 00:00:00 +0000

I’ve mentioned a few times that I’ve been doing more knitting recently. Nothing for the most part to get excited about. But, now that I have a bit more free time its became apparent that I can’t write all the time and it’s good to have something to do with my hands when other things require a bit of extra attention.

An explanation of my current knitting requires a bit of a back story. There always is and I think that is part of the joy.

The last sweater I started during college is probably the best one I’ve ever made. This isn’t to say that it’s the most impressive, or that it took me the longest, or was the most complicated, or was the most striking, or the had simplest pattern. No. If I had a dozen of this sweater, I’d wear them constantly all winter. It’s warm without being unbearably warm except during the coldest week of the year. The sleeves are big enough to support layering without weird bunching. The sweater fits me without being too tight or too baggy. It looks great over long sleeve t-shirts and oxfords. And it was a lot of fun to knit, both because of the yarn (Harrisvile Shetland,) and the pattern (a modified snowflake design of Swedish inspiration.)

A lot of the sweaters I knit after that were too complicated: I was trying to show off my knitting prowess, or I was experimenting with different yarns. In any case, I’ve come to the following conclusion:

Two color stranded patterns are the best kind of pattern. The pattern affects the shape of the knitting stitches (in a good way!) and the drape of the eventual fabric, in a way that provides a bit more structure. If you make the right design decisions for the pattern itself, the act of knitting becomes so much more captivating and rhythmic.
The best neck lines are basically crew necks, but are open to mid-chest like polo or henley shirts. I don’t tend to put button holes or clasps and just leave them open. The effect on the sweaters is that they are wearable over any kind of shirt (unlike v-neck sweaters) and are well ventilated (unlike unmodified crew neck sweaters.) I tend to leave the front corners of the collar squared and unmodified, but they can be rounded.
I’ve gone back and forth on this a bit, but now, I’m firmly of the opinion that turned hems are better on stranded garments than corrugated ribbing. Knit a hem facing, purl two rounds, and then start the pattern, and join the hem at the appropriate moment. Done. Corrugated ribbing is really appealing, but it’s not really ribbing, and it’s very loud, on finished garments.
After much experimentation with different shapes, I’ve decided that plain old drop-shouldered fisherman-style shaped sweaters are the only way to go. I’ve gotten pretty good at making saddle shoulders, set in sleeves, modified drop shoulders, and pretty much anything else. But at the end of the day I have a chest of funny looking shoulders that I don’t really wear.
Patterns need to draw the eyes up along vertical lines, which is incredibly flattering on most people’s bodies and is more fun to knit because panels of pattern can interact in cool ways up and down and across the sweater.I’ve been in a Swedish/Scandinavian kick for a while now, and have been working on variations on a ~26 row snowflake pattern.
Shetland yarn is really the best yarn in the world. It’s robust without being too itchy, and it doesn’t pill. I like Harrisvile’s selection because you can buy the yarn on cones, it’s consistent, and I think they have enough colors.

So I’m making more sweaters like this.

Better Task Lists

Mon, 18 Jul 2011 00:00:00 +0000

Just about everyone keeps a task list of some sort, or has at some point. To the casual observer, task list management might seems like a simple problem that could be augmented with a little bit of automation for great effect. Fire up your nearest “app store” and I would bet money that you’ll find a at least a few developers that have had the same thought.

For such a seemingly simple engineering problem there is an inordinate amount of really bad software. While this might tempt us to reassess the complexity of the task management problem, I don’t think this is really true. What happens, I’m convinced, is that people (i.e. cyborgs) make lists of tasks to solve different problems in their realities, and these different lists often require different automation. So while there are 10-20 basic task management applications, the number of distinct usage profiles exceeds that by several times. That’s the theory at any rate.

Archetypes

Allowing for large amount of diversity, there are still a few generally useful task list “archetypes” that we can use to characterize how people use task lists. I just want to enumerate them, here for now. I might move them out to another page, and you should feel free to edit the page (it’s a wiki!) if you think I’ve missed anything!

Collaboration Facilitation: Teams need systems to keep track of and work on shared issue queues. These are “bug trackers,” and are totally essential when items or “issues:” take a significant time to complete, require the effort/input/awareness of multiple people, and need to be created or added by a number of people.
Memory Enhancement: People create todo lists when they’re working more things than they can comfortably remember at any one time. The lists tend to be ephemeral, the items can be quickly resolved, but we make these lists so we don’t have to remember a long list of things while working. Think post-it notes.
Obligation Management: These lists bring us close to calendars, but we keep them to make sure we remember to do required tasks. Often these lists are helpful in helping people make sure that they’re “caught up,” so that they can enjoy and use free time without interruption or nagging.
Task Prioritization: When time is limited, a list of tasks is useful in organizing an order, and making use of available time, so that it’s possible to keep track of all open tasks, while also being able to allocate effort and time to tasks in a smart way that accounts for available time, importance, and deadlines. The goal is to get the most crucial things done while also never wondering what one could be doing with a few free moments.
Progress Tracking: These lists are less to track things todo and more to track have done. When working on a number of long term and short term projects, a list of what’s open, what’s been finished, along with a status of where things are is useful to avoid loosing track of projects and tasks.

Based on this, I think we can distill a number of overriding qualities of task lists from these five archetypal use cases. That working list is:

item granularity,
project-level organization,
scheduling and deadlines, and
use of priority markers. ## Personal Case Studies

When I had an epic commute my issue was that while I had free time, it was all in 20 or 40 minute segments on trains. The challenge was to be organized and focused enough to really use this time. It was early and while I was awake enough to get work done, I wasn’t always awake enough to figure out what needed my attention. And I didn’t have enough routine minutia or enough other free time to be able to spend these blocks of time doing minutia. I needed a task list that told me what to do and when to do it. I needed Task prioritization with a little obligation management or something close to that. I chopped every task into the smallest actionable items, which is annoying in creative projects and often not very useful, and then scheduled items out so that I could do 6 or so things a day, and anytime I opened the laptop there was a list of things I could jump into. It worked, more or less.

Now, I have free time. I even have a few hours strung together. The issue is less that I need help filling in every little moment with something to do, and more that I have too much that I could be working on, and I need help figuring out what the status is of ongoing projects are and where I ought to things are and what needs my attention the most. Progress tracking, more or less with a little task prioritization but in a very different way than I’d been doing. it. I’ve got a lot of things, and I need to be able to see where projects, and what needs attention now. I’ve not figured out the best solution, but I think less scheduling and bigger conceptual task objects is more the way to go.

Does this way of thinking about things make sense to other people?

Public Transit Information Overload: A Lesson

Tue, 12 Jul 2011 00:00:00 +0000

Philadelphia is replacing, or at least promising to replace, the trains that run the commuter rail system. The new trains are 35-40 years newer than the usual fair, and are replete with “new technologies,” one of which is an automated (I believe GPS-based) announcement system, which figures out what station is next, and which line you’re on. This is great in theory, but there’s a problem.

This system gives you too much information. Trains in Philly are named by their terminus, and all trains converge (and pass through) downtown. There’s history here which makes things a bit easier to understand if you’re a transit geek, but after every stop--including outlying stops--the train tells you what line you’re on, and which stops it makes or skips. The problems:

At most outlying stations, you can tell by the station you’re at, which line you’re on. It’s sometimes useful to know where the train you’re on is headed, but the trains only tell you that on the outside of the train, until you get downtown, when the announcements change from “where you’ve been,” to “where you’re going.”
The “this is the train you’re on,” announcements don’t change as you pass stops, so you hear where the train’s been at every stop after even you passed the relevant stops. The announcements make sense, as there are 5 or six “main line” stops that some trains stop on, and others don’t, so as you’re heading towards doubtful stops, it’s useful information, when you’re passed them: less so.
All announcements are displayed on screens in written form and read by a speech synthesizer. I understand the accessibility concerns, but there are still conductors and I’m not sure that the information is presented in a way that is usable by people who don’t already have a significant understanding of the transit system.

Given this background, as a technical writer, and someone who geeks out on information presentation, I felt that there are a number of things that can be learned from this case:

More information is sometimes confusing, and can make concepts harder to grasp.
Figuring out what people need to know in any given situation is more important (and more difficult) than figuring out what is true or correct.
Sometimes multi-modal presentation may not actually add value in proportion with the amount of annoyance it generates.
Presentation matters. The speech synthesizer does not sound very good and it’s inefficient.

Sweater Stories

Mon, 11 Jul 2011 00:00:00 +0000

I think the difference between writing technical documentation and knitting patterns is not terribly significant, and I’ve been known to talk at some length about this connection. While I’ve always been technically inclined, I started writing documentation after learning how to write knitting patterns. In the end, the things you have to know and do to write clear instructions is less about knowing about the technical underpinning, more about the mechanics of writing clear instructions and understanding process abstractly.

Shortly after I started my first technical writing job, I realized that all of the things I was learning about writing documentation was applicable to knitting patterns. In retrospect this makes a lot of sense to me: My interest in knitting is the process, not the design. At about this time, I came to terms with the fact that I liked to design boring knitting. Sweaters with plain shapes and fairly repetitive designs, look great and have a lot of appeal but I tend to make the same basic sweater with only minor variations, if any.

So I started writing what I hoped would be a series of essays about designs and sweaters that I’d knit. My hope was to get a collection of these pieces written and put together as a book, and I made some headway in this project I never finished one of these essays. I was writing essays that a knitter could read and be able to knit a sweater that looks like the one I knit, but also just enjoy the story and a collection anecdotes. Then, as life, singing, dancing, and writing got in my way knitting slowed and changed, and I was working on other writing projects, and for a thousand reasons it became a languished project.

And then my world calmed down, I started knitting again, and after beginning a new sweater (or two,) and I thought: it might be worth putting a little more time into the project. Which brings us mostly to the present.

The second piece of this is that, as a reader of knitting content (books, blogs, etc.) I’m most interested in learning about design decisions, creative inspirations, and how people’s knitting parallels and reflects life beyond the needles. When I look at a sweater I’ve made, I can instantly remember where I was when I was knitting it, and why I decided to attach the sleeve just so. The great thing about these stories is that unlike shelf space for finished sweaters, or potentially garish designs, they are endless and endlessly useful: as inspiration, as comfort, and sometimes just plain comedy.

I’ve not written about knitting on this site for quite a while, and I think that knitting is one of those subjects that might be better served by a little bit of segregation from my more regular fare. So this post is the first in a new knitting series that I’ve put together. Posts will still show up in the main index

I won’t be posting full descriptions of sweaters, I want to experiment with this writing apart from the blog and publication cycle for a while, but I think it’ll be nice to post notes on progress or on interesting little parts of sweaters. If nothing else a little blogging ought to make it easier for me to stay on track with knitting for the project.

Onward and Upward

Hyperlinks

Fri, 08 Jul 2011 00:00:00 +0000

Though short, this week has been pretty good. I’ve been doing cool things at work, I’ve been writing and posting blog entries, and fiction(!), I’m on top of email, and the sweater is growing. I hope this isn’t just a fluke and that I can keep this up and also expand slightly into doing a bit more reading. Small steps.

I’ve done a little bit of work on the wiki and site. Notably, selected entries are mirrored on Planet Emacsen. Also consider the following links to updates on the wiki and other sites:

Discussion of “/posts/make-emacs-better”, which has been incredibly productive and an interesting discussion of emacs adoption and use by non-programmer niches. I think this discussion and the original post connect pretty well with a post that made the rounds a few weeks ago: Lets Just Use Emacs
A link from my father on the history of computing. I replied to him with Code Quarterly Interview with Hal Abelson.
I’ve also collected a few LaTeX System links, of related projects, ideas and collaborators. I’ve also had a few conversations that I need to transcribe into the wiki. Watch this space.
Then there’s emacs-instapaper, which isn’t exactly a FaiF web service, but the functionality is great for the subway, and I like being able to keep Firefox closed more of the time.

That’s all for now!

Critical Practice

Thu, 07 Jul 2011 00:00:00 +0000

Being a critic is not simply looking for the points of failure, shortcomings, and breaking points in cultural artifacts (e.g. music, art, literature, software, technology, and so forth.) Criticism is a practice of comparison and rich analysis and a way of understanding cultural production. One might even call criticism a methodology, though “methodologizing” criticism does not give us anything particularly useful, nor does it make any practices or skills more concrete.

Criticism is really the only way that we can understand culture and cultural products. In short, criticism renders culture meaningful.

I wrote the above in response to this “On Being a Critic” post that a long time reader of this site wrote a while ago. Most of the differences between our approaches to criticism derives from technical versus non-technical understandings of critical practice. With that in mind, and in an effort to consolidate some thoughts about methodology, criticism, and theoretical practice, I’d like to provide two theses that define good critical practice, and provide some starting points for “getting it right:”

Criticism is comparative. If you analyze a single thing in isolation, this analysis is not criticism. By contrast, one of the best ways to make poor criticism more powerful is to include more information (data) to strengthen the comparison. Comparison should highlight or help explain the phenomena or objects you are critiquing, but should always serve agenda and goals of the criticism to avoid overloading readers with too much information.
Criticism ought to have its own agenda. It is impossible to avoid bias entirely, and from this impossibility springs criticism’s greatest strength: the power to productively examine and contribute to cultural discourses. While critical essays are perhaps the most identifiable form of criticism, there are others: novels, lectures, films, art, and perhaps even technology itself, can all be (and often are) critical practices in themselves.

Everything else is up for grabs.

Big Data Impact

Tue, 05 Jul 2011 00:00:00 +0000

I’ve been milling over this post about big data in the IT world for quite a while. It basically says that given large (and growing) data sets, companies that didn’t previously need data researchers suddenly need people to help them use “big data.” Everyone company is a data company. In effect we have an ironic counter example to the effect of automation on the need for labor.

These would be “data managers” have their work cut out for them. Data has value, sure, but unlike software which has value as long as someone knows how to use it,¹ poorly utilized data is just as good as no data. Data researchers need to be able to talk to developers and help figure out what data is worth collecting and what data isn’t. Organizations need someone to determine what data has real value, if only to solve a storage-related problem. Perhaps more importantly data managers would need to guide data usage both technically (in terms of algorithms, and software design) and in terms of being able to understand the abilities and shortfalls of data sets.

There’s a long history of IT specialist positions: database developers, systems administrators, quality assurance engineers, release engineering, and software testing. Typically we divide this between developers and operations folks, but even the development/operations division is something of a misnomer. There are merits to generalism and specialization, but as projects grow, specialization makes sense and data may just be another specialty in a long tradition of software development and IT organization.

Speicailization also makes a lot of sense in the context of data, where having a lot of unusable data adds no value and can potentially subtract value from an organization.

A Step Back

There are two very fundamental points that I’ve left undefined: what “data” am I talking about and what kinds of skills differentiate “data specialists” from other kinds of technicians.

What are big data?

Big data sets are, to my mind, large collections of data, GIS/map based information, “crowd sourced” information, and data that is automatically collected through the course of normal internet activity. Big data is enabled by increasingly powerful databases and the ubiquity of the computing power, which lets developers process data on large scales. For examples: the aggregate data from foursquare and other similar services, comprehensive records of user activity within websites and applications, service monitoring data and records, audit trails of activity on shared file systems, transaction data from credit cards and customers, tracking data from marketing campaigns.

With so much activity online, it’s easier for software developers and users (which is basically everyone, directly or otherwise) to create and collect a really large collection of data regarding otherwise trivial events. Mobile devices and linkable accounts (OpenID, and other single sign-on systems) simplify this process. The thought and hope is all this data equals value and in many circumstances it does. Sometimes, it probably just makes things more complicated.

Data Specialists

Obviously every programmer is a kind of “data specialist” and the last seven or eight years of the Internet has done everything to make every programmer a data specialist. What the Internet hasn’t done is give programers a sense of basic human factors knowledge, or a background in fundamental quantitative psychology and sociology. Software development groups need people who know what kinds of questions data can and cannot answer regardless of what kind or how much data is present.

Data managers, thus would be one of those posistions that sits between/with technical staff and business staff, and perhaps I’m partial to work in this kind of space, because this is very much my Chance. But there’s a lot of work in bridging this divide, and a great deal of value to be realized in this space. And it’s not like there’s a shortage of really bright people who know a lot about data and social science who would be a great asset to pretty much any development team.

Big Data Beyond Software Development

The part of this post that I’ve been struggling over for a long time is the mirror of what I’ve been talking about thus far. In short, do recent advancements in data processing and storage (NoSQL, Map Reduce, etc.) that have primarily transpired amonst startups, technology incubators, and other “Industry” sources have the potential to help acdemic research? Are there examples of academics using data collected from the usage habits of websites to draw conclusions about media interaction, reading habits, cultural particpation/formation? If nothing else are sociologists keeping up with “new/big data” developents? And perhaps most importantly, does the prospect of being able to access and process large and expansive datasets have any affect on the way social scientists work? Hopefully someone who knows more about this than I do will offer answers!

Thankfully there are a number of conventions that make it pretty easy for software designers to be able to write programs that people can use without needing to write extensive documentation. ↩︎

The Structured and Unstructured Data Challenge

Tue, 05 Jul 2011 00:00:00 +0000

The Debate

Computer programmers want data to be as structured as possible. If you don’t give users a lot of room to do unpredictable things, it’s easier to write software that does cool things. Users on the other hand, want (or think that they want) total control over data and the ability to do whatever they want.

The problem is they don’t. Most digital collateral, even the content stored in unstructured formats, is pretty structured. While people may want freedom, they don’t use it, and in many cases users go through a lot of effort to recreate structure within unstructured forms.

Definitions

Structured data are data that is stored and represented in a tabular form or as some sort of hierarchical tree that is easily parsed by computers. By contrast, unstructured data, are things like files that have data and where all of the content is organized manually in the file and written to durable storage manually.

The astute among you will recognize that there’s an intermediate category, where largely unstructured data is stored in a database. This happens a lot in content management systems, in mobile device applications, and in a lot of note taking and project management applications. There’s also a parallel semi-structured form, where people organize their writing, notes, content in a regular and structured manner even though the tools they’re using don’t require it. They’d probably argue that this was “best practice,” rather than “semi-structured” data, but it probably counts.

The Impact

The less structured content or data is the less computer programs are able to do with the data, and the more people have to work to make the data useful for them. So while we as users want freedom, that freedom doesn’t get us very far and we don’t really use it even when we have it. Relatedly, I think we could read the crux of the technological shift in Web 2.0 as a move toward more structured forms, and the “mash up” as the celebration of a new “structured data.”

The lines around “semi-structured” data are fuzzy. The trick is probably to figure out how to give people just enough freedom so that they don’t feel encumbered by the requirements of the form, but so much freedom that the software developers are unable to do really smart things behind the scene. That’s going to be difficult to figure out how to implement, and I think the general theme of this progress is “people can handle and developers should err on the side of stricture.”

Current Directions

Software like org-mode and twiki are attempts to leverage structure within unstructured forms, and although the buzz around enterprise content management (ECM) has started to die down, there is a huge collection of software that attempts to impose some sort of order on the chaos of unstructured documents and information. ECM falls short probably because it’s not structured enough: it mandates a small amount of structure (categories, some meta-data, perhaps validation and workflow,) which doesn’t provide significant benefit relative to the amount of time it takes to add content to these repositories.

There will be more applications that bridge the structure boundary, and begin to allow users to work with more structured data in a productive sort of way.

On a potentially orthogonal note, I’m working on cooking up a proposal for a LaTeX-based build system for non-technical document production that might demonstrate--at least hypothetically--how much structure can help people do awesome things with technology. I’m calling it “A LaTeX Build System.”

I’d love to hear what you think, either about this “structure question,” or about the LaTeX build system!

Are We Breaking Google?

Thu, 30 Jun 2011 00:00:00 +0000

Most of the sites I visit these days are: Wikipedia, Facebook, sites written by people I’ve known online since the late 1990s, people who I met online around 2004, and a few sites that I’ve learned about through real life connections, open source, and science fiction writing. That’s about it, it sounds like a lot, and it is, but the collection is pretty static.

As I was writing about my nascent list of technical writing links, I realized that while I’ve been harping on the idea of manually curated links and digital resources for for a single archives for a couple of years now, I’ve not really thought about the use or merits of manually curated links to the internet writ large.

After all you can find anything you need with Google. Right?

I mostly assumed that if I could get people to curate their own content, “browsing” would become more effective, and maybe Google technology could adapt to the evolving social practice?

Though the inner workings of Google are opaque, we know that Google understands the Web by following and indexing the links that we create between pages. If we don’t link, Google doesn’t learn. Worse, if we let software create all the links between pages, then Google starts to break.

Put another way: the real intelligence of Google’s index isn’t the speed and optimization of a huge amount of data--that’s a cumbersome engineering problem--but rather our intelligence derived from the links we make. As our linking patterns change, as all roads begin to lead back to Wikipedia, as everyone tries to “game” Google (at least a little) the pages that inevitably float to the top are pages that are built to be indexed the best.

And Google becomes a self-fulfilling prophecy because whatever page we do find out about and create links to are the links that we’ve found using Google. We’ve spent a lot time thinking about what happens if google becomes evil, to think about what happens to us as Google stops providing new and useful information. We’ve spent considerably less work considering what happens when Google becomes useless.

Make Emacs Better

Wed, 22 Jun 2011 00:00:00 +0000

I love emacs. I’m also aware that emacs is a really complex piece of software with staggering list of features and functionality. I’d love to see more people use emacs, but the start up and switch cost is nearly prohibitive. I do understand that getting through the “emacs learning curve” is part of what makes the emacs experience so good.

That said, there really ought to be a way to make it easier for people to start using emacs. Think of how much more productive some developers and writers would be if the initial experience of emacs was less overwhelming. And if emacs were easier to use, developers could use emacs as a core (embeded, even) component of text-editing applications, for instance, some sort of specific IDE built with emacs tools, or a documentation creation and editing toolkit built with emacs. I’d go for it, at least.

To my mind there are three major challenges for greater emacs usability. Some of these may be pretty easy to change non-intrusively, others less so. Feedback is, of course, welcome:

1. The biggest problem is that there’s no default configuration. While I appreciate that this provides a neutral substrate for people to customize emacs for themselves, you have to write lisp in order to do pretty much anything in emacs other than write lisp. And customize-mode is well unmentioned, but not particularly usable.

Perhaps one solution to this problem would be to create a facility within emacs to build “distributions,” that come configured for specific kinds of work. That way, emacs can continue to be the way it is, and specialized emacs can be provided and distributed with ease.

2. Improve the customize interface. I like the idea of customize, but I find it incredibly difficult to use and navigate, and end up setting all configuration values manually because that’s easier to keep track of and manage. I’d prefer an option where you configure your emacs instance the way you want (through some sort of conventional menu system), and then have the option of “dumping state” to an arbitrary file that makes a little more sense than the lisp structure that customize produces. Then, as needed, you could load these “state file(s),” But then I’ve never used the menu-bar at all, so perhaps I’m not the best person to design such a system.

This strikes me as a more medium term project, and would make it easier for people who want to modify various basic behaviors and settings. I don’t think that it would need to totally supplant customize, but it might make more sense.

3. Improve and add the ability to extend emacs beyond emacs-lisp. I initially thought emacs-lisp was a liability for emacs adoption and I don’t think this is uncommon, but I’ve since come to respect and understand the utility of emacs lisp. Having said that, I think offering some sort of interopperability between emacs and other languages and interperators, might be a good thing. Ideas like ParrotEmacs and using the Guile VM to run existing emacs-lisp in addition to other new code would be great.

This is a longer term project, of course, but definitely opens emacs up to more people with a much more moderate learning curve.

I’ve been working (slowly) on getting my base configuration into a presentable state that I can push it to a git repository for everyone to see and use, which (at least for me) might start to address problems one and two, but three is outside of the scope of my time and expertise. The truth is that emacs is so great and so close to being really usable for everyone, that a little bit of work on these, and potentially other, enhancements could go a long way toward making emacs better for everyone.

Who’s with me? Let’s talk!

The Death of Blogging

Fri, 17 Jun 2011 00:00:00 +0000

I think blogging died when two things happened:

1. A blog became a required component in constructing a digital identity, which happened around the time that largely-static personal websites started to disappear. Blogs always dealt in the construction of identities, but until 2004, or so, they were just one tool among many.

2. Having a blog became the best, most efficient way for people to sell things. Blogging became a tool for selling goods and services, often on the basis of the reputation of the writer

As these shifts occurred, blogs stopped being things that individual people had, and started being things that companies created to post updates, do out reach, and “do marketing.” At about the same time, traditional media figured out, at least in part, what makes content work online. The general public has become accustomed to reading content online. The end result is that blogs are advertising and sales vectors, and this makes them much less fun to read.

When blogging was just a thing people did, mostly because it let them present and interact with a group of writers better than they could otherwise, there was vitality: people were interested in reading other people’s blogs and comment threads. This vitality makes it more interesting to write blogs than pretty much any kind of content. The excitement of direct interaction with readers, the vitality of blogging transcends genre, from technical writing and documentation to fiction to news analysis and current events.

The vitality of blogging is what makes blogs so attractive to traditional media and to corporations for marketing purposes, so maybe you can’t have the good without the bad.

Everyone blogs. And perhaps that’s a bit of the problem: too much content means that it’s hard to have a two way conversation between blogs and bloggers. Who has time to read all those words anyway? Blogging is great in part because it’s so democratic: anyone can publish a blog. This isn’t without a dark side: we run the risk of blogging without audience, or without significant interaction with the audience, as a result of the volume of content which threatens the impact of that democracy. But it makes sense, New forms and media don’t solve the problem cultural participation and engagement, they just shift the focus a little bit.

Never Ending

Wed, 15 Jun 2011 00:00:00 +0000

I’m still not totally settled into my new routine, and I think that’s apparent in the blog. These things happen, and I just realized that this is the third summer in a row with some sort of major life change. Maybe I’ve forgotten how to exist in a summer routine. While I should probably give myself a break, I think it’s more realistic to accept a certain level of disruption as “the new normal,” and figure out how to develop a routine around that. That’s the hope at any rate. So, I’m getting there, slowly.

I’ve posted a number of new rhizomes in the last week. They are:

Security isn’t A Technological Problem, A post in my series about addressing problems in IT as human-issue, that need documentation and training rather than more software.
These Shoes Were Made for Cyborgs, which attempts to limit the potential for overly expansive theorizing of “the cyborg,” in a common but not overly productive manner.
Little Goals and Big Projects, a list of projects that I want to work on. Think mid-year resolutions meet five-year plan, meet time management review.

I’ve also done some maintenance (gardening?) on the wiki and added or edited the following pages:
I imported some comments from Facebook regarding my intellectual-practice post onto the discussion page. These comments are pretty valuable and I’ve found the conversation useful, hopefully you will too. Feel free to add your own comments there.
Similarly, I imported some comments onto the discussion for the Career Pathways post.
In response to one of the comments the Intellectual Practice post, I put together a pedagogy page, including some very rough descriptions of “writing classes I wish I’d taken and would love to teach.”
Not strictly tychoish related, but I revised my personal profile at tychogaren.com to be a bit more up to date and generally less weird/awkward.

Security Isn't a Technological Problem

Wed, 08 Jun 2011 00:00:00 +0000

Security, of technological resources, isn’t a technological problem. The security of technological resources and information is a problem with people.

There.

That’s not a very ground breaking conclusion, but I think that the effects of what this might mean for people doing security¹ may be more startling.

Beyond a basic standard of “writing and using quality software” and following sane administration practices, the way to resolve security issues is to fix the way people use and understand the implications of their use.

There are tools that help control user behavior to greater or lesser degrees. Things like permissions control, management, auditing, and encryption, but they’re just tools: they don’t solve the human problems and the policy/practice issues that are the core of best security practice. Teaching people how their technology works, what’s possible and what’s not possible, and finally how to control their own data and resources is the key to increasing and providing security services to everyone.

I think of this as the “free software solution,” because it draws on the strengths and methods of free software to shape and enhance people’s user experience and to improve the possible security of the public network as a whole. One of the things that has always drawn me to free software, and one of its least understood properties, deals with the power of source code to create an environment that facilitates education and inquiry. People who regularly use free software, I’d bet, have a better understanding of how technology works than people who don’t, and it’s not because free software users have to deal with less polished software (not terribly true), but has something to do with a different relationship between creators and users of software. I think it would be interesting to take this model and apply it to the “security problem.”

With luck, teaching more people to think about security processes will mean that users will generally understand:

how encryption works, and be more amenable to managing their own cryptography identities and infrastructure. (PGP and SSH)
how networking works on a basic level to be able to configure, set, and review network security. (LAN Administration, NetFilter)
how passwords are stored and used, and what makes strong passwords that are easy to remember and difficult to break.
how to control and consolidate identity systems to minimize social engineering vulnerabilities. (OpenID, OAuth, etc.)

There’s a lot of pretty basic knowledge that I think most people don’t have. At the same time, I think it’s safe to say that most of the difficult engineering questions have been solved regarding security, there’s a bunch of tooling and infrastructure on the part of various services that would make better security practices easier to maintain (i.e. better PGP support in mail clients). In the mean time….

Stay smart.

Security, being a process, rather than a product. Cite. ↩︎

These Shoes Were Made for Cyborgs

Mon, 06 Jun 2011 00:00:00 +0000

“Do eye glasses make us all cyborgs?” Someone asked me a few days ago.

I was annoyed more than anything.

Of course they do. Corrective lenses are a non-biological technology that shape our experience of the world and of our bodies. By this logic, pretty much every tool developed as a product of “technology” (applied science; otherwise known as tinkering with stuff,) renders us cyborgs.

I like the notion that cyborgism is the rule and not the exception in the course of human history, but it makes the conversation about the cyborg moment more banal. A more banal cyborg moment makes it harder to think about the parts that I think are most interesting: the internet, distributed collaboration, free software and open source, and the impact of technology on literature and reading/writing.

As a retort, I said something like, “Perhaps, but if you accept that eye glasses create cyborg beings, then you’d have to accept that shoes also create cyborgs. And the effects of shoes are much more interesting.”

Shoes affect how far people can walk, the speed of independent locomotion, they prevent all sorts of awful injuries, and probably lengthen the lifespan as a result. Shoes probably also change our feet and make us dependent upon wearing shoes, and more prone to certain kinds of injuries when barefoot. Fascinating stuff.

All other things being equal, I’m going to stick to internet and the cyborgs resulting from the encounter of humans and that technology.

Little Goals and Big Projects

Fri, 03 Jun 2011 00:00:00 +0000

Given all of the recent changes in my life, I found myself building and rebuilding a list of things that I’d like to “in the future,” sometime. The list is more than a set of mid-year “New Years Resolutions,” and it’s not quite as impressive as a “5 Year Plan.” As a blogger with firmly ingrained habits, I feel compelled to share this list with you:

Properly resume my knitting and spinning habit.
Read more veraciously. A book a month, plus the Asimov’s subscription with which I’ve done a bad job of keeping current. And that’s probably just a starting point. I have a backlog of things I want to read, there’s new stuff that I want to read coming out all the time, and I’m not going to get that done without attending to it.
I’d like to refresh/redo the cyborg institute site to reflect the reality of what I want to accomplish there, what I’m actually doing, and make it be updatable in a meaningful sort of way. I also want to get a number of project that I’ve been sitting on for a while together and published there. Those are:

The ikiwiki templates and configuration for tychoish.com.

A general purpose emacs “starter configuration,” based on my configuration. It would be fairly simple to generate, but I want to write a build script so that I can keep it updated as I tweak my own configuration. I would also need to write some documentation.

Similarly, I would like to produce a basic configuration for StumpWM that would make it easier to use Stump.

A couple of other specifications along the lines of the Sygn.

I would like to start dancing regularly with a Morris Dance team.
I would like to reestablish a thrice-weekly non-dance exercise regime that includes some weight training.
I want to apply to go to Clarion. But not before 2013, and preferably before 2016.
I’d like to try learning to play melodeon for Morris, and maybe other tune good playing times.
I’d like to go to Camp FaSoLa in 2012 or 2013. I’m also a little interested in maybe learning how to key sacred harp. Maybe I’ve been singing too much tenor recently and found myself as one of the not-new basses more than a few times that I’m getting a bit more confident.
I’d like to finish this damn novel sometime soon, revise the first bit into a passable short story, and begin shopping the story around.
I’d like to try writing short stories for a little while. But I also want to revise Knowing Mars, finish Station Keeping in some form, and potentially figure out a way to make the other open/lingering story on my plate become Knowing Mars, Part 2.
Write more regularly here. I think about a lot of things that I think are interesting, and writing ideas and asking questions is really the only way to make progress on scholarly, intellectual, and/or creative projects. I’m aware of a lot of potential in this work on tychoish.com, and I’d like to be able to make significant progress.

Methodology and Bootstrapping Intellectual Practice

Fri, 27 May 2011 00:00:00 +0000

Continuing from the discussion regarding intellectual practice, I’ve been talking with a number of people (my father in particular) about graduate school and the prospect of “bootstrapping” a scholarly practice using “new media,” like blogging, and wiki making. I want to explore both my thoughts graduate school and bootstrapping with new media, and as you’d expect both of these ideas are rather intertwined. My initial gloss follows:

Bootstrapping for Success

The “new media,” even 10 or more years on, is still quite new. The media shift and technological changes have had a pretty clear impact on economic and industry practices. At the same time, reading, participation, and writing are still in flux. People say, “oh look, blogging and wikis; we can use this as a teaching and learning tool!” and then there are classes, tools, and software to integrate blogging into courses and learning management systems, but the media itself is still in flux and I’m not sure that anyone has blogging and wikis (as an example) figured out.

While the changes in new media are important, the changes to education itself is probably more important. Educators of all kinds have begun to take this we begin to think about the was that traditional education has changed and will change. Given new media, a changing job market, and the shifting economics of education it’s hard to think that education isn’t changing.

I’m not sure it’s changing that much.

There are cases of successful auto-didacts, and people who’ve been able acheive success

I’d love to be wrong about this, but I’m pretty sure that the only people who blog/wiki and have found real success in fields are people with some other more conventional route to success: people who are already successful and figure out how to use new media, people who have conventional training or have achieved success in traditional media and then moved to using blogs:

Some examples: Cory Doctorow began publishing fiction conventionally and doing freelance work for Weird magazine, and became a blogger and used that to multiply existing success. John Scalzi (and Tobias Buckell) published non-fiction and had successful professional writing careers before beginning to blog and write and publish fiction. The Valve is a successful academic blog/publication/forum, but as near as I can tell all of the contributors have traditional literary training, and all/most have academic postings. Bitch, Ph.D. has/had a formal background.

Samuel Delany doesn’t have formal training but has had a scholarly career, and while his is an inspiring story there’s not much that’s reproducible from it given some historical constraints: he started publishing before the demise of SF pulp magazines and Ace Double, because creative writing hadn’t been established when he entered the academy, etc.

I’m certainly willing to believe that my sample is skewed, and that people have been able to move in the other direction (from online success to conventional success, or been able to bootstrap their own success online,) but I can’t think of a single anecdote. I’d love to be proved wrong here.

Disciplining and Formal Education

I think that working as a technical writer is something to which I am very suited, something that provides a great deal of value, gives me access to the kinds of people that I’m interested in talking with (software developers, admins.) And writing experience and skill is largely fungible, so the skill I’m honing and developing is very transferable.

So, while I’m not opposed to doing academic work eventually, I’m pretty sure that no matter what kind of industry work I end up doing (product management, community management/organization, training, etc.) I’ll sill basically be a technical writer. And here’s the thing, if graduate school has no effect on my career except dominating my time and earning potential for a few years? It becomes very difficult to justify.

The equation that keeps going through my head is: two job searches within a few years years¹ and a hundred thousand dollars or more,² for what amounts to a personal betterment project. It’s not getting any easier to justify.

Here’s the catch: I’m a decent writer and I’m getting better all the time can I write or help people write books, articles, essays, stories, and a whole host of more specific forms. I’m not really sure that I could write a quality academic paper without an unreasonable amount of effort. I don’t know the process, I don’t know how to start, which literature to look at for resources, or for models, I’m not sure where the line between concision and complexity is in academic prose, and so forth. That’s the kind of knowledge that I’m certain I could get out of graduate education. And perhaps I’ve been a technical writer for too long, but I think not being able to “write like a scholar” makes it hard to participate in scholarly discussions.

The Remains of the Practice

I’m not sure where this leaves me. I’m thinking about seeing if I can take a seminar and a methods class at CUNY in the next year I might be able to get what I need. The right collaborative project might be a good way to build the required skills, but that’s even more complicated. As far as using the blog/wiki to build and participate in a conversation about new media practices, collaboration, and digital labor practices… there is much work left to be done.

i.e. getting into graduate school, getting a post-graduate school job. ↩︎
the 100k number is mostly opportunity costs, and assumes a funded/cheap 2 year masters program. ↩︎

An Intellectual Practice

Wed, 25 May 2011 00:00:00 +0000

What I want, it seems to me, isn’t a career--I have one of those--but to sustain intellectual life and practice. I would like to be able to ask questions, read seriously, participate in important conversations, and to write about this work and practice effectively for an audience that is invested in these discussions. This post is a follow up to my “career pathways” post.

I have a blog and wiki, I can read, and my writing continues to improve. How hard can it be to achieve these goals and establish this practice on my own? Famous last words.

The thing is, I hate auto-didacticism as an approach to knowledge production and learning. Sure it works, sometimes, and professionally I think I’ve been able to succeed on the basis of being able to learn things on my own. At the same time, self teaching at more advanced levels, and avoiding formal study feels like a mechanism for people to use to avoid challenging themselves or their assumptions about the world. The challenge here, in addition to discipline (in a number of senses of the word,) is to avoid scholarly isolationism.

Conversely, it might be true that sufficiently advanced study is always already self-lead and self-taught anyway. That’s not a conjecture I have the experience or specialty to comment upon, but it’s a possibility.

In any case, my success at being able to do meaningful and fulfilling work, hinges upon:

being able to write and interact effectively for your communications medium. In my case this means, use blogging and wikis well.
being able to maintain an active presence and participation in the discussions and work you want to do. This means posting regularly, in addition to writing, reading, and thinking about various projects. Work needs to be sustained and ongoing.
being able to make leisure time sacrifices to support the work. There’s only so much time in the day, and I think it’s also important to manage expectations somewhat in recognition of this fact.
being able to find or establish and interact with a community of peers. Regardless of interest or focus, it’s important to find colleagues who do work that is enough like yours to allow them to grasp the intricacies of your work and different enough to infuse the conversation with useful context and ideological breadth.

At least, that’s my hope. What am I forgetting?

Inevitable Returns

Mon, 23 May 2011 00:00:00 +0000

I started writing this post on Thursday, which was my actual birthday, to write a post blathering about the things I was working and about routines and forming new habits, and some changes that I’ve made to the site. And then I got swept into work and doing things, and the writing just never happened. Friday and the weekend were filled with family time, dancing, and my goal for this comparatively quiet Sunday afternoon is not so much to get caught up on various projects, but to get a little bit done to jump start my momentum for the week.

The biggest development that I’ve made last week, during that hiatus, is that I merged the “essay” and the “rhizome” section of the site. Everything’s a rhizome, though if a post is seeming particularly “essay”-like the essay page will sill pull those out. This seems to be the best technological solution and it solves the logical overhead of needing to maintain two sites. Maybe other people can deal with maintaining more than one site or blog, but I really can’t deal with. This is one of those things that seems like a good idea every couple of years, and then I give up and merge everything back together.

I also wrote up a project spec called A LaTeX Build System, which describes (very roughly) a notional piece of free-software infrastructure that would make LaTeX easier to use in and for itself but also designed in such a way as to make LaTeX based systems preferable for all sorts of publishing operations. Read the page for more info, but it’s basically a way to sand offf all the rough edges of LaTeX so that everyone who makes documents (that’s most people) can make beautiful consistent documents easier than with any conventional method.

I finished reading Player Of Games, last week. It’s another one of Iain M. Banks' “Culture” novels, which I like. They’re frustrating because they all (so far) have a lot of plot that circles around itself endlessly, and seems really important but you know that anything that you might find out in the plot going to has already happened in the set up. The result is this an ironically claustrophobic novel feels like a really drawn out world building experience. While the experience works, it doesn’t feel like it ought to to work. And there you are.

Speaking of reading, I finished reading the book above on my new phone which is quite nice. I’m not sold on the Kindle Mobile app for reading short fiction periodicals, as it doesn’t save/sync pages, and I find it hard to read an entire novella in a single sitting. I’ve started paying for Readability, which is a great tool for bookmarking, reading and archiving articles and other medium-to-long form pieces on the web. I’ve started paying, because I think they’re doing something really cool that I really want to succeed, and I like being able to use it as a way of getting content to my phone for reading. I’m a little frustrated that there’s no good way to load up the phone with articles for reading while on the subway. Get on that, ye horde of mobile developers!

I’ve started knitting again. Just reached the bottom of arm holes (armscye for the pedantic) for a new sweater that I’ve been working on (or ignoring more likely) for a few months. That’s exciting, and it’s nice to get a few rows done most days. I’m not obsessive (much) about the knitting, and certainly not in the way that I have been in the past, but it’s a nice thing to do and a good change of pace when I get tired of looking at screens. I’ve long toyed with the idea of writing knitting stories something sort of between an essay and a knitting pattern and if nothing else I think doing some of that writing will require a regular knitting practice. Add that to the list.

Speaking of lists, I ought to work on making some progress on my list! With luck I’ll be around a bit more this week!

Packaging Technology Creates Value

Tue, 10 May 2011 00:00:00 +0000

By eliminating the artificial scarcity of software, open source software forces businesses and technology developers to think differently about their business models. There are a few ways that people have traditionally built businesses around open free and open source software. There are pros and cons to every business model, but to review the basic ideas are:

Using open source software as a core and building a thin layer of proprietary technology on top of the open source core. Sometimes this works well enough (e.g. SugarCRM, OS X,) and sometimes this doesn’t seem to work as well (e.g. MySQL, etc.)
Selling services around open source software. This includes support contracts, training services, and infrastructure provisioning. Enterprises and other organizations and projects need expertise to make technology work, and the fact that open source doesn’t bundle licensing fees with support contracts doesn’t make the support (and other services) less useful or needed for open source.
Custom development services. Often open source projects provide a pretty framework for a technology, but require some level of customization to fit the needs and requirements of the “business case.” The work can be a bit uneven, as with all consulting, but the need a service are both quit real. While the custom code may end up back in the upstream, sometimes this doesn’t quite happen for a number of reasons. Custom development obviously overlaps with service and thin-proprietarization, but is distinct: it’s not a it doesn’t revolve around selling proprietary software, and it doesn’t involve user support or systems administration. These distinctions can get blurry in some cases.

In truth, when you consider how proprietary software actually convey value, it’s really the same basic idea as the three models above. There’s just this minor mystification around software licenses, but other than that, the business of selling software and services around software doesn’t vary that much.

James Governor of Red Monk suggests a fourth option: Packaging technology.

The packaging model is likely just an extension of the “services” model, but it draws attention to the ways that companies can create real value not just by providing services and not just by providing a layer of customization, but by spending time attending to the whole experience, rather than the base technology. It also draws some attention to the notion that reputation matters.

I suppose it makes sense: when businesses (and end users) pay for proprietary software, while the exchange is nominally “money” for “license” usage rights, in reality there are services and other sources of value. Thus it is incumbent upon open source developers and users to find all of the real sources of value that can be conveyed in the exchange of money for software, and find ways to support themselves and the software. How hard can it be?

New Feeds, Habits, and Jobs

Sat, 07 May 2011 00:00:00 +0000

(I’ve been tinkering on this post all weekend, and I wanted to get it out of the door before it’s next week. Here goes!)

I always forget, and there’s no good reason for this, how difficult it is to establish new routines and new habits. Two weeks ago, I moved for the second time this year. this week I started a new job and even though I have more free time than I did before, I’m still coming up at loose ends and I find myself wondering why I have a hard time concentrating and getting into “the grove.” There’s so much to do, so many tasks collecting dust on my todo list, and I’m only half keeping ahead of everything.

I have two things to report that I missed on the last update: I have new full-text feeds for posts about org-mode and emacs. Hopefully these will get included in relevant planets soon for your reading pleasure.

There’s also been some :discussion on the “Bad Org Mode Habits” post. You may be interested.

As an aside: the astute among you will notice that Matt Lundin and I have made a folk page that is automatically updated anytime there’s a page that links to or is tagged with Matt’s handle (i.e. “madalu.") This includes an RSS feed that he (or you) can use to track his updates and mentions. Use the edit page functionality to see how to make such a page for your own notification purposes.

Everything else on my list is pretty boring. I’m, slowly trying to follow my own advice in bad org mode habits.

The organization I was using for my lists and notes worked really well when I was commuting all of the time and working off of laptops on the go. Among other limitations, I think I basically had to give up any sort of really complex project. Now that I have more time, I can tend to more gnarly projects that I’ve wanted to tinker with that I just haven’t had the time for. Without a train ride and “home time” to define my free time for fiction writing and other projects, it’s been hard to adjust.

It has also been hard for me to get a real sense of how my free time remains limited (because that’s the nature of free time,) even if there’s a lot more of it to go around. Adjustment is always hard and changes, particularly big changes, have a ripple effect. Things I’ve been doing differently include:

I’ve made some big changes to the blog post writing tooling, so that new blog posts are written in my org-mode files rather than in their own directory. (I updated the above emacs code with some shell functions that make the publication process easier (if you’re using that code.) This seems minor, but is pretty big in terms of how I’m using org I’ve never really used org for anything other than notes and one off projects. It’s a good shift.
While I used to dock my laptop to the desk and use it with an external monitor, I’m switching to just using the laptop dock and working on the laptop on the desk. This might not be ergonomically ideal, but it feels better and is a bit more coherent. Particularly with the addition of a third laptop for work.
I caved and installed emacs on my work laptop (Windows.) Rather than adapt all of my emacs crap to work with Windows, I’m basically copying and pasting the important parts, and starting from scratch. It’s not pretty, but it works. And being able to use emacs and do the things that I want to do there, is a good thing indeed.
With only a few thousand more words to go on the novel. I’m taking a bit of a break to rethink things, and hopefully this afternoon rewrite a few outlines so I have a good way of drawing this project to a close. Then writing, then lots of editing and lots of other writing.
A bit more than two weeks ago, I got a new cell phone. It’s a HTC Inspire (ATT “4g,") and I like it rather a lot. I still think that Blackberry does email and messaging better. This is a better computer to have in your pocket. The Kindle App is really usable. I have a text editor/note program that works great, and all the other little incidentals just seem to work and be there. If only the messaging where a bit better. eh.
The new job is going well, though I’m still in the “I wonder what this will look like when I’m actually fully up to speed” phase. I expect that I’ll write even less about this job than my last job, and retreat further into “tycho.” I like this. I may, however, write some features of the new job: the fact that I’m using Windows on my work machine and various aspects of digital collaboration, which I still find fascinating.

That’s all the news that’s fit to print!

Career Pathways

Tue, 03 May 2011 00:00:00 +0000

I always thought that I would go to graduate school sometime in my twenties. I ask questions that are pretty geeky and difficult to answer, I think learning and research are pretty important, and I want to talk to people about ideas, projects, and theories. There are skills I need to be able to address the questions I have and background literates that I wish I were way more familiar with.

It seems like a good fit. Right?

Right. Well, right only if we accept that graduate school is a mechanism for personal betterment. While that has to be part of it, mostly graduate school is a job and the first in a long line of possible jobs. The academic career path has merits and demerits, but it’s still work, and I think to ignore this, makes it possible to accept atrocious labor practices in the academic world.

Somehow, without much intentionally on my part, I’ve found a career that I enjoy. Even more curious is the fact that being a professional writer with some technical background is the kind of thing that enjoys a certain kind of perpetual demand. And better yet, it’s impossible to get a degree to support this career: as near as I can tell literature degrees, history degrees, theology degrees, theater degrees, and psychology degrees are all equally relevant and irrelevant.

While I’m not convinced that I never want to teach, if the “getting a job” portion of going to graduate school is somewhat moot, then I’m left with a couple of questions:

If additional schooling doesn’t affect career options and possibilities, then does it make sense to spend significant time in pursuit of an advanced degree?
How do I develop and maintain an intellectual and scholarly practice without graduate school?
How do I prevent my career from stagnating and from getting stuck in less than ideal jobs in mid-career and late career stages?
I work in field where the need for human labor is constantly (and ideally) being automated away. The conventional wisdom is “develop specialties, but don’t get too cemented in a particular function so that you have options for after your job gets replaced. Combined with the orthogonal issue that writing and the work of writers is horribly misunderstood by just about everyone, figuring out “career paths is not necessarily easy. How do I deal with this long term concern in a more manageable way while being mindful of the future concerns.

While I sometimes feel like this blog can stray into the “overly meta”, I think that prefer intentionally over aimless wandering. Indeed, I think this career issue might have been a great deal easier for me had I figured some of these things out earlier. I know that we don’t always find clear and definitive answers to these problems and that solutions come in pieces and very slowly.

This series is about thinking about these issues to increase the possibility of intentionally and to document my process so that people can provide feedback. With luck, this will also help form a model for people who want to think about ways of contributing to scholarly conversations and grow intellectually, but needn’t do that in the context of the academic training and labor market.

I look forward to hearing from you and working with you all!

Moving the Furinture

Mon, 02 May 2011 00:00:00 +0000

A few weeks ago we moved the furniture around. The biggest objective was to get some desk space setup in a more usable fashion. I find myself doing this a couple times a year. It’s easy for me to get in the habit of having everything a certain way without realizing that the chair I like to sit on is always in the sun during the mornings when I like to write, or that I never put the dishes away, because the cabinets were arranged backwards. The same sort of thing happens with computers and the way we use technology.

Matt Lundin and I were talking about this a couple of weeks ago. I think at the core of the issue is that the kind of work we do with computers, changes subtly: either what we do actually changes as a result of our progress, or we learn more about what we’re doing and what we ought to be doing from completing work. Like the furniture, it’s easy to get accustomed to doing things in one particular way and not realize that there’s a better way.

It’s useful then to spend a little time every now and then to do a little bit of digital spring cleaning. There are two major parts of this process:

3. Figure out if “what you do” has changed significantly. As our projects grow the work we do changes, sometimes in ways that make it hard to continue to organize our efforts and tools in the same way.

4. Looking for new tools that you can use to do what you do. This includes flat out new tools of which you weren’t aware, little automations that you may be able to build, and tools that solve problem domains that you’d previously ignored. It’s also sometimes useful to look at your tools and figure out what is more trouble than it’s worth. Ideally these don’t all change annually, but it’s worth doing a review.

There’s no guarantee, of course, that you’ll find a solution to any issue that you come across but you might. You might also be able to gain new insights into issues that have nagged at you by approaching a number of issues and pain points all at once. It’s also, I’m pretty sure, more productive to spend most of your time doing stuff (writing, coding, etc.) even if your world isn’t perfectly optimized, than it is to spend all your time tinkering with things.

And the sofa really does look better over there.

Onward and Upward!

The Week that Was

Fri, 29 Apr 2011 00:00:00 +0000

This has been a strange week. It’s strange, but not terribly surprising, that given a comparative expanse of free time with not a lot to do except write, that every little thing I’ve meant to do for a while but have been putting off suddenly came up and required all of my attention.

This isn’t to say that I’ve spent all of my time doing chores and running errands. I’m actually taking this as proof in favor of my general theory that writing and other avocational pursuits ought to be habitual and ongoing rather than the focus of occasional binges. But there are some things that take a lot of time and its worth binging on a little. This week, I got some writing done, but mostly did things like tinkering with my computers and tending to some long overdue systems administration tasks. I also switched to a new cell phone, oversaw delivery of furniture, sang, danced, and cooked. Not a bad week.

I do have some links and notes for you.

I made a rough introductory page for my efforts to run the xen hypervisor on a laptop.
There was some confusion this week as I reorganized the blog post files on the wiki. Basically, all blog posts are now in /posts/ directory (no more /posts/ and /posts/) with the different feeds generated by tags. So I did post some cool stuff, but it didn’t get out. This includes:
Bad Org-Mode Habits about the best ways to use emacs' org-mode.
Poetry Has a Purpose which addresses a little bit of literary theory/criticism/practice that got under my skin and into my head.
I’ve imported a few recent discussions from Facebook threads associated with my posts here onto discussion pages pages, which I think you might enjoy:
I spent some time this morning working on a system to capture notes into org-mode using procmail and my cellphone’s email client. See that here.
Most of this post, excluding the links, was written on the aforementioned cell phone. It’s still not quite the same as writing from a computer, but for getting raw material out of my head, I kind of like it. We’ll see if that sticks.

That’s about all I have for now!

The new job starts next week, so I would say that my posting schedule may be a bit erratic. But if you’ve been reading tychoish for any length of time, you’re probably used to a far more irregular posting pattern.

I’ll see you on the flip side.

Poetry Has a Purpose

Thu, 28 Apr 2011 00:00:00 +0000

I was reading an Ian M. Banks novel, `The Use of Weapons <>`_, and there’s a scene where the main character reports on having spent a few years “trying to become a poet.” And we get this very idyllic tale of him surrounding himself with beauty and simplicity. It didn’t work for the character and I suspect it wouldn’t work for you.

This reminds me of a post I wrote some time ago that made the argument that poems are made of words rather than images or some sort of spiritually inspired emotional state. In short, poetry is about form, structure, and the conveyance of meaning at the level of the word and that “transcendence” doesn’t really play into the craft or purpose of poetry. Or at least, poetry is not exceptional among literary forms in this regard. I think that this is basically true, and is probably a good way of approaching creative writing and texts in general. Maybe this is just a tychoish thing.

Here’s a corollary, and I don’t think that it’s too contradictory even though it nearly sounds it:

Poetry, and texts in general, exist in the real world and require a purpose to succeed. Poets don’t just spend a lot of time “living like a poet” and are then write poems that grow from these experiences. Writing well requires some skills and some basic training, but beyond that foundation writing needs a purpose or a goal. It’s easy to see how this might be true in prose forms, but I think the exact same is true of poetry. Perhaps more so.

The character in the novel was attempting to achieve some sort of aesthetic, working from an idea of “how poets should work.” While it can be hard to learn what the poet’s ulterior purpose is, that is a different issue. The most important thing is that you have something to say, regardless of the form.

Hear Hear!

Bad Org-Mode Habits

Tue, 26 Apr 2011 00:00:00 +0000

Org-mode is great. Org is this super-task management package that merges outlining and structured document editing abilities with task management glue. It sounds like a weird combination at first, when you realize that it means that you can effectively do work and organize your work in the same set of files without needing to “switch modes,” between a planning/task list interface and a “doing things” interface, the effects are amazing.

I’ll leave “the awesomeness org-mode” to other people and other posts, and spend time here focusing on why “org-mode is awesome but won’t do your work for you,” and “if you’re new to org-mode you will probably want to do things in a certain way, but don’t” issues. Org-mode is great and it can be even better if you avoid developing a few bad habits.

Use `org-capture <http://orgmode.org/manual/Capture.html>`_ as much as possible. Org-capture is a quick interface within emacs that lets you open a temporary buffer, take a few notes, and save the notes into a file. It has advanced features for more complex data entry features and data types. The temptation is to use it as a “quick entry” tool for task list items, but don’t just use it to capture new tasks. Capture links and bookmarks, store notes and important information, If org-mode is your outboard emacs-based brain, then org-capture is it’s main input device. Fighting it, means that your org files end up being less useful.

Avoid using org-mode as a simple task list, and particularly avoid constructing the content in your org files to “game” your agenda view. The agenda compiles a working task list for your actionable notes, working backwards from this means your notes are less than useful things can get lost and all of the really cool things that org lets you do aren’t accessible to you.

This may just be the application of a good general information management practice, but: distribute information evenly throughout all levels of the organizational hierarchy of the file. Use the org-narrow-to-subtree function to focus your current work on a specific portion of a file, and don’t bury information or have it sitting around in one big unorganized pile.

Hope this helps!

Flaws with SSH

Mon, 25 Apr 2011 00:00:00 +0000

In response to 9 Awesome SSH Tricks some posted the following quote (on the old commenting system):

The workarounds have become smoother and some of the things we can do with networks of Unix machines are pretty impressive, but when ssh is the foundation of your security architecture, you know things aren’t working as they should.

-- Rob Pike, 2004

So let’s clarify things a bit. SSH is great as an end user protocol, and great for dealing with the realities of our distributed computing environment in an exigent manner, SSH lets us:

connect securely to remote systems.
quickly establish tunnels through remote machines.
admister remote systems securely.
provide end-users with key-based authentication.

SSH is great for providing end users with a secure way of interacting with computer systems in networked environment. It’s not, however, the magic bullet for security policy. If you or your organizations security practices revolve entirely around SSH tunnels, then you’re probably in trouble or about to be in trouble. Use traditional VPNs and TLS/SSL when it makes sense and develop a sane security policy.

But don’t forget SSH and if you do use SSH, know that there are some really awesome things that OpenSSH makes possible.

Undercover

Thu, 21 Apr 2011 00:00:00 +0000

This is a post drafted a couple of weeks ago, while I was still doing the commute.

I wonder sometimes, what the people on the trains I take actually do. While there’s some variation on who’s aboard, there’s a very common and consistent cast of characters on each leg of my journey. God only knows what they’re doing.

One fellow, I’ve spent the winter thinking of as “the short dorky one.” He’s white and pretty pale, has darkish-blond with hair that’s seems too long, under 5'6," and wears glasses that are several years behind current styles, his teeth are noticibly in a poor condition. I should also point out that my commute is largely “pre-rush hour,” and that I see this fellow around 6:20 in the morning.

Today, he wasn’t wearing the oversized and a bit more than “slightly worn” jacket and I noticed two things: first he was muscular in a way that suggested he put some time into the appearance and, two his entire right arm was covered by a tattoo. I would have never pegged him as the type, and none of the fantastic stories I would have thought to tell myself about him included either of these details.

Most of the time, my primary reaction to my fellow commuters is annoyance when they won’t stop talking or using their cell phones. Sometimes, I’m just confounded.

Welcome to life.

Changes and Resetting

Wed, 20 Apr 2011 00:00:00 +0000

A week ago, I quit my job.

For a number of reasons this shouldn’t come as a huge surprise: I’d been burning out on a 2-hour each way commute from Philadelphia where I live(d), nominally. The boyfriend in New York City, didn’t make things easier. And while, I really liked a lot of things about my job, I knew that it was getting to be time to move on. My last day was Friday.

I spent the intervening time packing all of the stuff that’s going to New York into my car, and driving it to New York. For the rest of the week, I get to donate a bunch of my stuff that I’ve been dragging from place to place for the last 2 plus years. Add in some car maintenance, a few dinners and visits with friends, and a visit with my mother. She has awesomely agreed to drive said car back to the Midwest where it will be infinitely more useful. And I’m trying to get some writing done, and readjust my brain and body to a different work/life schedule.

Next week, there is settling in, more adjustment, waiting for deliveries, and more writing. The adjustment thing is tough: no matter how late I manage to stay up I wake up feeling rested and like I’ve slept in luxuriously. And then I look at the clock and see that it’s 6am. Time to get up and start writing, I guess.

The new job, is in many ways much like the old one. Different technology, different customers, different kind of writing, different sorts of problems, but I’m really excited. When I started my last job, I was not a very confident writer, and the experience of writing professionally for the last two years has increased my confidence as a writer and the strength of my identity as such.

I think being forced to not only write something regularly for a couple of years, but also be responsible for maintaining and improving that text can be very eye opening. If nothing else, I’ve become very aware of how my writing has changed and improved.¹ While it’s still occasionally surprising that other people want to pay me to write things for them, I’ve gotten better at recognizing the tangible value of my work and abilities. Somewhere along the line, I stumbled into one of these “career” things and it suits me. Who would have guessed?

I don’t tend to write very much about what I do for my day job for a number of reasons, chief among them the fact that I want this blog to be my own. Having said that, what I’m doing at work undeniably influences and shapes my thinking and writing here, and I don’t expect this to change. In the mean time, this whole “changing jobs and moving to New York,” moment in my life has a pretty significant impact in how I think about the future of my career, my writing, and possible graduate school trajectories. I think I’ll be writing a series of posts on the blog over the next couple of weeks on this topic. You’ve been warned.

Onward and Upward!

As it turns out, the process of finding a new job has also made me a better editor of my own work. I hope this, and the residual improvements from the lessons I’ve learned at work are apparent in my writing here. ↩︎

Leadership in Distributed Social Networks

Tue, 19 Apr 2011 00:00:00 +0000

Let us understand “social networks,” to mean networks of people interacting in a common group or substrate rather than the phenomena that exists on a certain class of websites (like Facebook): we can think of this as the “conventional sense” of the term.

As I watch Anonymous, the ‘hacktivist" group, to say nothing of the movements in Egypt and Tunisia, I’m fascinated by the way that such an ad hoc group can appear to be organized and coherent on the outside without appearing to have real leadership. External appearance and reality is different, of course, and I’m not drawing a direct parallel between what Anonymous is doing and what happened in Egypt, but there is a parallel. I think we’re living in a very interesting moment. New modes of political and social organizing do not manifest themselves often.

We still have a lot to learn about what’s happened recently in Egypt/Tunisia/Libya and just as much to learn about Anonymous. In a matter of months or years, we could very easily look back on this post and laugh at its naivete. Nevertheless, at least for the moment, there are a couple of big things that I think are interesting and important to think about:

We have movements that are lead, effectively, by groups rather than individuals, and if individuals are actually doing leadership work, they are not taking credit for that work. So movements that are not lead by egos.
These are movements that are obviously technologically very aware, but not in a mainstream sort of way. Anonymous uses (small?) IRC networks and other collaborative tools that aren’t quite mainstream yet. The Egyptian protesters in the very beginning had UStream feeds of Tahrir Square, and I’d love to know how they were handling for internal coordination and communication.
I think the way that these movements “do ideology,” is a bit unique and non conventional. I usually think of ideology as being a guiding strategy from which practice springs. I’m not sure that’s what’s happening here.
The member activists, who are doing the work in these movements are not professional politicians or political workers.

The more I ponder this, the more I realize how improbable these organizations are and the more I am impressed by the ability of these groups to be so effective. In addition to all of my other questions, I’m left wondering: how will this kind of leadership (or non-leadership) method and style influence other kinds of movements and projects.

Breaks and Dodging Writer's Block

Tue, 12 Apr 2011 00:00:00 +0000

Over the past few weeks, I have developed (redeveloped?) the habit of pushing through a few hundred words of fiction writing a day on the train. If nothing else, this means I get a few pages done a day. I sometimes wish that I were able to write more, writing something is better than nothing, and I’m within a chapter and some change of the end of this beast, so I’ll take it. Some, dare I say most, days the writing is pretty easy, but it can get a bit rough. The problem is really when the writing is impossible. After all, if writing were easy, than we wouldn’t really think of it as work.

Most of the time, I don’t feel “blocked,” but I got close last week. I got to the end of a section and I realized I was in a bit of a corner. I’d changed the order of a few little events, and I didn’t think it would change much.

Except it did, which left me pretty stuck.

It wasn’t an unsolvable problem, but because I was faced with writing a scene under a different set of assumptions than the one I’d been thinking about, I didn’t know how to proceed. My solution was to take a step back, think about things, and then redesign the next scene so that it accomplished what I needed. This took a day or two.

Most of the time the only real marker for progress in writing is in “number of words recorded,” but the truth is this is the fun part of writing. The hard part is, and the part that I had gotten stuck in was, figuring out what’s worth writing. Here are some interesting lessons about writing for your consideration:

Sometimes the most productive thing you can do as a writer is to not write and spend time figure out what doesn’t need to be said. It doesn’t make your writing longer, but it does make your writing more clear and more useful as a result.
As a corallary, sometimes the most important thing you can do to a text is remove stuff that is distracting.
Although having too many “open projects” on your plate can make you feel like you’re spread too thin, it’s true that having a lot of things in the air increases your chance of getting thigns done, because having multiple projects in different stages lets you get more work done in general.

Sometimes the best way to solve these kinds of problems with writing is to take a break and go for a walk. Or a couple of walks. This is difficult to do on the train and can be hard to practice effectively in more conventional situations life. What’s the difference between taking a break to clear your mind, and breaking your writing ritual/habit? It’s hard to say.

If you do it right, the great thing is that after a break and some serious thinking time, a clear mind makes it possible to get past bigger challenges accomplish something important. I’ll leave the judgements of “how much time away is enough,” and “how much time is too much,” to the reader (and myself!) as an exercise, but don’t always be afraid of taking a break.

And if you won’t, I won’t either.

Onward and Upward!

Progress Reports

Fri, 08 Apr 2011 00:00:00 +0000

I wish there was a good way, in the context of a blog post or some other convenient digital media, to regularly say: “I’ve done some things, you may be interested in them,” and “I’d like to do some things, here are some notes of what I think I’d like to be working on in the next little bit.” I’ve yet to find a good way to get in the habit of writing this kind of post.

My instinct is to have something that I can template and automate pretty strongly so that I can mostly focus on doing things rather than writing blog posts about doing things. Or as is presently the case, blog posts blog posts about doing things. Oy. Ideas and suggestions on this topic would be most appreciated.

So what have I been up to? Well…

I got back into writing fiction this week after several weeks away. I was in a difficult part of the story and life got very… full… and I stopped writing regularly. These things happen, but it’s good to be back at least some.

After the work I did just last night, I realize that I’m much closer to the end of this novel than I had thought. Even if things go very slowly for the next few months, I think I’ll be able to get this thing done by the end of May. I just have to write about a thousand words to finish Chapter 11, and there are three to four little vignettes in chat per 12 to wrap the whole thing up. And it’s going to be awesome.

I’m super excited to be done with this project and to be able to spend some time cleaning it up and making it an awesome text, but also being able to work on making other things I’ve written better, and to be able to write new things, with my undivided attention.
I’ve updated the /posts/anti-rodentia and the associated discussion page with some changes that I think make the system work much better.
I’ve upgraded to Firefox 4, which I really like a lot: it’s much faster, it’s much more minimal (visually) which is great. Also all of my plugins and extensions work without issue. Good job!
I went through a blast from my past when I pulled together the queer theories page, which does some--at this point utterly redundant--definitional work about what it means to be queer, and potential limitations on queer identity/politics in the era of a broadly defined queer.
I’ve started adding recurring tasks to keep me on track with writing and posting entries to this post. This means I don’t end up posting things months after I write them. As was the case with Mutt Sucks Less, a post I wrote many months ago and posted with some revisions last week.
I made a folk page for frequent comm enter and discourse participant Matt Lundin that includes a snippet which makes a list (and RSS feed!) of all pages that link to “madalu” or are tagged with “madalu” (his handle). You can see this by editing the page, and use it when making your own folk pages. Which you should totally do.

Have a good weekend!

Talk Proposals

Wed, 06 Apr 2011 00:00:00 +0000

At POSSCON there were a lot of talks, most of which did little to interest me. I don’t think this was the fault of the conference: I’m a weirdo. I tend to be developer-grade geeky, but am still not a developer and I wasn’t otherwise representative of the general audience. By the end, I was starting to think that the thing most people talk about at conferences isn’t very cutting edge. I don’t think it’s just POSSCON (surely not!) but I’ve not been to enough conferences to be able to speak definitively. In any case, I’d like to propose in open forum (i.e. this wiki,) a number of conference presentations that I’d like to see or would be willing to present.

If you’re interested in any of these presentation, or want to help/inspire me to work up notes, please create or add to the wiki pages linked to below.

Emacs Productivity and Production, Org-Mode and Beyond

Emacs, with its extensive feature list, endless customizations, and arcane approach to user interface, is often the butt of many jokes. While some of this is certainly valid, there are many incredibly innovative and intensely useful pieces of software written for Emacs. This talk would center on the org-mode package, but would branch out to talk workflows and automation in Emacs and using Emacs to help people make awesome work.

The Year of The Linux Desktop: Amazing Window Manager Paradigms

I’m always distraught by the way that discussion of “The Linux Desktop” revolves around convincing people that the major desktop environments (KDE/GNOME) either: are feature comparable to the Windows/OS X desktop or are able to “out-Windows” and “out-OS X” each other/Windows/OS X. Both of these propositions seem somewhat tenuous and unlikely to be convincing in the long run, and do little to inspire enthusiasm for the platform. This is sad because there is a lot of very interesting activity in the Linux desktop space. This talk would present and explore a couple of projects in the tiling window manger space and explain why this kind of software is what should drive adoption of the Linux desktop.

Cloud Independence, Infrastructure, and Administration

The “cloud computing” paradigm and the shift to thinking about technology resources as service based raises some interesting questions about software/computing freedom and the shape of data ownership in the contemporary moment. This talk would address these questions, provide an overview of how to “go it alone,” and how to be responsible for managing and administering for your own personal “cloud infrastructure.”

Mutt Sucks Less

Tue, 05 Apr 2011 00:00:00 +0000

I use a mail client called mutt. The quality of this software may largely explain my opinion this post on the continued relevance of email

I think mutt warrants a bit of extra attention for two reasons. First, because I think there are enough people out there who don’t use mutt who could and perhaps should, and I’d like to do a little encouraging; and second,like all fundamentally wonderful pieces of software, mutt can teach us something important about what makes technology great and pleasurable to use.

Working with any new kind of software is always a challenge. It is unfortunate that “features” and “functions” are the currency by which we judge software. Which is unfair to both the technology and ourselves, as the utility and quality of these features/functions depends on a number of subjective/individual factors. That said, with regards to mutt, my list is as follows:

Mutt is agnostic on the editor question. I suspect the fact that I could use any text editor I wanted to write email was probably my original reason for switching to mutt in the first place. It’s amazing what a sane editing experience can do for the overall experience of writing emails.
Support for PGP/GPG encryption. Signing and encrypting emails with PGP is probably only a minor advantage, and of limited actual utility, but I think it’s important and valuable to have this capability in your email client. After all, the success of PGP depends on a crowd effect: if it’s easy, sign all your email and hope that others will join you. Mutt makes this easy, which is a good thing indeed.
Mutt operates independently of mail transmission protocols, which are universally flawed. In many ways, by not including support for mail transmission, mutt is more useful and more flexible than it would be if it was designed to handle mail transmission. Having said that, recent versions of mutt have internal support for IMAP/POP/SMUT. Not that I’d use it or recommend that you do use it and I suspect most mutt users don’t either.
Mutt operates independently of mail storage format: you can maintain complete control over your mail data, and store email pretty much however you like. While this may be a burden to some, I’m somewhat controlling when it comes to data storage and preservation, and I think email archives are incredibly important. And I’m a weirdo about email storage.
Mutt’s “sidebar patch” isn’t even a part of the core of the software, but it’s absolutely crucial to my experience of the software. Basically it gives you a heads-up-display of your mailboxes and tells you at a glance: if there are new messages and how many messages (new, flagged, read) are in an mailbox. While it eats into some screen real estate, it’s generally unused screen space and it’s more than worth the expenditure of pixels.
Mutt runs on console and can be compiled on pretty much any contemporary UNIX-like system. Chances are there are packages for most operating system. So I feel pretty confident that I’ll pretty much be able to use mutt no matter what kind of system I end up using. Also console apps generally run pretty well in screen, which makes them accessible (and persistent) across the internet.

Onward and Upward!

Desks and Stationary Mobility

Mon, 04 Apr 2011 00:00:00 +0000

This is a post about mobile technology in an unconventional sense. I think I’m probably an extreme “mobile” technology user: I ride a lot commuter rail and use my laptop extensively on the train. Then, I work on a laptop all day. In the evening, I often do at least a little additional work, again on the same laptop. There is, after all, always writing (like this post!) to fill any remaining free time.

I’m not a terribly typical mobile user. My main “mobile device” is a little ThinkPad (and sometimes a larger ThinkPad,) running Linux and a lot of Lisp (emacs and otherwise.) It’s not ideal for every situation: there are times when I just can’t bare to open the laptop again or it’s unfeasible (and there’s always the Kindle for times like those.) Most of the time it works well.

It’s hard to omit discussion of the “tablet” and the iPad. For me, the fact that tablets are not general purpose computers is a huge deterrent. This is probably not the case for everyone, though there are lots of shades to this debate. I think the more interesting question is not “do people need general purpose tablets?” and more “how will more ubiquitous embeded-type systems effect the way people will approach ‘general purpose’ computing environments” from here on out? Honestly, this in computing practice has already happened, but I think it will continue to pose important questions for users and developers as it continues.

The struggle, for me, revolves less around the question “how do I work remotely?” and more around “how do I also work when I’m at a desk?” The adjustment can be hard: For a while, I was so used to working on the train, and in random chairs, that I had a hard time focusing if the computer wasn’t actually on my lap. Bad ergonomics is only the start of this.

The current solution is to set up desks and workstations that use the same laptops and systems so that I’m not perpetually switching between fixed computers and mobile computers. I’m also keen for these desks to have their own appeal: bigger monitors, nice keyboards, and easy to attach power cords. I’ve also attempted to tie together all of the “I’d like to switch between laptop-mode and desk-mode,” functions (e.g. network connection, monitor attachment, window layout) into easy to trigger operations, so I can get started more quickly. Nice. Seamless. Efficient.

The lessons: There are many ways to maintain technical (cyborg) coherence despite/during geographical movement and sometimes that technology isn’t particularly cutting edge. Sometimes the best way to break yourself of a habit you don’t like is to play a game with yourself where you establish a more attractive option. Finally, a very small change or automation can be enough to take something difficult and make it much easier or something unpleasantly and make it workable.

Long Live the Tycho Wiki

Sat, 02 Apr 2011 00:00:00 +0000

I realized that I made a lot of changes to the way my websites were arranged and worked recently, and I thought that it would be good to write a few about the hottest new thing, and my rationale for messing everything up. Again.

Basically, I took the CriticalFutures.com site, and turned it into the posts section here, moving a bunch of content back to tychoish.com that used to be here. This time, everything is powered by ikiwiki, and while I callously dropped all comments¹, I think the move is one of general and overall improvement.

At the same time, I think there are challenges. Rather than comment pages, there’s now a “discourse” link at the top left of the pages, which lets you sign in and create a new page for discussion related to the page you’re currently viewing. I liked the name “discourse” rather than “discussion,” but I fear there may be some confusion. I can change it over back without much issue, if any one feels strongly.

Mostly, it’s still the same blog that it’s always been, and I’m having fun with it. The essay/rhizome divide may turn out to be tenuous in the long term, and it might disappear, but for now it makes sense. I’ve not been doing as many wiki-based things, though I have worked on a queer theories page, and the technical writing section is something that I’m pleased with. Wikis are much more difficult to use and produce than one would initially guess. The concept is simple, the software is simple, and the work is remarkably time consuming and difficult to do right. And then there’s the problem of contributors.

I don’t know how it’s going to sort out, but I look forward to hearing from you, and making stuff on this domain.

Which totally sucks, but I had already given into using disqus, the migration would be really difficult, and the truth be told, I’d much rather participate in discussions that involved the creation of new wiki pages, and that involved invitations to email lists, and direct emails contact over IRC or IM. Maybe this is more fragmented, and maybe I’m shooting nascent conversation in the foot before it has a chance to get off the ground, but I think “conversation at any cost,” is not the best model for a community. ↩︎

The Inevitability of Open Source

Thu, 31 Mar 2011 00:00:00 +0000

I recently attended POSSCON as part of my day-job. I don’t usually blog directly about this kind of stuff (“You like to keep your church and state separate,” a fellow attendee said, which fits.) But, I had a number of awesome conversations with the speakers, attendees and sponsors, that may spawn a series of brief posts here. POSSCON is a regional open source convention that drew developers, leaders of informational technology departments, and IT consultants of various types.

I had a number of conversations that revolved around the adoption of open source in opposition to proprietary systems. People asked questions like “what do we have to do to get more people to use open source software?” and many people apologized for doing work with proprietary software for mostly economic reasons (e.g. “I have a .NET development job,” or “people need windows administration and I can’t turn away work.")

This led me to have one of three reactions:

1. Working with any specific (proprietary) technology, particularly because you have to make ends meet should never require excusing. There are cases where “working with proprietary technology,” may more like “building a business model on proprietary technology,” and that sort of thing needs to be watched out for, but I don’t think it’s morally ambiguous to make a living.

2. I’m not sure that the success of technology, particularly open source, is determined solely on the basis of adoption rates. Successful technology is technology that efficiently allows people/cyborgs to do work, not overwhelmingly ubiquitous technology.

3. In many many contexts, open source technology has triumphed over proprietary alternatives: Linux-based systems are the dominant UNIX-like operating system. OpenSSH is the dominant SSH implementation (and remote terminal protocol/implementation). Darwin/FreeBSD is incredibly successful (as Mac OS X.) Other domains where open source packages have very high (dominating) adoption rates: OpenSSL, gcc, perl/python/php/ruby (web development), Apache/Lighttpd/nginx (web servers) etc.

While I think the end-user desktop isn’t unimportant, I think there may be merit in playing to the strengths of open source (servers, infrastructure, developers.) Additionally, it seems more productive to have the discussion about “how do we advance open source,” couched in terms of a battle for technological dominance in which open source has already won.

And Free Software/Open Source has won. While there remain sectors and domains where non-free software remains prevalent and business models that don’t value user’s freedom, I think that most people who know anything about technology will say that all paths forward lead toward a greater level of software freedom.

Maybe this is a symptom of the situation in which I work and maybe I’m being too optimistic, but I don’t think so. Thoughts?

Anti-Rodentia

Tue, 29 Mar 2011 00:00:00 +0000

I hate computer mice. A lot.

The closest I’ve gotten to liking a pointing device is an acquiescence to the TrackPoint on the laptops I use. That’s the little red dot in the middle of the ThinkPad keyboards. My problem with computer mice is the context switch between “typing-mode” and “mousing-mode.” Moving between the modes is jarring and inefficient. I’ve been using StumpWM and other similar window managers for years now and as my need for the mouse decreases my irritation with needing to use a mouse increases.

I’ve been struggling for a few months with a bit of a problem: Several months ago I got a new bigger ThinkPad, a T510, while it has my beloved TrackPoint there is also a TouchPad. After years of only using laptops with the red dot, this was very disconcerting. How did I keep from triggering the touchpad with my wrists? Couldn’t I just turn the damn thing off?

I did, and everyone who tried to use the computer after that was dismayed, and I didn’t care. Except, I found out that, apparently, disabling the touchpad also disables all non-TrackPoint pointers. So when I plugged the laptop into the docking station, the external mouse didn’t work.

Blast.

The solution to disabling and enabling the mouse on the fly, that follows isn’t as pretty as I’d like, but it works.

UPDATE: Turns out that my original procedure only appears to work. I’ve made the following modification to the toggle-mouse script, using a stock xorg.conf file.

File: /usr/bin/toggle-mouse

#!/bin/sh

TOUCHPAD=`xinput list | egrep “TouchPad” | sed -r ’s/.id=([0-9]).*$/1/'`

if [ [xinput list-props “$TOUCHPAD” | egrep -o “[0-9]$” | head -n1]{.title-ref} -eq 0 ]; then xinput set-prop “$TOUCHPAD” “Device Enabled” 1 elif [ [xinput list-props “$TOUCHPAD” | egrep -o “[0-9]$” | head -n1]{.title-ref} -eq 1 ]; then xinput set-prop “$TOUCHPAD” “Device Enabled” 0 else xmpp-notify “Your mouse is probably screwed up somehow” fi

Test the output of “xinput list | egrep "TouchPad" | sed -r 's/.*id=([0-9]*).*$/\1/'”, and inspect “xinput list” to make sure that the value of $TOUCHPAD is the xorg id of the touchpad (or other device) that you want to disable.

I’d actually recommend not putting this in /usr/bin/, but just so long as it’s in the path. Then run toggle-mouse at the command line. You may need to run this as root-suid, for your system system is configured. Tweak the TOUCHPAD variable as needed.

If you have a better solution, I would be terribly interested in hearing about it.

City Infrastructure

Thu, 17 Mar 2011 00:00:00 +0000

I’m always interested in how the lessons that people learn in IT trickle down to other kinds of work and problems. This is one of the reasons that I¹ am so interested in what developers are interested in: if you want to know what’s happening in the technology space, it’s best to start at the top of the food chain. For this reason this article from IBM, which addresses the use of IT/Data Center management tools outside of the data center was incredibly interesting for me.

When you think about it, it makes sense. IT involves a lot of physical assets, even more virtual assets, and when projects and systems grow big enough, it can be easy to lose track of what you have, much less what state it’s in at any given time. Generalized, this is a prevalent issue in many kinds of complex systems.

As an aside, I’m a little interested when software that provides asset management and monitoring features, will scale down to the personal level. That’ll be interesting too. There are the beginnings of this kind of thing (e.g. iTunes, and git-annex) but only the beginnings.

I’m left with the following questions:

Obviously moving from managing and monitoring networked devices to managing and monitoring infrastructure objects like water filtration systems, storm water drainage, the electrical grid, snow removal, etc. presents a serious challenge for the developers of these tools, and this adaptation will likely improve the tools. I’m more interested in how cities improve in this equation. And not simply with regards to operating efficiencies. What do we learn from all this hard data on cities?
Will cities actually be able to become more efficient, or will they need to expand to include another layer of management management, that nullifies the advances. There are also concerns about additional efficiency increasing the “carrying capacity of cities,” into unsustainable levels.
Can the conclusions from automated city-wide reporting lead to advancements in quality of service, if we’re better at determining defective practices and equipment. In this vein, how cities share data between them will also be quite interesting.

I’d love to hear from you!

RedMonk also use a similar argument. ↩︎

Do You Read Philosophy?

Wed, 16 Mar 2011 00:00:00 +0000

“What do you do?”

“I’m a technical writer.”

“Do you write other stuff?”

“A bit, sometimes.”

“Poetry?”

“No Poetry.” I laugh.

“Fiction?”

“Yeah, Some.”

“Do you read philosophy too?”

“A bit.”

“Oh, good! Materalist or Idealist?”

“Materialist.”

“Who do you read?”

“I’m a bit of an unreformed Deluzian.”

“Deleuze wasn’t a materialist.”

“Yeah, but he wanted to be. Really bad.”

“There’s that. I’ve been reading Hegel recently.”

“Oh, really.”

Professional Content Generation

Tue, 15 Mar 2011 00:00:00 +0000

I’m a writer. I spend most of my day sitting in front of a computer, with an open text editing program, and I write things that hopefully--after a bit of editorial work--will be useful, enlightening, and/or entertaining as appropriate. I’ve been doing this since I was a teenager and frankly it never seemed to be particularly notable a skill. The fact that I came of age with the Internet a member of its native participant-driven textual culture had a profound effect, without question. This is a difficult lineage to manage and integrate.

Obviously I’m conflicted: on the one hand I think that the Internet has been great for allowing people like me to figure out how to write. I am forever thankful for the opportunities and conversations that the Internet has provided for me as a writer. At the same time, the Internet, and particularly the emergence of “Social Media” as a phenomena complicates what I do and how my work is valued.

Let’s be totally clear. I’m not exactly saying “Dear Internet, Leave content generation to the professionals,” but rather something closer to “Dear Internet, Let’s not distribute the responsibility of content generation too thinly, and have it come back to bite us in the ass.” Let me elaborate these fears and concerns a bit:

I’m afraid that as it becomes easier and easier to generate content, more will start creating things, and there will be more and more text and that will lead to all sorts of market-related problems, as in a vicious cycle. If we get too used to crowd sourcing content, it’s not clear to me that the idea of “paying writers for their efforts,” will endure. Furthermore, I worry that as the amount of content grows, it will be harder for new content to get exposure and the general audience will become so fragmented that it will be increasingly difficult to generate income from such niche groups.

Some of these fears are probably realistic: figuring out how we will need to work in order to our jobs in an uncertain future is always difficult. Some are not: writing has never been a particularly profitable or economically viable project, and capturing audience is arguably easier in the networked era.

The answer to these questions is universally: we’ll have to wait and see, and in the mean time, experimenting with different and possibly better ways of working. My apologies for this rip-off, but it’s better to live and work as if we’re living in the early days of an exciting new era, rather than the dying days of a faltering regime.

Perhaps the more interesting implication of this doesn’t stem from asking “how will today’s (and yesterday’s) writers survive in the forthcoming age,” but rather “how do these changes affect writing itself.” If I don’t have an answer to the economic question, I definitely don’t have an answer to the literary question. I’m hoping some of you do.

As an interesting peak behind the curtain, this post was mostly inspired as a reaction to this piece of popular criticism that drove me batty. It’s not a bad piece and I think my objections are largely style and form related rather than political. Perhaps I’m responding to the tropes of fan writing, and in retrospect my critique of this piece isn’t particularly relevant here. But that article might provide good fodder for discussion. I look forward to your thoughts in comments or on a wiki page.

Onward and Upward!

News, Fit to Sing

Mon, 14 Mar 2011 00:00:00 +0000

It’s sometimes easy to forget all of the little things that I do during the week, and how they add up to something of note. In the moment--any moment--most things seem much smaller and much less important than they do with a little bit of perspective and when viewed out of context with coordinating achievements. So here’s the highlights from last week:

I gave up on the “having two sites to maintain thing,” and have merged Critical Futures into the tychoish.com blog/wiki. I really like this, and it gave me some time to get elbows deep into the wiki system, which means things work better, the display is a bit cleaner, my life is easier, and I’m very happy with it. Also, as part of this process I’ve been revising the index page, and I think it is in a state that I’m really pleased with.

Some dancing friends on Facebook said “wouldn’t it be nice if there was a wiki for contra dancers.” Now there is. Contra Dance Wiki is there for you all if you want it. I’ve done some preparatory work on the index page, and I’ll continue to add things as I can.

The work I’ve been doing in the last few weeks with ikiwiki, both for tychoish and now for the CDW means that I’m really close to being able to share all of the assorted templates and configuration files I use to make this work. It’s all in the git repository for tychoish, but I’m going to pull together a more generic version so people can get started easier. It might also make sense to write a deployment script of sorts. We’ll see.

I meant to post this on Friday, but posted my review of maple morris, because I didn’t want that to linger. In any case, I had a brilliant weekend singing (and dancing) in Western Massachusetts. It has however left me a bit under the weather, so I’m spending a day recovering and doing some writing. Good stuff there. I’ll write about WMSHC soon.

Maple Morris Review

Fri, 11 Mar 2011 00:00:00 +0000

A few weekends ago, I went to very weird get together. The week before I went to upstate New York to a festival that drew 5000 attendees. Then, I went to Washington, DC to dance with 24 or 25 other people from across the country and Canada. I think of it as “my generation,” Morris dance gathering.

This May marks my 10th anniversary of being a Morris dancer. I’ve spent most of that time, easily being twenty years away from the next-youngest Morris dancer on my team. Morris isn’t aging quite that fast: but there are a lot of quirky things that happen given the small sample sizes.

I’ve been involved in the folk world for years. Lots of folk dance and traditional music. I’m so accustomed to this, that I’m not really sure what people who aren’t do with their time. When I think about other communities, I always reach back to experiences and phenomena that I’ve seen in the folk world.

While I grew up in the folk dance world, I’m coming to terms with a couple of things: First, folk communities are different in different parts of the country/world, and the community in Boston (or New York, or Philadelphia) is very different from what I grew up with. Second, I’m realizing that while I’m “a young person” who grew up with music and dance, I’m no longer “a folk dance kid,” (and that’s a nifty thing to experience.)

Given this, I’ve had the following Morris related thoughts, that seem worth recording:

It’s really nice to be part of a single age cohort in this activity, mostly because I’ve not had significant opportunity to dance with people in my general age group.
I quite enjoy being more than just a familiar face in a contra dance line, or someone that you see across the square when singing sacred harp.
These weekends always challenge me to be a better dancer, and make me realize that I need to focus and work on certain aspects of my dancing.
While it’s in-ideal to have gatherings capped at really small numbers, having a small group means a greater strength of connection between everyone, and it means that a few people can do the organizing work without much institutional/organizational overhead. That’s really cool.
Once again, my motto is “you don’t have to do everything.” Which is particularly difficult around Morris. But I think by avoiding overdoing it I’m able to: avoid injury and have greater successes at the things I do try. Can’t argue with that. I’m young and I hope to dance for many years to come, and there’ll be time enough for Sherborne then.

Dance Flurry, Review

Tue, 08 Mar 2011 00:00:00 +0000

I went to the Dance Flurry a couple of weeks ago (!) and I wanted to write a few notes here about the experience, and a little bit of reflection. I hope you’ll spare me the indulgence.

A year ago, I was pretty new to the East Coast: I didn’t really know people, and while I’d been dancing for a while and I wasn’t a bad dancer by any means, but I wasn’t quite comfortable in my own skin in big dance events.

This year, many things were different. I’d been to a number of other important regional events: I knew more people, I knew the bands and the callers, I knew the venue, and I knew what to expect.

It was great. I got to dance with friends that I hadn’t seen in months. I got to dance to great bands. The callers were top notch, and there was never a shortage of great ways to spend my time.

My motto for the weekend was “you don’t have to do everything.” Which meant not staying until 1am, just because there was a dance going on; or not showing up at 9am because that’s when things started. Prevailing sanity is an amazing thing.

It meant that I didn’t hurt myself; I didn’t come back from the vacation more tired than I was when I left; and I still had a great time.

It’s amazing.

Git Sync

Mon, 07 Mar 2011 00:00:00 +0000

With the new laptop, I once again have more than one computer, and with it a need to synchronize the current state of my work. This is a crucial function, and pretty difficult to do right: I’ve had multiple systems before and condensed everything into one laptop because I didn’t want to deal with the headache of sitting down in front of a computer and cursing the fact that the one thing that I needed to work on was stuck somewhere that I couldn’t get to.

I store most of my files and projects in git version control repositories for a number of reasons, though the fact that this enables a pretty natural backup and synchronization system, was a significant inspiration for working this way. But capability is more than a stones throw from working implementation. The last time I tried to manage using more than one computer on a regular basis, I thought “oh, it’s all in git, I’ll just push and pull those repositories around and it’ll be dandy.” It wasn’t. The problem is, if you keep different projects in different repositories (as you should, when using git,) remembering to commit and push all repositories before moving between computers is a headache.

In the end synchronization is a rote task, and it seems like the kind of thing that was worth automating. There are a number of different approaches to this and what I’ve done is some very basic bash/zsh script¹ that takes care of all of this syncing process. I call it “git sync,” you may use all or some of this as you see fit.

git sync lib

The first piece of the puzzle is a few variables and functions. I decided to store this in multiple files for two reasons: First, I wanted access to the plain functions in the shell. Second, I wanted the ability to roll per-machine configurations using the components described within. Consider the source.

The only really complex assumption here is that, given a number git repositories, there are: some that you want to commit and publish changes too regularly and automatically, some that you want to fetch new updates for regularly but don’t want to commit, and a bunch that you want to monitor but probably want to interact with manually. In my case: I want to monitor a large list of repositories, automatically fetch changes from a subset of those repositories, and automatically publish changes changes to a subset of the previous set.

Insert the following line into your .zshrc:

source /path/to/git-sync-lib

Then configure the beginning of the git-sync-lib file with references to your git repositories. When complete, you will have access to the following functions in your shell: gss (provides a system-wide git status,) autoci (automatically pulls new content and commits local changes to the appropriate repository,) and syncup (pulls new content from the repositories and publishes any committed changes.

syncup and autoci do their work in a pretty straightforward for [...] done loop, which is great, unless you need some repositories to only publish in some situations (i.e. when you’re connected to a specific VPN.) You can modify this section to account for this case, take the following basic form:

syncup(){

   CURRENT=`pwd`

   for repo in $force_sync_repo; do
       cd $repo;

       echo -- syncing $repo
       git pull -q
       git push -q

   done
   cd $CURRENT

}

Simply insert some logic into the `for`` loop, like so:

for repo in $force_sync_repo; do
   cd $repo;
   if [ $repo = ~/work ]; then
      if [ `netcfg current | grep -c "vpn"` = "1" ]; then
          echo -- syncing $repo on work vpn
          git pull -q
          git push -q dev internal
      else
         echo -- $repo skipped because lacking vpn connection
      fi
   elif [ $repo = ~/personal ]; then
       if [ `netcfg current | grep -c "homevpn"` = "1" ]; then
          echo -- syncing $repo with homevpn
          git pull -q
          git push -q
       else
          echo -- $repo skipped because lacking homevpn connection
       fi
   else
      echo -- syncing $repo
      git pull -q
      git push -q
   fi
done

Basically, for two repositories we test to make sure that a particular network profile is connected before operating on those repositories. All other operations are as in the first example. I use the output of “netcfg current”, which is an ArchLinux network configuration tool that I use. You will need to use another test, if you are not using Arch Linux.

git sync

You can use the functions provided by the “library” and skip this part if you don’t need to automate your backup and syncing process. The whole point of this project was specifically to automate this kind of thing, so this--though short--is kind of the cool part. You can download git sync here.

Put this script in your $PATH, (e.g. “/usr/bin” or “/usr/bin/local”; I keep a “~/bin” directory for personal scripts like this in my path, and you might enjoy.) You will then have access to the following commands at any shell prompt:

git-sync backup
git-sync half
git-sync full

Backup calls a function in git-sync to backup some site-specific files to a git repository (e.g. crontabs, etc.) The half sync only downloads new changes, and is meant to run silently on a regular interval: I cron this every five minutes. The full sync runs the backup, commits local changes, downloads new changes, and sends me an xmpp message to log when it finishes successfully: I run this a couple of times an hour. But there’s an exception: if the laptop isn’t connected to a Wifi or ethernet network, then it skips sync options. If you’re offline, you’re not syncing. If you’re connected on 3g tethering, you’re not syncing.

That’s it! Feedback is of course welcome, and if anyone wants these files in their own git repository so they can modify and hack them up, I’m more than willing to provide that, just ask.

Onward and Upward!

I wrote this as a bash script but discovered that something with the way I was handling arrays was apparently a zsh-ism. Not a big fuss for me, because I use zsh on all my machines, but if you don’t use zsh or don’t have it installed, you’ll need to modify something in the array or install zsh (which you might enjoy anyway.) ↩︎

9 Awesome SSH Tricks

Tue, 01 Mar 2011 00:00:00 +0000

Sorry for the lame title. I was thinking the other day, about how awesome SSH is, and how it’s probably one of the most crucial pieces of technology that I use every single day. Here’s a list of 10 things that I think are particularly awesome and perhaps a bit off the beaten path.

Update: (2011-09-19) There are some user-submitted ssh-tricks on the wiki now! Please feel free to add your favorites. Also the hacker news thread might be helpful for some.

SSH Config

I used SSH regularly for years before I learned about the config file, that you can create at ~/.ssh/config to tell how you want ssh to behave.

Consider the following configuration example:

Host example.com *.example.net
User root
Host dev.example.net dev.example.net
User shared
Port 220
Host test.example.com
User root
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
Host t
HostName test.example.org
Host *
Compression yes
CompressionLevel 7
Cipher blowfish
ServerAliveInterval 600
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

I’ll cover some of the settings in the “Host *” block, which apply to all outgoing ssh connections, in other items in this post, but basically you can use this to create shortcuts with the ssh command, to control what username is used to connect to a given host, what port number, if you need to connect to an ssh daemon running on a non-standard port. See “man ssh_config” for more information.

Control Master/Control Path

This is probably the coolest thing that I know about in SSH. Set the “ControlMaster” and “ControlPath” as above in the ssh configuration. Anytime you try to connect to a host that matches that configuration a “master session” is created. Then, subsequent connections to the same host will reuse the same master connection rather than attempt to renegotiate and create a separate connection. The result is greater speed less overhead.

This can cause problems if you' want to do port forwarding, as this must be configured on the original connection, otherwise it won’t work.

SSH Keys

While ControlMaster/ControlPath is the coolest thing you can do with SSH, key-based authentication is probably my favorite. Basically, rather than force users to authenticate with passwords, you can use a secure cryptographic method to gain (and grant) access to a system. Deposit a public key on servers far and wide, while keeping a “private” key secure on your local machine. And it just works.

You can generate multiple keys, to make it more difficult for an intruder to gain access to multiple machines by breaching a specific key, or machine. You can specify specific keys and key files to be used when connected to specific hosts in the ssh config file (see above.) Keys can also be (optionally) encrypted locally with a pass-code, for additional security. Once I understood how secure the system is (or can be), I found my self thinking “I wish you could use this for more than just SSH.”

SSH Agent

Most people start using SSH keys because they’re easier and it means that you don’t have to enter a password every time that you want to connect to a host. But the truth is that in most cases you want to have unencrypted private keys that have meaningful access to systems because once someone has access to a copy of the private key the have full access to the system. That’s not good.

But the truth is that typing in passwords is a pain, so there’s a solution: the ssh-agent. Basically one authenticates to the ssh-agent locally, which decrypts the key and does some magic, so that then whenever the key is needed for the connecting to a host you don’t have to enter your password. ssh-agent manages the local encryption on your key for the current session.

SSH Reagent

I’m not sure where I found this amazing little function but it’s great. Typically, ssh-agents are attached to the current session, like the window manager, so that when the window manager dies, the ssh-agent loses the decrypted bits from your ssh key. That’s nice, but it also means that if you have some processes that exist outside of your window manager’s session (e.g. Screen sessions) they loose the ssh-agent and get trapped without access to an ssh-agent so you end up having to restart would-be-persistent processes, or you have to run a large number of ssh-agents which is not ideal.

Enter “ssh-reagent.” stick this in your shell configuration (e.g. ~/.bashrc or ~/.zshrc) and run ssh-reagent whenever you have an agent session running and a terminal that can’t see it.

ssh-reagent () {
  for agent in /tmp/ssh-*/agent.*; do
      export SSH_AUTH_SOCK=$agent
      if ssh-add -l 2>&1 > /dev/null; then
         echo Found working SSH Agent:
         ssh-add -l
         return
      fi
  done
  echo Cannot find ssh agent - maybe you should reconnect and forward it?
}

It’s magic.

SSHFS and SFTP

Typically we think of ssh as a way to run a command or get a prompt on a remote machine. But SSH can do a lot more than that, and the OpenSSH package that probably the most popular implementation of SSH these days has a lot of features that go beyond just “shell” access. Here are two cool ones:

SSHFS creates a mountable file system using FUSE of the files located on a remote system over SSH. It’s not always very fast, but it’s simple and works great for quick operations on local systems, where the speed issue is much less relevant.

SFTP, replaces FTP (which is plagued by security problems,) with a similar tool for transferring files between two systems that’s secure (because it works over SSH) and is just as easy to use. In fact most recent OpenSSH daemons provide SFTP access by default.

There’s more, like a full VPN solution in recent versions, secure remote file copy, port forwarding, and the list could go on.

SSH Tunnels

SSH includes the ability to connect a port on your local system to a port on a remote system, so that to applications on your local system the local port looks like a normal local port, but when accessed the service running on the remote machine responds. All traffic is really sent over ssh.

I set up an SSH tunnel for my local system to the outgoing mail server on my server. I tell my mail client to send mail to localhost server (without mail server authentication!), and it magically goes to my personal mail relay encrypted over ssh. The applications of this are nearly endless.

Keep Alive Packets

The problem: unless you’re doing something with SSH it doesn’t send any packets, and as a result the connections can be pretty resilient to network disturbances. That’s not a problem, but it does mean that unless you’re actively using an SSH session, it can go silent causing your local area network’s NAT to eat a connection that it thinks has died, but hasn’t. The solution is to set the “ServerAliveInterval [seconds]” configuration in the SSH configuration so that your ssh client sends a “dummy packet” on a regular interval so that the router thinks that the connection is active even if it’s particularly quiet. It’s good stuff.

/dev/null .known_hosts

A lot of what I do in my day job involves deploying new systems, testing something out and then destroying that installation and starting over in the same virtual machine. So my “test rigs” have a few IP addresses, I can’t readily deploy keys on these hosts, and every time I redeploy SSH’s host-key checking tells me that a different system is responding for the host, which in most cases is the symptom of some sort of security error, and in most cases knowing this is a good thing, but in some cases it can be very annoying.

These configuration values tell your SSH session to save keys to `/dev/null (i.e. drop them on the floor) and to not ask you to verify an unknown host:

UserKnownHostsFile /dev/null
StrictHostKeyChecking no

This probably saves me a little annoyance and minute or two every day or more, but it’s totally worth it. Don’t set these values for hosts that you actually care about.

I’m sure there are other awesome things you can do with ssh, and I’d live to hear more. Onward and Upward!

How Hacker News and Social Editing has Jumped the Shark

Mon, 28 Feb 2011 00:00:00 +0000

How’s that for a Malcolm Gladwell-style subtitle title?

Alex Payne posed a piece on why he thought that, basically Hacker News had jumped the shark. He’s right, of course, but I think that his analysis of the root cause of the problem, and therefore the solution he proposes, is a bit too optimistic.

It seems to me that all of these socially edited link-based blogs where lots of people submit links and then the community votes on these stories to generate headlines and a filtered and sorted selection of content based on the appetites of the communities.

Hacker News tried, and succeeded for a time because the contributors (people who voted on, submitted, and commented on items) was small and focused enough that the content managed to coherent overall. There wasn’t enough content to necessarily overwhelm, both the readers and the potential contributors, and most things were interesting to most people. It was a golden age.

I think the problem with this model of generating content is that the golden ages don’t last very long. Sites “jump the shark” as the tightly focused content of the early, gives way to a more loose, less specialized, and more self-interested content submissions and selections. The factors that I think lead to this are:

There’s too much content. Such sites should be “filters,” and their fundamental service is to take the whole internet and tell readers “you should be reading this because we think it’s interesting/important.” As communities of editors grow, as the marketing power of filter sites increases, and if the “cost” of submitting a link remains constant, then the use of the filter breaks down and everyone gets overwhelmed.
There’s not enough focus/responsibility on the part of the editors. When you have a few dedicated and professional editors, you begin to see consistency and perspective and approach in the content that is curated. When this function is distributed among a large group of community members: amazing things can happen but that’s not a guarantee.
Community edited sites tend to become incredibly self interested after a certain point. There’s a certain kind of story either about the community itself or that strikes submitters as being “about the kind of people that participate in the community, or “the kind of thing that people who read the site might like,” rather than things that interest them. Perhaps some of this comes from the game-based dynamic of rating systems and karma, perhaps it’s just a thing that happens.

In many ways, Trivium is probably the best example of a link filter blog. Alex Payne points to MetaFilter as an example of a site which has solved this problem, and although I have had a MeFi account for years, I’ve never been able to really get into it. Long story short, we still need editors and editorial vision, and the issues we’re seeing isn’t about the “focus” of a community, as much as it is a property of communities themselves.

Java and Me

Wed, 23 Feb 2011 00:00:00 +0000

Stan reminded me recently that I have now written two posts about the Java programing and software development. For something that I admittedly don’t particularly care for, and don’t know a great deal about I’ve sure ranted a lot about it. I think I keep returning to think about Java because of how incredibly important Java is to the technology we use and how prevalent Java development remains.

Maybe I’ve read too much by Red Monk folks, but they tend take a very productive approach to these kinds of things. For reference, my posts on the subject are:

Most “end-users” don’t really care much about things like Java except when it doesn’t work, as is the case when some component of the Java platform isn’t present when you want to run a Java program or when you run a “cross platform” java application that doesn’t really work as indented on your platform. For a long time these two issues were prevalent enough that being written in Java was a decernable quality of an application. In most situations, computer programs are just computer programs.

With one exception.

The way software developers use computers and interface with technology leads and constrains the technological reality for the rest of us. For instance, in Google Reader you can scroll up and down using the “j” and “k” keys, which is derived from interaction paradigm of the Vi text editors. It’s great, but was almost certainly put into the software because a developer on the project was a Vi user. While a most features are driven by formal design processes, so much of the way software works is lead by the way developers think about software.

Ultimately Java developers are what make Java is important, not just for me but for everyone.

Searching for Known Results

Tue, 22 Feb 2011 00:00:00 +0000

(Note: I was going through some old files earlier this week and found a couple of old posts that never made it into the live site. This is one of them. I’ve done a little bit of polishing around the edges, but this is as much a post for historical interest as is a reflection of the contemporary state of my thought.)

This post is a follow up to my not much organization post, and as part of my general reorganization, I’ve been toying with anything for emacs which is a tool, or set of tools, which provide search-based interaction with some tasks (opening files, finding files, accessing other information, etc.) in a real-time search-based paradigm. Mmmm buzzwords. Think of it as being like quicksilver or launchy, except for emacs. I’ve come to a conclusion, that I think is generalizable, but made particularly obvious by this particular problem space.

Search, as an interface to a corpus, is only more effective than other organizational methods when you don’t know what the location of what your looking for is, or don’t understand the organizational system that governs the collection where your object is located. When you do know where the needed object is, search may be more cumbersome.

This feels obvious, when put in this way, but is counter to contemporary practice. Take the Google search use case where you find websites that you already know exist. You’d be surprised at how many people find this site by searching for “tychoish” or “tycho garen blog.” These are people who already know that the site exists and are probably people who have visited the site already. Google is forgiving in a way that typing an address into a search bar is not.

This works out alright in the end for websites: there’s no organizing standard for mapping domain names to websites. This is mostly due to the fact that you don’t, in the present practice, use the domain name system in the way that it was originally intended, in that the content of domain names are “brands” rather than a domain of systems and services described by the content of the domain. In the end this is not a huge problem since Google is around to help sort things out.

Similarly “desktop search” tools are helpful when you have a bunch of files scattered throughout file systems, with lots of hierarchy (directories and sub-directories). When you know where files are located, search less helpful. This is not to say that they’re ineffective: you’ll find what you’re looking for, it’ll just take longer.

I think this theory on the diminishing utility of search tool holds up, though I don’t exactly know how to do the research to further the develop the idea in a more concrete direction. Having said that, I think the following questions are important.

Are there practical ways to organize our files, that don’t require too much over-thinking before a collection grows unmanageable that make “resorting to search” less necessary?
Is (or might) building search tools for people who work with a given body of data (and therefore are familiar with the data, and are less likely to need search) different from building search for people who aren’t familiar with a given corpus?

Onward and Upward!

Anti-Social Media

Thu, 17 Feb 2011 00:00:00 +0000

I’ve been playing with this idea for a Critical Futures blog post for a few days, so you’ll probably see this again at some point. Still, I wanted to pose a couple of questions that have been nagging at me for a while:

Does the fact that we think of content as something that is becoming increasingly user generated, or generated outside of traditional professional structures, affect writers' ability to survive from an economic perspective?
Does this “crowd sourcing” (if you’ll indulge me,) mean that everyone will think of themselves as writers henceforth? While that’s potentially inspiring from the perspective of democracy, it feels hard to maintain from a literary/textual culture perspective. If everyone is a writer, is there an audience of readers for any kind of writing (fiction, critical, or non-fiction) separate from writers? If not, is there enough audience amongst fellow writers to support the project writing? (Answer: doubtful.)
I’m totally willing to accept that the publishing industry as we’ve come to know it is (and will continue) to undergo great change. At the same time, “great change,” means (I think) that some practices will need to change in a fundamental sort of way. It is not enough to manipulate the length publishing schedules of periodicals in order to get them to appear profitable for a while, it’s not enough to publish lots of small runs of books tight budgets that break even really fast.

These strategies delay the inevitable, but don’t address several fundamental problems:

The group of readers (e.g. audience) is significantly smaller than the public at large. If we want to grow audiences for our books, blogs, wikis, and other (hyper?)textual products we need to enlarge the group of people who read.
Most people who fail read any given text on any given day, fail to do so because they didn’t know it existed, and probably because they felt like they didn’t have time to do so.

Largely I think these issues can be extrapolated to other forms of media, I’m just a writer and think of things in terms of essays, articles, stories, and novels.

In any case, I think the way to save the “media industry” and media creators in particular, is to figure out how to get more people to read and figure out how to improve the discovery process. Hefty challenges, for sure.

Onward and Upward!

Caring about Java

Tue, 15 Feb 2011 00:00:00 +0000

I often find it difficult to feign interest the discussion of Java in the post Sun Microsystems era. Don’t get me wrong, I get that there’s a lot of Java out there, I get that there are a number of technological strengths and advantages that Java has in contrast some other programming platforms. Consider my post about worfism and computer programing for some background on my interest in programing languages and their use.

I apologize if this post is more in the vein of “a number of raw thoughts,” rather than an actual organized essay.

In Favor of Java

Java has a lot of things going for it: it’s very fast, it runs code in a VM that lets the code execute in a mostly isolated environment which increases reliability and security of the applications that run on the Java Platform. I think of these as “hard features” or technological realities that are presently implemented and available for users.

There are also a number of “soft features,” that Java has that inspire people to use it: an extensive and reliable standard library, a large expanse of additional library support for most things, a huge developer community, and it has inclusion in computer science curricula so people are familiar with it. While each of these aspects are relatively minor, and could theoretically apply to a number of different languages and development platforms, they represent a major rationale for it’s continued use.

One of the core selling points of Java has long been the fact that because Java runs on a virtual machine that can abstract differences between different operating systems and architectures, it’s possible to write and compile code once and then run that “binary” on a number of different machines. The buzzword/slogan for this is “write once, run anywhere.” This doesn’t fit easily into the hard/soft feature dichotomy I set up above, but it nevertheless and important factor.

Against Java

Teasing out the history of programing language development is probably a better project for another post (or career?), but while Java might have once had a greater set of support for many common programming tasks, I’m not sure that it’s sizable standard library and common tooling continues to overwhelm it’s peers. At best this is a draw with languages like Perl and Python, but more likely the fact that the JDK is so huge and varied increases incompatibility potentials. And needing to download the whole JDK to run even minimalist Java programs. Other languages have addressed the tooling and library support in different way, and I think the real answer to this problem is write with an eye towards minimalism and make sure that there are really good build systems.

Most of the arguments in favor of Java revolve around the strengths of the Java Virtual Machine, which is the substrate where Java programs run. And it is undeniable that the JVM is an incredibly valuable platform, and every report that I’ve seen concludes that the JVM is really fast, and the VM model does provide a number of persuasive features (e.g. sandboxing, increased portability, performance gains.) That’s cool, but I’m not sure that any of these “hard” features matter these days:

Most programing languages use a VM architecture these days. Raw speed, of the sort that Java has, is less useful than powerful concurrent programing abilities and is offset by the fact that computers themselves are absurdly fast. It’s not to say that Java fails because others have been able to replicate the strengths of the Java platform, but it does fail to inspire excitement.

The worth of Java’s “cross platform” capabilities are probably negated by service-based computing (the “cloud,") and the fact that cross platform applications, GUI or otherwise, are probably an ill gotten dream anyway.

The more I construct these arguments, I keep circling around the idea that while Java pushed a lot of programmers and language designers to think about what kind of features that programing languages needed. The world of computing and programming has changed in a number of significant ways, and we’ve learned a lot about the art of designing programming languages in the mean time. I wonder if my lack of enthusiasm (and yours as well, if I may be so bold) has more to do with a set of assumptions about the way programing languages should be that haven’t aged particularly well. Which isn’t to say that Java isn’t useful, or that it is no longer important, merely that it’s become uninteresting.

Thoughts?

Philosophy of the Present, Egypt

Tue, 15 Feb 2011 00:00:00 +0000

I’ve been watching the Egyptian revolution, off and on since it started. There’s so much interesting stuff going on: the pragmatics of political organization, the foundations of revolutionary movement, the evolving state of American political power, and the way that Egyptians are racialized particularly in contrast to Iranians and Tunisians.

The aspect that I’m most interested is in what western analysis of “Janurary 25th” tells us about how the west has made sense of past revolutionary moments in the last ‘50 years notably the Iranian revolution and “May'68.”

Largely I fear that it is way too soon to really say anything terribly useful on the subject. That hasn’t stopped people, of course.

I’m not sure what to make of either Ken MacLeod’s or Slavoj Žižek’s articles on the subject. There are aspects of this kind of theorizing that I really like and that really appeals to me. At the same time, optimism seems foolhardy.

There’s work still to be done: both theoretical and revolutionary. But isn’t there always?

The Rise and Fall of Netbooks

Mon, 14 Feb 2011 00:00:00 +0000

“I’m old fashioned,” R. said to me in an email, with that link to an article about how tablets have replaced and supplanted netbooks.

In many ways, you have two netbooks: the little one that’s been broken since may that I’m fixing and your real laptop. Which is to say: the advancement of netbooks was not, so much, the small form factor, but the fact that they were under powered computer systems meant to be used mostly with web-based applications.

Initially, netbooks were to have low capacity solid state hard drives and run Linux-based OSes. That was cool, for a while, but the cheap solid state drives turned out to perform more poorly than people expected and conventional hard drives became very cheap/available. Also Microsoft got scared and having seen that small-form factor computing was a real thing, adapted its strategy to seriously target these kinds of devices.

At which point everyone realized that there wasn’t a lot of point in making really small laptops: they were hard to type on and fundamentally they did everything for which you needed “big computers.” So companies started making “big netbooks.” The end result most 14"-15" laptops are basically big netbooks (including having similar resolutions.) The extra size is nice for most mundane uses, and for most the “mobility niche” is filled by smartphones anyway, rendering netbooks-sized device useful. Except they’re still around in different packaging.

Technology, I think, rarely fails. Rather, it gets reimplemented and reabsorbed by the next iteration of the technology. If we don’t pay attention we may miss the connections between iterations, but they are there.

Interestingly, and perhaps orthogonally, Linux lead the development for netbooks. Though tablets are different, and the history is less easily accessible, I think the same thing is happening there. It’ll be interesting to see how that pans out.

Progressions

Thu, 10 Feb 2011 00:00:00 +0000

I’ve been somewhat remiss in posting here. Nevertheless, I’ve managed to get rather a lot of things done in the couple of weeks that I think merits an update post.

I’ve posted two new things to Critical Futures, my final post--for now--about dexy, called make all dexy. and a post about new media on What we Learn from WikiLeaks.

I’ve also updated the Critical Futures Archive, particularly the posts on technical writing series and the new media series. I think hand crafted archives are incredibly valuable, but they’re hard to maintain, which I suppose is the point, and I’ve been a poor steward over the past few years. At the moment, the full archives of Critical Futures (nee tychoish.com, nee tealart.com) are listed on the archive page. The content has never gone anywhere, nor do I think the old content is any better than I used to, but it’s good to not have it totally hidden. I’m also trying to keep the hand crafted archives more up to date. It’s a struggle.

I’ve also been doing some wiki work around these parts, mostly in the technical writing section, including new pages about automicity, automation, compilation, dexy (tag), and /technical-writing/filters. I’m also working on ways of marrying this wiki with the blog, by way of the critical futures section. It’s in progress, of course but there’s a blurb at the bottom of all CF posts that says:

Disqus comments are provided to support legacy comments on old posts, and as a last resort in cases where other means of communication are ineffective. Otherwise please consider leaving a comment.

The link doesn’t work as well as I want to, and maybe I need to set up a specific comments page adding widget, as on the submissions page. Couldn’t hurt. Also on my list of things to do: make a more fitting index page that draws attention to all sorts of content on the wiki, not just rhizome. That’s on the list.

Also on the list:

Writing fiction like crazy: I’m working on the penultimate chapter of the novel.
Something with the Cyborg Institute. It needs to happen.
More regular blogging here. It’s on the list!

What We Learn from Wikileaks

Tue, 08 Feb 2011 00:00:00 +0000

Wikileaks, and the drama that has surrounded it for the past few months, brings forth images of the Internet as a very lawless and juvenile place, exactly the kind of thing that the cyberpunks of the 1980s were predicting. This isn’t always far from the truth, but the story of spies and international espionage, and digital attacks and counter attacks may distract us from thinking about other issues. Obviously Wikileaks causes us think about censorship, but issue of publishing, journalism, audience and community participation, transparency, and globalism in the digital context are also at play. Lets take the highlights:

Censorship

In the print world, I tend to think of post-facto censorship is incredibly difficult. Once print copies of something exist they’re there and it’s hard to get every copy and you can’t get people to “unread” what they’ve already seen. In the digital world, it’s really difficult to get content taken down, and once there are copies in people’s hands, the cost of making additional copies is low enough that censorship stops working. Right?

I suppose the appearance of multiple mirrors and copies of Wikileaks post-takedown proves this, but it also proves that there are aspects of the world wide web that are not decentralized and it’s possible to pull a domain and site off the Internet at a single point. That’s a very scary proposition. While information survives, I think many people thought that “you couldn’t effectively censor the Internet,” and Wikileaks says “yes you can, and here’s how.” (cite)

In response I think people have started to think about the shape and structure of the network itself. The Internet is designed to be resilient to this kind of thing, and this is a very startling example that it’s not.

Publishing, Journalism and Wikis

I suppose the thing that I get most offended by “Wikileaks” for is the appropriation of the term “Wiki,” because in the current (or last) form, Wikileaks wasn’t a wiki. Content was not edited or supervised by a community, the Wiki process didn’t allow the site to provide consumers with a more diverse set of opinions, it didn’t increase transparency. In this respect, Wikileaks mostly resembles a traditional “old media,” publication.

Once wikileaks stops being this radical new journalistic departure, and this community-driven site, what remains may be pretty difficult to reconcile. Is it useful for journalists to publish raw source material without analysis? What audience and purpose does that serve? The censorship of Wikileaks is problematic and requires some reflection, but there are problems with wikileaks itself that also require some serious thinking.

And just because it’s a sore point, particularly since Wikileaks is/was a traditional publication and is not a platform for independent speech, we have to think about this as “freedom of the press” issue rather than a “freedom of speech” issue.

Community Involvement

The involvement of a community in Wikileaks, is over-shadowed by the groundswell of “community” activity by Anonymous and other groups. Look to Gabriella Coleman (twitter) for more thorough analysis. I don’t have any answers or conclusions but: the role and effectiveness of (distributed) denial of service attacks in this instance is really quite important.

Usually DoSes are quick, messy and easily dealt with affairs: DoS someone, get noticed, target and attacking addresses get taken off the air and people loose interest and things return to normal. This back and forth, seems a bit unique (but not unheard of) the fact that the 4chan/Anon gang picked Wikileaks jives with their ethos, but it is impressive that they were able to get organized to support Wikileaks. I find myself curious and more surprised that someone was able to, at least for a while, throw something in the neighborhood of 10 gigabits (an unverified number, that originates with Wikileaks itself, so potentially inflated) at the original Wikileaks site. That’s huge, and I think largely unexplored.

Transparent Operations

In July, Quinn Norton wrote about transparency and wikileaks, basically, that exposing information doesn’t solve the problem that governments don’t operate in a transparent manner, and that access to documents and transparency are the result of a more open way of doing business/government. Particularly in light of this, the fact that wikileaks focuses on data dumps rather than more curated collections of information or actual analysis, is all the more problematic.

Similarly, “wiki” as practiced in the original model (as opposed to wikileaks,) is about editing and document creation in an open and transparent manner. Thus the issue with wikileaks is not that they have/had a professional staff, but that they didn’t disclose their process.

Onward and Upward!

Learning Python

Mon, 07 Feb 2011 00:00:00 +0000

I’m not a programmer. I find myself saying this with great frequency. Which is weird, because I end up writing a fair amount of content that looks like (and probably is) code. Almost always this takes the forms of little shell scripts that do the things that I would other wise have to do by hand.

Very early on in my development as a programmer and Linux user, bear told me that the greatest thing about the shell was that once you figured something out, you could save it and reuse it whenever you needed it, and you never had to figure out that thing again. That’s sort of the basis of how I live my life with computers these days.

And it works great, and it means that I’m pretty comfortable with a lot of really powerful shell tools. I’m personally amazed by everything that I can make sed do. The problem is that there are limitations to what one can do with shell programing (nothing for the web, and no way to edge yourself into object orientation) or other more advanced programming techniques. Though certainly there are a lot of things that I could do using the shell, I sort of know enough to know the bounds of what’s possible and what’s absurd.

So I’ve been trying to teach myself Python. Because it seems like a good option. Though I’m not a coder by any stretch, I read a bunch of code, and know enough about all of the various languages to appreciated Python and it’s possibilities. Seems like a good neutral option and I’m pretty sure it can do everything I want or need it to for a while, and the worst case scenario is that the skills will be transferable if I need to change later.

My first “project,” I think is to rewrite build novel to Python. I get what needs to happen, and it mostly just mangles some text files into a number of other files to keep source and publication versions of Knowing Mars in step with each other. The hope being that the script will be a bit more resilient, and also be more easily generalized to publishing other projects as I have them to publish.

I’ve gotten the hang of mangling variables and reading settings from configuration (YAML rocks, and is killer easy to use.) Next up, learning more about creating, writing, and copying files, with a likely side course in regular expressions for Python. I’ll keep you posted.

:author: tychoish

Sat, 05 Feb 2011 00:00:00 +0000

The Price of Free

Eventually, the Facebooks and Twitters of the world may be dethroned and replaced by provider-agnostic protocols, in much the same way AOL is no longer synonymous with email. However, no one is going to build protocols that threaten their bottom line. So long as “free” is paid for by surveillance, the Internet will represent a Faustian bargain for radical social movements.

-- The Price of Free

Creature of Habit

Thu, 03 Feb 2011 00:00:00 +0000

I’m doing my commute somewhat differently this morning. Same wake time, same destination, and many of the same trains, but a special event necessitated a different pattern. It’s amazing how much this shift has thrown me out of sync with my day. Not in a bad way, but in the way that I was sort of jittery about missing my train in a way that my normal patter has really ceased to evoke.

I told my fellow commuter that, while I could probably live without the time expenditure of the commute I’d grown fond of the rhythm and structure of the commute. It’s nice to have my body in programmed to get up at a specific time, it’s nice to have a fixed departure time to encourage me to get work done early and effectively, it’s even nice to have some firewalled time on the train to get things done like emails and blogging. Obviously if it was possible to change some things, I would, but all in all its not as bad as it could be or as it may sound.

The main take away lesson from this is: that habits and patterns make it easier for us to cope with the world. And not just in a “habits make stressful situations, like commutes and compulsions easier to live,” but having a routine established make it easier to spend your available time and brain cycles doing things you really care about rather than on keeping your head above water.

Also, I think as a secondary point, being able to take a block of available time and accomplish something concrete is an incredibly valuable skill. One of my biggest frustrations is with “dead time” where I’m waiting for something to happen, and I can’t do anything with that time. Some technology helps with this: long battery life and systems that suspend/resume quickly and preserve “state” are great. Having a list of things that you can do that are easy to pick up and for a number of different situations, with different amounts of free time. Ideally, you don’t want to have to spend time that you could be doing something, on thinking about what you ought to be doing.

Which I suppose is just another way of approaching habits.

The Ones that Stop at North Broad

Thu, 03 Feb 2011 00:00:00 +0000

There’s something about public transit, particularly train systems, that seeps into the riders. Little facts that make getting around easier. “You’re waiting for the R3? It’s always late,” they’ll say, or they’ll be able to tell you which stop is next by looking out the window in what looks like total blackness. Routines do that to you, I guess.

I was on a train last week, not my usual evening train--which is a short turned local train that ends one stop past mine--that only has two intermediate stops before my stop, and as we were getting of the train, a group of regular shad a conversation that resembled the following. I suppose I should also preface this by saying that we were all sitting near a person who discussed their health insurance coverage woes at great depth and volume with their seatmates.

“It’s nice to take these express trains, I usually just get on whatever’s next.”

“If this counts as express.”

“I mean, it’s better than the ones that stop at--”

[unison] “--North Broad.”

See, there’s this stop in northern Philadelphia that a lot of trans pass, but very few stop at, and hardly anyone gets on or off at. It’s near a subway stop too, which decreases it’s usage. It’s main purpose, I think, is to irritate those of us who *just want to get where we’re going.

It’s also, apparently, good for allowing irritated and tired commuters to commiserate over the unreclaimable minutes of their lives that they loose on a regular basis. It’s these kinds of interactions and phenomena that make urban living so attractive.

git readd and git aliases

Wed, 02 Feb 2011 00:00:00 +0000

git is a version control system that does a number of things rather “differently.” By now, I suspect most people are familiar with git so I will refrain from focusing too much time here on explaining what git is or how it works. Git does things differently, and to make a long story short, git has no concept of a file rename. To be honest git doesn’t have a lot of sense of individual files at all, but that’s another story. As a result git looks at the contents of the files, and if a new file is similar enough to an old file that was deleted git recognizes this as a “rename,” or a moved filed. It can process file copies in the same manner.

You’d think this would be a problem, but it turns out that it isn’t. In fact it’s a great thing. You can store a bunch of data on a file system, use conventional tools to manipulate, move, and rename those files in the normal manner, and then do a little magic, and git automatically knows everything that it needs to know about what files were renamed and when.

This post is about that magic.

What you need to do is find out the names of the files that existed in the last commit but have been removed from the file system. If you run “git rm” on these files and then add everything that remains, git will be able to pick up the rename operations implicitly, and the next commit will reflect the current state of the file system.

I’ve thrown this little snippet in the code section of the site: `git-readd </code/git-readd>`_.

I’ve used this process as part of scripts in the past, and it’s great for managing Maildirs with git, but I found myself needing this operation from time to time as I’m manging a git repository. Enter, “git aliases.”

Aliases let you create git sub commands from your own scripts. Add the following lines to the ~/.gitconfig file on your system, and create it if necessary:

[alias]
readd = "~/scripts/git-readd"

Now you can run this command in any git repository on your system with the following command:

git readd

Magic!

make all dexy

Tue, 01 Feb 2011 00:00:00 +0000

See “Why The World Is Ready For Dexy” and “Dexy and Literate Documentation” as well as the technical writing section section of the tychoish wiki for some background to this post.

The brief synopsis: dexy is a new tool for handling the process of the documentation work flow between writing and publication. It takes snippets of code, and bits of documentation and passes these atomic chunks through filters to generate technical articles, manuals, and other high quality resources. It’s exciting because it may provide a way to write and manage the documentation process in a manner that is more effective than many other options and has the potential to produce better quality technical texts.

The reason, I think, is that dexy treats documentation like code. This is different, fundamentally, from systems that expect that developers write documentation. The former has interesting implications about the way technical writers work, and the later is nearly always a foolhardy proposition doomed to end in failure.

Documentation has a lot in common with code: documentation is written and revised in response to software versions, so the process of iterations has a lot in common. Documentation should typically be compiled, and the build process should produce a static product, between iterations. Documentation, like code, must also be maintained and fixed in response to new bugs and use-cases as they are found/developed.

If we accept this analogy, Dexy begins to look more like a tool like make which is used to manage compilation of code. make figures out what source files have changed, and what needs to be rebuilt in order to produce some sort of binary code. That doesn’t sound like a very important tool, but it is. make makes it easy to automate tasks with dependencies, without writing a bunch of novel code to check to see what’s been done and what has yet to be done, particularly when build processes need to be done in parallel. Furthermore, make is one of these typical “UNIX-like” utilities that does only one thing (but does it very well) and ties together functionality from a number of different kinds of programs (compilers, configuration tools, etc.)

Dexy is kind of like that. It manages a build process. It ties together a group of existing tools, thereby saving developer time and building something that can be more flexible and consistent.

This is, however, imperfect analogy: I think Dexy isn’t “make for documentation,” because it would be possible to use make to manage the build process for documentation as well as code.¹ Dexy manages text processing, make can work one level up--if needed--to build coherent texts from Dexy-processed documentation units. Dexy and make are glue that turns documentation and code into useful resources.

There are obviously some situations where this developer-like workflow may be overly complicated. For starters, Dexy, like make, really provides a framework for building documents. A portion of creating every project and text in this manner would necessarily go to developing build-related infrastructure. It’s not a huge burden, but it’s the kind of thing that requires a little bit of thought, and maybe some good “default” or base configuration for new projects and texts. Dexy is a great first step into a new way of thinking about and working with documentation, but there is much work yet to be done.

Onward and Upward!

I should thank, or perhaps blame, a coworker for planting this idea in my mind. ↩︎

Rule By Community

Tue, 01 Feb 2011 00:00:00 +0000

When Oracle acquired Sun Microsystems, there was a lot of (warranted) concern for what would happen to the Open Source projects that Sun started, supported, and championed. Though the acquisition began years ago, now, I think we’re still in the process of figuring out how all of this filters down. The concern, largely grows from the fact that while Sun licensed a lot of work under free and permissive terms, and was an active contributor to communities, Sun (and now Oracle) remains the ultimate owner of that work moving forward. Thus, Free Software/Open Source projects are somewhat stranded because the leadership of these projects were supported by Sun, are supported by no one.

In response, there’s been a lot of movement among these projects to formally establish “rule-by-community” systems, that give community contributors authority and voice in the management of these projects.

Nevertheless, it feels a lot like writing policy to save oneself from the mistakes of the past. It also seems like an action of democracy fetish rather than bottom up democratic organization. I’m always struck by how not democratic open source projects are, in practice, and how little this matters. Python and Perl are iconic examples of this, but most other projects have one or two people who are in charge of everything, and while they know that their “rule” is subject to the confidence of the community, their leadership isn’t terribly democratic. I think this recent event in the org-mode community is a particularly useful example.

Maybe Sun was inflating an Open Source bubble of sorts, maybe corporate-sponsored open source, and community/consultancy-sponsored open source are different things that need different sorts of approaches.

Knitting Pattern Complexity

Wed, 26 Jan 2011 00:00:00 +0000

I’ve been writing a series of terribly self indulgent posts about thinking about creative projects as work and about the habit and rhythm that I’ve been using to get work done, even under adverse circumstances. As exciting as the increased blogging and progress on fiction projects is, I’m just as excited about figuring out how to squeeze in knitting and reading time into my days.

And not to overshadow the importance of figuring out how to make the best of my time, but the other major factor in my return to knitting is that I finally finished the plain sweater that I’d been working on for 7 months (or so) and got to cast on a sweater with a two-color pattern, in a nifty yarn, and a pattern that’s strongly reminiscent of my favorite sweater.

I’ve discovered that while plain knitting is soothing, in many ways, it’s not particularly fulfilling or engaging. I’ve knit--at time of writing--about 9 inches of the body section. This means, I’m about half way to the under arm of the sweater. I don’t know if this will linger, but at the moment, I’m so looking forward to this sweater, that I think I might make a series of sweaters based on this, or similar patterns.

I suspect I’ll be blogging about this project for a little while, but here are the basics: the pattern is Swedish/Norwegian in styling. The yarn are black and a hand-dyed, from a dyer in Wisconsin that I had made specifically for this project. The background color is called “tree bark,” and it’s very autumnal and the foreground color is black, so it has a sort of stained glass/water color look to it. The contrast isn’t as strong as I had hoped, but the base yarn is similar to the “Shaffer Anne” sock yarn (if that’s meaningful to any of you,) and it feels lovely.

Good enough for me!

Documentation isn't Content

Tue, 25 Jan 2011 00:00:00 +0000

See “Why The World Is Ready For Dexy” and “Dexy and Literate Documentation” as well as the technical writing section section of the tychoish wiki for some background to this post.

Let’s establish some basics. Content, as I think of it, in the context of new/web/digital media, is all of the stuff we read and right on the web. Documentation are those texts which supports the use and creation of technical tools, and explains technical concepts. While obviously read literally, documentation is content, but I think the way we’ve come to understand other kinds of digital content provides an incomplete basis for understanding how technical texts are published and consumed on line.

Consider the following assumptions that we can make about most forms of content on the web:

The basic unit of content is pretty short. 1000-1500 tops is the upper boundary for most blog posts and articles, and while some kinds of content can sneak by with slightly longer units--particularly in well structured contexts--these are exceptions.
Most content on the web is time-sensitive. While everything gets archived, the focus of publication is often on volume, which increases the chance of producing something that “goes viral” and gets a lot of attention. All other things being equal, the most successful on-line publishers are the ones with the shortest publication processes and the most regular publication cadences. After a short period of time, what’s in the site archives is probably largely irrelevant.
Given the way that some content competes with other content, success is often determined by specialization, and the tightness of focus. It’s easier to be the loudest voice in a very small room than it is to be the loudest voice in a large room or a lot of small rooms. Content, thus, needs to be very focused and address very niche interests.

in contrast:

Documentation texts tend to be pretty long, and while there are some “quick reference”-scoped texts, and some very complete texts that are quite long, on average documentation is substantially longer than “content.” This means it needs to be produced differently, and we can expect different usage patterns.
In general people don’t read documentation. This isn’t just that people don’t “rtfm,” but that if generally people’s interaction with a piece of documentation begins with a specific question or problem. They don’t say “oh, that manual for $xvz product looks interesting, i’ll read it,” and i think the “i should read the documentation for $uvw before i begin doing $task,” is much less common than we’d like to think.

People read documentation very tactically. So it’s important that documentation exist and be complete, but we should have no illusions that people read any documentation from beginning to end as “clean slates.”
Documentation is always already very tightly focused, unlike content. While some technical publishers may publish “second hand documentation,” and thus be able to focus on documenting different aspects of the user experience, most documentation producers must aim to cover as much as possible, and allow users to find and take advantage of whatever information that is most useful for them.

As a result, it’s absolutely crucial that we don’t think of and produce documentation as being crucial. I think treating documentation as something that needs to be compiled. is probably the first step in “doing right” by documentation. Build tools like dexy, similarly, are great because they let writers and developers produce documentation in ways that make sense.

If you write or produce documentation--and better content as well--I’m interested in hear what you think about these issues. Onward and Upward!

Specific Rhythms

Mon, 24 Jan 2011 00:00:00 +0000

One part of finding momentum and working to dedicate a bit of energy to some projects that I’ve been neglecting for a while. I’ve written a bit about finding the rhythm of my current routine, I’ve not written about what I’m actually doing. I hope this isn’t too boring for words.

Since the end of July, I’ve chosen to undertake a 2 hour each-way commute. Thankfully it’s all by rail, so the thought is that while I might not have a lot of time at home, I have more than enough time to get work done, of any sort, on the train. In practice my ability to actually get work done on the train is awful. In the past few weeks I’ve gotten a bunch better at this, and have actually managed to write fiction, get back on the blogging bandwagon, and stay on top of emails, and even reading a bit.

Here’s how a typical days works. I have a 20 minute train ride, a three to five minute walk, a 20 minute train ride, 10-15 minutes of waiting, and then 40 minutes of a train ride. Reversed in the evening. The first/last trains are sometimes standing room only. I try and spend the second and third train in the morning on the computer writing blog posts and fiction. If I can write first thing in the morning I can take it, but I always think about what writing needs to be done. On the way home I wrap up loose ends, read email, and at least a couple of days a week I get some knitting try to read and knit.

And it works. Most of the time, or at least enough of the time, for me to be able to notice that I’m actually doing things. I think as much anything, I’ve been able to get a much clearer idea of what I can expect to accomplish in any given time span, which makes it much easier and more productive to make a todo list. I think the main purpose of keeping a todo list is to focus working times so that I never find myself thinking “I have a few free moments, what should I work on today,” because the list tells me these things. If I know how much time I have, about, I know how to best spread out tasks for optimal working without ever needing to waste time wondering what I ought to be working on. It seems to work.

In the end, the novel keeps getting longer and closer to the end. I’m starting to think that I may be a bit too ahead of myself on the blog. My sweater-in-progress gets longer, and the progressing bar on my kindle progresses. That’s worth something.

Infrastructure and Place

Sun, 23 Jan 2011 00:00:00 +0000

I’ve been playing around with writing a story or a few stories about transpiration infrastructure,

I spend a lot of time on trains. And I in a part of the world where there’s a lot of transportation infrastructure, but most of it is nearly 100 years old. So while it’s nice that there are trains, it’s also apparent that the trains are designed to serve a reality which is radically different from the one that people today occupy.

I think I’m drawn to stories about people living in worlds that are out of sync with their reality. The story I’m writing now revolves around this idea, using the problem of relativistic space travel as a device to pull people “out of time,” but I think there are ways that people’s relationship to their present is shifted without science fictional devices: by history, by political rhetoric, by development, and by illness.

The two ideas I’ve been playing with recently are:

a story about the people who build some sort of railroad system on a new colony or outpost. I’m thinking Mars Trilogy (Robinson) meets Heart of Darkness meets Left Hand of Darkness (Le Guin).
a story about a commuter on the Moon mid/post-Accelerondo. Sort of “Philadelphia in Space.” This is probably more of a short story/vignette sequence, and I have a little bit sketched out but it’s all in first person and I need to make that not be the case.

Art Work

Fri, 21 Jan 2011 00:00:00 +0000

I’ve written between 100 and 300 words on this novel that I’ve been working on for two and a half years, for the past two weeks. After too many months of writing much less than this, and I’m pretty sure that it’s been much longer since I’ve been able to write this regularly (regardless of length) on a fiction project. This feels amazing, and for the first time in forever, I feel like a real writer. Which is kind of an amazing thing.

This Project; My Life

Even when I’m not writing fiction, I’m pretty prolific, so I suppose, I almost always feel like a writer. At the same time, writing endless emails and writing stories is very different. I think it takes a few months for life to settle down, for new routines to establish themselves, and the for habits to emerge that make it possible to write. This project has lingered along for way too long, and while I’m still interested in it, and interested in the ideas that it raises, I’m in a very different place in my life right now, and I would like to put this to bed so I can work on other things.

I want to write short stories, and work on pulling together application together for Clarion (or Viable Paradise?) for 2012 or so. And I have this story that I’ve been pondering for months about non-normative family geometries and adult relationship structures (in the way that Le Guin’s Left Hand Darkness addressed similar issues regarding gender.)

I’m putting together the final scenes of Chapter 10. I think I have another week or two here. Then there’s a lot of explosion and general climatic goings on in 11, and I’ve only planned 12 chapters. Actually in a moment of foresight, I planned out 11 chapters, skipping Chapter 9 and leaving the words “overflow here.” And really, the prospect of being able to write the awesomeness that is Chapter 12 has been the main thing that’s kept me going through all of this.

Art as Work; Art as Habit

There are two parts of this. First, I think art isn’t the kind of thing where you take raw talent, and wait for divine inspiration, and find quality creative work. No. Doing art is about participating in conversations, about pushing yourself to create better work, learning skills, and figuring out a way to make a living doing your art (or something related) so that you’re able to continue to be involved and productive. Art is hard work, and perhaps that’s (part) of why it’s so important. I’m okay with this.

I am also a very strong proponent of the idea that being successful as a creator/writer/artist--is less about having “great days” and more about getting a certain amount of writing done every day. The hardest part of writing is always in getting started. Once you have a little momentum, it’s easy to write a modest amount every day, and that’s the kind of writing that adds up and makes novels.

At least that’s the plan.

Updates: Tech Writing and Wiki Gardening

Wed, 19 Jan 2011 00:00:00 +0000

Yesterday, I posted over at Critical Futures the second of, an apparent, series of four posts on dexy. Dexy is cool because it’s a pretty nifty tool for doing documentation in a new and potentially very powerful manner. While I think playing around with Dexy is cool and important (and I’m having fun tinkering a little, seeing the world in a slightly different way and then running away to absorb and re-assess) it’s also very important to think about the reasons why tools like Dexy are incredibly important.

Read on for: `Dexy and Literate Documentation <http://criticalfutures.com/2011/01/dexy-and-literate-documentation>`_

Also, as part of this post and series (archive recently updated!) I’ve been developing a technical writing section of this wiki, which has included a couple of little pages and snippets that I’ve been hanging onto for a while, and a number of pages that I think I’d like to write wiki pages on. These include three major issues in technical writing and the tools used to build documentation resources that build tools that yesterday’s addresses. They are: /technical-writing/atomicity or smaller “atomic” units of documentation, /technical-writing/compilation or generating documentation statically before publication rather than dynamically on view, and flitering as an approach to document generation.

I’d like to perceptively apologize if my blogging here is not particularly interesting, or topical except in a very “meta” sort of way. One of my primary goals for tychoish.com-as-a-wiki is to be able to keep track of and document the work that I’m doing on line that isn’t necessarily blog-post related. Blogging is great, but it’s hard to balance writing blog posts and providing a good media outlet and the kind of work required for developing wiki pages and moving forward on other sorts of projects. These posts, in addition to spreading links, and offering short partially formed thoughts, will likely serve to draw attention to various things I’m working on.

The idea of wiki gardening isn’t new, but I like it. Wiki’s are so simple and so easy to create that the only thing you can really do to cause a wiki to fail is to neglect it. Not that the proper care and feeding of a wiki isn’t challenging, but it is simple. In any case I suspect many of the meta posts will be “gardening” posts in the sense that they’ll draw attention to recent edits of wiki pages and note the new and developing pages in solicitation of your contribution.

Shovels up!

Dexy and Literate Documentation

Tue, 18 Jan 2011 00:00:00 +0000

See “Why The World Is Ready For Dexy” for the lead into this post. The short version: most tools for building documentation are substandard, and most attempts at “fixing documentation processes” are flawed. But there’s this new project called “Dexy” that is doing something that is pretty exciting.

Basically, Dexy is a text filtering framework, you write documentation, code samples, and code, and then you tell Dexy how to stitch everything together, and bingo. It’s success, or potential success, is built around its simplicity and flexibility.

This model Dexy proposes something called “Literate Documentation,” which is a cool concept, which expands upon the notion of “literate programming,” both concepts require a little bit of unpacking.

Literate programing is the idea that code, documentation, and all specifications should be contained in one file, with blocks of machine readable code and human readable text should be interleaved with each other. Literate programming tools, then take this “mega source” and build programs that do cool things. There are a number of literate programming tools, and some notable programs that are written in this manner, but it’s not particularly popular: code and text tend to flow in different ways, and a manageable literate programming text, is often not particularly maintainable software.

Literate documentation, on the other hand, as implemented by Dexy is documentation where the documentation is compiled from an amalgamation of text and code which can be run and tested at build time. You write code snippets and documentation snippets, and a tool like Dexy takes all of it, runs the code and stitches together a document out of all the pieces. Then, anytime you need to make a change to the code, or the text, you rerun Dexy and the documentation mostly tests itself. Good deal.

I think it’s not yet obvious if Literate Documentation will actually be a “thing.” It’s a great idea but, like literate programming, it’s unclear of how this kind of practice will actually catch on, and how useful/feasible writing documentation will be in this manner. “Dexy the method” may or may not find greater acceptance, because “Literate Documentation” may depend on developers writing documentation. At the same time I think that “Dexy the tool,” is certainly a valuable contribution to the field of technical writing. Nevertheless, I think there are some important things about the way Dexy works that are worth extrapolating.

(Links to `tychoish wiki <http://tychoish.com/readers-guide/>`_ pages concerning `technical writing <http://tychoish.com/technical-writing/>`_ in some state of existence.)

Atomic Documentation. Dexy reinforces the idea that documentation should be written in very small units that are self sufficient, and address very small and specific topics, questions, and features. The system which builds and displays documentation should then be able to either usefully present these “atomic” units or stitch more complete documentation together from these units. This makes documentation easier to maintain, and arguably makes documentation more valuable for users.
Compiled Documentation. The “end-product” Documentation should be statically compiled, unlike (most) web-based content that is dynamically generated. This allows writers and teams to verify the quality of the text prior to publication, and allows the “build system” to automate various quality control tests. Documentation is particularly suited to this kind of display generation because it changes very irregularly (no more than a few times a day, and often much much less often.)
Pipes and Filters. the process of publication can--like code--is basically passing text (and examples) through various levels of processing until the arriving at a “final product.” Dexy is very explicit about this and provides writers/developers a framework to manage a complex filtering process in a sane manner.

I look forward to thinking about these aspects of documentation and documentation systems, and about how writing texts with Dexy, or in the “Literate Documentation,” mode affects the writing process and the shape that texts take. I look forward to hearing your thoughts in the comments or on the wiki pages!

Laptop Queuing System

Fri, 14 Jan 2011 00:00:00 +0000

Here’s the problem: most of my computing happens on laptops which are both unreliably active (i.e. suspended) and also have unreliable network connections. (i.e. trains, etc.). I’ve done a lot of work to make it possible for my digital life to continue without interruptions. This includes writing cron jobs that exit before performing network intensive operations and making sure that most data can be downloaded and consumed in offline formats. But this is not quite ideal.

And I thought, “wouldn’t it be great if, I could just bundle up tasks into little chunks and throw them in a pile that the computer will process and perform when it can, so that I don’t have to remember to do things when I have a network connection and I can just drift in and out various states (active network, low-quality network, no network, and suspend/resume) without worrying about managing these states.” Sort of like a cross between cron and some sort of queuing/bus system.

Wait. A queue.

As many of you who are still reading know, the contemporary solution (and, actually historical as well, but we won’t get into that.) to enabling web applications to scale to be able to handle large amounts of work is to use queuing systems which allow applications to distribute and absorb bursts of activity, by spreading work out between high and low utilization periods. Basically: if you don’t have to do everything all at once, split everything into little “atomic” jobs, make a list and then process, and do jobs as you can until everything is done.

These are high performance systems, meant to handle nearly incomprehensible amounts of activity, but the idea is the same: I (and perhaps you too?) need a system that can figure out what state a machine is in and can save tasks and run them when the conditions are right. Simple, right?

Anyone want to work on this with me? Come on, it’ll be fun!

Link Storm

Fri, 14 Jan 2011 00:00:00 +0000

I have a bunch of links that have been hanging around too long, that I’d like to share with you, so here they are. Enjoy!

Distrust Simplicity is a great short form blog, that I’m really liking.
The Singularity Summit is a cyborg/futurist event that a reader wrote me a note about a rather long time ago. If you’re interested in the singularity and issues and ideas related to that.
Hypo a literate programing (with org-babel) based asset management system for game development. (hypo example.
Common Lisp Support in Org-Babel. I feel like there’s probably a joke here.
Mango.io a markdown-based CMS, using Python tools. I’m not sure if it’s the kind of thing that I’m likely to ever want to use myself but it’s an idea.
A New York Times Article about Podcasting and Leo Laporte’s TWiT Network While the content of this article is interesting in it’s own right, there are a couple of “bigger” picture things happening. First, the old media (i.e. “the Times”) covering the new media (i.e. “TWiT,") is always interesting. Secondly, and less obviously, it’s interesting how the biggest successes in the “New Media,” are by veterans of the “Old Media,” (like Laporte, who had a career in Radio and television before doing TWiT.)
Social Text Journal: The Dramatic Face of Wikileaks, is a meta-meta-meta look at wikileaks and the “new media” moment that it represents.

Make All Novella

Fri, 14 Jan 2011 00:00:00 +0000

When I decided to publish my novella Knowing Mars, I decided that I wanted to use my existing publication system and that I wanted to automate the process of generating of all the necessary versions that made it possible to keep my original files in sync, without needing to duplicate effort. I decided that the few hours it would take to write a script would both save a lot of time later and make it more likely to maintain the text.

So I have this script that:

Copies the source files into the publication directory.
Generates: full html files for every chapter, plain text files for every chapter, a full html edition of the complete text, a plain text edition of the complete text, a minimally styled html edition of the complete text.
Keeps these editions synchronized.
Keep the original source files synchronized.
Ideally provide a tool that would prove useful in the future.

I’ve included the code of what I came up with in this wiki, at `</code/build-novel>`_, and you can find the full source of tychogaren.com and Critical Futures here. For the full source of theKnowing Mars text consider the gitweb. I would be very grateful for any feedback or input.

Why The World is Ready for Dexy

Tue, 11 Jan 2011 00:00:00 +0000

At one time or another, I suspect that most programmers and technical writers have attempted to “fix” technical writing in one way or another. It’s a big problem space:

Everything, or at least many things, need to be documented, because undocumented features and behaviors cause problems that *one really ought not need to review the source code and understand the engineering to fix (potentially) trivial problems, every time the occur.
The people who write code are both not suited to the task of writing documentation because writing code and writing documentation are in fact different skills. Also, I think the division of labor makes some sense here.
Documentation, like code, requires maintenance, review, and ongoing quality control, as the technology and practice change. That’s a lot of work and particularly for large projects, that can be a rather intensive task.
Lots of different kinds of documentation are needed, and depending on the specific needs of the user, a basic “unit of documentation,” may need to be presented in a number of different ways. There are a number of ways to implement these various versions and iterations, but they all come with various levels of complexity and maintenance requirements.

The obvious thing to do, if you’re a programmer, is to write some system that “solves technical writing.” This can take the form of a tool for programmers that encourages them to write the documentation as the write the code, or it can take the form of a tool that enforces a great deal of structure for technical writing, to make it “easier” for writers and programmers to get good documentation. Basically “code your way out” of needing technical writers.

You can probably guess how I feel about this kind of approach.

There is definitely a space for tooling that can make the work of technical writing easier, as well as space for tools that make the presentation of documentation clearer and more valuable for users. Tools won’t be able to make developers to write, at least not without a serious productivity hit, nor will tools decrease the need for useful documentation.

It’s a difficult problem domain. While there is a lot of room for building programs that make it easier to write better documentation, the problem is that the temptation to write too much software is great. Often the problems in the technical writing process, including high barriers to entry, complicated build/publication systems, and difficult to master organizational methods, which are easy to address in programs. Meanwhile, most of these issues can be traced to overly complex build tools and human-centered problems, which are harder to address in code.

And since documentation takes the form of simple text, which seems easy to deal with, developers frustrated by documentation requirements, or technical writing teams, are prone to trying to write something to fix the apparent problem.

Which brings us to the present, where, if you want to write and publish documentation, your choices are:

Use a wiki, which isn’t documentation but the software generally does a good job of publishing content, and wiki engines mostly don’t have arcane structures of their own that might get in the technical writer’s way. Downside: it’s the wrong tool for the job and it forces writers and editors to maintain style themselves across an entire corpus, which is difficult and eventually counterproductive.
Use some other existing content management system. Typically these aren’t meant for documentation, they have difficult to use interfaces, because they’re meant to power websites and blogs, and they almost impose some sort of structure (like a blog,) which isn’t ideal for conveying documentation.
Use an XML-based documentation tool-set. This is probably the best option around, at the moment, as these tools were built for the purpose of creating documentation. The main problems are: they’re not particularly well suited for generating content for the web (which I think is essential these days) and as near as I can tell they make humans edit XML by hand which I think is always a bad idea.
Build your own system from the ground up. Remember text is easy to munge and most of the other options are undesirable. Downside: homegrown projects take a lot of time, they’re always a bit more complex than anyone (except the technical writers?) expect, and it’s easy to almost finish and that’s bad because half-baked documentation systems are most of what get us into this problem in the first place.

So it’s a thorny problem and one that lots of people have (and are!) trying to solve. I’ve been watching a tool called dexy for the last few weeks (months?) and I’ve been very interested in it’s development and the impact that it, and similar tools, might have on my day-to-day work. This post seems to be the first in a series of thoughts about the tools that support technical writing and documentation.

Wiki Blogging

Mon, 10 Jan 2011 00:00:00 +0000

I’ll probably do a fair piece of this “metablogging” thing here, and I’m sorry.

Also, I totally intended for rhizome to be a much shorter form blogging project, and while the posts are shorter here than at critical futures, they’re not exactly short, and I’m not exactly as prolific as I might like to be.

I think I might just be somewhat long-winded.

I’ve also found that I’ve mostly not succeeded as using the wiki functions. Which is probably as much a result of my minimal posting as it is a function of my inclination.

Though I do realize that I’m pretty set in my existing blogging habit, and it takes a lot for me to break out of this form. Though I think it’ll be good to try. You can help by editing pages and continuing conversations that I start. I think I need to tweak some of the templates, to include “discuss this further” links (both here and on critical futures.) to remind people that anyone can edit and contribute.

Also you should all check out On Wireless Data, if you haven’t already. This is a post that I wrote a while back (and posted last week) about the that the technological constraints of wireless data networks constrains and effects the kinds of applications that are developed for these kinds of environments. This in turn affects the ways that people use technology. Which is interesting (and important) to think about.

And that’s all for now. I’ll see you around later!

Upgrade SBCL and SLIME

Wed, 05 Jan 2011 00:00:00 +0000

This is a little bit of documentation/technical writing around an issue that I had for a while. SBCL is a Common Lisp implementation that I use, and would recommend as a good starting point for people interested in tinkering with Common Lisp. SLIME is an emacs-based development tool kit that lets you interact with a lisp session in real time.

SLIME works as you’re writing code and makes it possible to connect to (potentially any) running lisp process and execute code and access documentation, among other functions. The connection between Emacs/Slime and the running application is provided by a connector called “Swank.” Lisp is pretty cool, Common Lisp is really nifty, but SLIME is what makes working with Lisp fun/easy and really powerful.

Ok. Here’s what happens. You upgrade SBCL (which happens every now and then for me with Arch Linux,) and you probably have to recompile a number of things to work with the new package. That’s a bit annoying, but it’s not a huge burden. Then you try and load Swank, and it bombs. You reinstall Slime but no dice, and you still can’t connect to your application in slime.

This is where I lingered for about 3 months. No working Slime meant going back to interacting with my lisp applications in the conventional manner, which kind of sucked.

Here’s the fix, and it’s crazy simple. In emacs run “M-x slime” Restart the application or reload the swank loader, and then try and connect to the application with Slime. Bingo.

Turns out that Slime (and therefore swank) build a few .fasl files that are version specific to SBCL that hang around after the upgrade. The only way these files are rebuilt is if you load slime, which you may only do by way of swank, which won’t work unless you reload slime. It’s a chicken and egg issue.

Problem solved. Sorry it wasn’t more interesting: most aren’t terribly.

On Wireless Data

Tue, 04 Jan 2011 00:00:00 +0000

It’s easy to look around at all of the “smart phones,” iPads, wireless modems, and think that the future is here, or even that we’re living on the cusp of a new technological moment. While wireless data is amazing particularly with respect to where it was a few years ago--enhanced by a better understanding of how to make use of wireless data--it is also true that we’re not there yet.

And maybe, given a few years, we’ll get there. But it’ll be a while. The problem is that too much of the way we use the Internet these days assumes high quality connections to the network. Wireless connections are low quality regardless of speed, in that latency is high and dropped packets are common. While some measures can be taken to speed up the transmission of data once connections are established, and this can give the illusion of better quality, the effect is mostly illusory.

Indeed in a lot of ways the largest recent advancements in wireless technology have been with how applications and platforms are designed in the wireless context rather than anything to do with the wireless transmission technology. Much of the development in the wireless space in the last two or three years has revolved around making a little bit of data go a long way, in using the (remarkably powerful) devices for more of the application’s work, and in figuring out how to cache some data for “offline use,” when it’s difficult to use the radio. These are problems that can be addressed and largely solved in software, although there are limitations and inconsistencies in approach that continue to affect user experience.

We, as a result, have a couple of conditions. First that we can transmit a lot of data over the air without much trouble, but data integrity and latency (speed) are things we may have to give up on. Second that application development paradigms that can take advantage of this will succeed. Furthermore, I think it’s fairly safe to say that in the future, successful mobile technology will develop in this direction as opposed against these trends. Actual real-time mobile technology is dead in the water, although I think some simulated real-time communication works quite well in these contexts.

Practically this means, applications that tap an APO for data that is mostly processed locally. Queue-compatible message passing systems that don’t require persistent connections. Software and protocols that assume you’re always “on-line” and are able to store transmissions gracefully until you come out of the subway or get off of a train. Of course, this also means designing applications and systems that are efficient with regards to their use of data will be more successful.

The notion that fewer transmissions that consist of bigger “globs” of data will yield better performance than a large number of very small intermediate transmissions, is terribly foreign. It shouldn’t be, this stuff has been around for a while, but nevertheless here we are.

Isn’t the future grand?

Some New Years

Tue, 04 Jan 2011 00:00:00 +0000

Typically, I’m writing my new years post on January 3rd, knowing full well that I won’t get a chance to publish it until the 4th. This probably explains how my 2010 closed and how my 2011 is shaping out.

I’ve been writing emails today and over the weekend to family and friends, making plans for various events and weekends over the next six months. There are only so many weekends, and there is so much to do. My preliminary outline for May/June is--I think this is par for the course when one is a Morris dancer--exhausting, and I have five or six months to prepare. The rest of spring is similarly exciting.

The last year was, on the whole, a good year: I have started new relationships and enriched existing ones in was that I am quite pleased with, I had a move that’s been good for me in a number of ways, I’ve been able to travel regionally a great deal, and I’ve learned a lot in my travels. It wasn’t all positive: I definitely didn’t finish the writing projects I wanted to, I didn’t knit or read as much as I would have liked to, and I have a number of personal projects that I’m throwing a lot of energy into that I hope to resolve in the next few months. But all of the low points can be directly correlated to the high points: success requires sacrifice, and on balance it’s been a successful year.

I hope that in the new year, we all are able to have great successes, without needing to make untenable sacrifices. I think the core of all new years resolutions is a wish to make our lives a little bit better than they were previously, hopefully in small manageable ways. So I’m going to keep working on making my world (and self) a better place to be me.

I hope, if you too are embarking on any kind of project like this, that you will succeed, and I look forward to sharing parts of my journey with you in this blog.

Cheers!

(real content will resume shortly, I promise!)

Obsessive Knitting

Mon, 27 Dec 2010 00:00:00 +0000

So I think I’m back to being a knitter. I started a sweater last May: something fine gage, very very plain, using my “default, this sweater is awesome” pattern in my head. It has had its ups and downs, but its a good project: and like all good projects, I’ve learned something.

First, the sweater is much larger than I wanted it to be: thankfully, it’s going to fit my roommate perfectly and I’ve been meaning in to knit him something for a while. At the same time, it’s such a fine sweater that the extra size means that it took extra long to knit.

The second thing, and perhaps more importantly, when I started the sweater I hadn’t really been knitting very much and I thought that what I really needed was something plain and simple and meditative. Apparently, except when I need distraction, plain knitting is not what I need, and I ended up being far too bored to actually want to work on this.

Thankfully, this week, I’ve mostly needed distraction, so I’ve been able to make rather impressive progress on the sleeves, and I expect to finish the last third of the second sleeve by this evening. That means I can start on new knitting projects, and I have a new sweater planed out and ready to go.

Very exciting, I know.

I got rid of a lot of yarn stash this fall--stuff that I had collected (on the cheap) that I didn’t have a project in mind for, with the hope that a smaller stash would let me focus on being able to knit on projects that I really wanted to knit. Which is sweaters, primarily sweaters in finer gages with nice two color patterns.

And so I will.

For this holiday week, in addition to the aforementioned plain sweater, I have a sweaters worth of the most amazing fingering weight yarn in two colors, and a graph for a new sweater.

It will be glorious.

Other People's Holidays

Sat, 25 Dec 2010 00:00:00 +0000

The following tweet/identica post made this morning, pretty much sums up my reaction to the incumbent holiday:

so last night I was like “why am I not getting email” and then “I better check on the server” and then “oh right #otherpeoplesholidays”

I’m getting on a plane later this morning to fly to my home town for the first time since last October. It’ll be good to see family and friends out there, and I’m always appreciative of a break from some aspects of my routine. Including the opportunity to be in my apartment during daylight for the first time in a month.

I often think of this holiday break as a “free time” to make progress on different things that I’ve had to neglect on the account of life for the fall. While obviously more pronounced when I’ve been a student, I am very much looking forward to the opportunity to get up early and write, and spending long afternoons working on a sweater.

I think I’ll have some other posts this weekend, and early next week about the knitting project.

There’ll be new Critical Futures on Tuesday as there was last Tuesday (about new technology). I hope you forgive my indulgence while I get the new rhythm sorted out.

And if this is your holiday, I hope you enjoy it!

Caffeine Dependence

Thu, 23 Dec 2010 00:00:00 +0000

Sometimes, particularly when it feels like I’m not getting very much done, I’d probably tell you that I’m not very good at keeping myself a routine and that I’m not terribly focused most of the time.

I’d probably be lying.

While I wish that I had additional routines and habits more firmly established (blogging, writing, excercise, the truth is (I think, in retrospect) that a lot of how I exist in the world has to do with routines, and winning the mind-over-matter game.

Since graduating from college, or there abouts I’ve been very keen to limit/control the amount of sleep I get in an effort to control normal fluctuations in mood. I’m pretty keen on waking up “early” so that I can spend a few hours in the morning before the day starts writing and tending to my own things in “me time.” I’m also pretty mindful of how I use caffeine to control awakeness and focus, and I primarily do exercise/gym things to modulate my mood and energy level, to control how much I sleep and am able to focus.

Welcome to my life.

I realized much to my delight, that I’d reached that stage of sleep deprivation where I’m able to fall asleep directly after drinking a cup of tea and stay asleep, until the morning. I’ve also managed to get in a habit, where, most nights, I roll over and check the clock between sleep cycles. It doesn’t make the sleep feel less restful, but I do feel more grounded in reality when I wake up, and tend to be less groggy as a result.

Frankly this is kind of exciting for me.

Yesterday, I made tea right before I left the office, but left the mug there, and my night was noticeably effected by the chemical difference (I usually drink up to 24 ounces of tea on the train home.) All is better today.

Almost Poetry

Tue, 21 Dec 2010 00:00:00 +0000

in no particular order:

“Aww, you look like you’re annoyed at the water for being wet. Good morning.”

“[The broken heater] had only the vaguest reminiscence of warmth.”

“Wasn’t your major Women’s Studies?” “Yes.” “Then how did you learn about the Internet?”

“You look like the kind of fellow who wants a jolly rancher.”

New Technology

Tue, 21 Dec 2010 00:00:00 +0000

I was originally going to write this post as a “reasons I don’t need a new computer,” piece explaining my current setup (one laptop, a virtual server, and a lot of bailing wire) and explaining that despite some problems (a lack of local redundancy and small screen size) a new computer wasn’t exactly warranted. Though I wanted one, particularly after seeing the new MacBook Air, and I’ve long thought about getting a 15 inch laptop as I still lament my last 15 inch machine. Since I didn’t really need a new machine and there wasn’t a convincing reason to do an upgrade, I was going to write about good reasons to avoid upgrading just ‘cause.

Clearly I failed.

Particularly, since I’m writing this post from a new laptop.

A few weeks ago I saw a very good deal on a current-model 15" Lenovo ThinkPad (T510) with all of the specifications that I wanted: the larger resolution screen, integrated Intel graphics and wireless, a bunch of RAM (4g) and a 7200rpm drive. It even has a Core i7 processor (quad proc), which was a pleasant bonus, and so I went for it.

I’m quite happy with it. Besides a great deal and in many ways an ideal machine, I decided that being dependent on one (and only one!) system for all work and non-work computing was probably a bad idea. Additionally, I’ve wanted to reorganize the way my laptops’ hard drive partitions in a way that requires at least a short period of down time, and a process that I didn’t want to attempt without some sort of back up.

It took me a few days to get everything sorted out on the new machine, as it usually does, and there are some cool new things that I can do that I have yet to get ironed out, mostly around figuring out some virtualization technology to do awesome things with this system. But for the day to day stuff, it’s perfect and works just as I like.

This is the first time in several years where I’ve regularly used two systems for day-to-day work, and it’s the kind of thing that I’ve tended to avoid as much as possible. It’s just a hassle to switch between systems in terms of getting everything synchronized. I’ve got a pretty clever setup sketched out that I hope to be able to share with you all shortly.

In the end, this might not have been an absolutely essential purchase, but I think it was wise (in terms of the redundancy,) it makes some interesting things possible (virtualization, more processor intensive tasks,) and for the kinds of things I do, the extra screen space is very appreciated.

I’m sure I’ll write here from time to time about these things, but for the moment: Onward and Upward!

Unfettered Monday

Mon, 20 Dec 2010 00:00:00 +0000

Fantisque Unfettered is a new fantasy magazine from the editorial team that did the Aether Age shared world project. You can learn more about this by listening to the latest version of the outer alliance podcast. I must confess that reading/listening to all of these things is still in my queue, but I have a lot of respect for the creators of all of these things, and I’d love for this wiki play host to a discussion of any of these works.

I pulled together an archive or collection of my Critical Futures posts that discuss systems administration, administrators, and what “normal” folks can learn about technology and techno/social phenomena from the practice of system administration. I’m calling it “Lessons from Systems Administration.” If anyone has an idea for a more archives for Critical Futures posts, see the archives.

I hope your week is awesome!

Delicious Aftertaste

Sat, 18 Dec 2010 00:00:00 +0000

Apparently Delicious is shutting down. This is either not news to you, because you’ve been hearing people natter on about it for days, or you don’t have a clue what Delicious is. Which is more likely, because it’s probably being shut down for under-use, which means most people don’t have a clue. Right. It’s a website that allows you to post links “bookmarks” and then tag them, and then search for new links based on tag, with everyone’s links so you can discover whats happening based on what everyone is posting and tagging with what. Nifty idea. It’s also worth pointing out that Delicious was one of the first big web 2.0 startups, and got to the whole idea of the social web pretty early.

Anyway, enough history, and on to the news of the day.

tofu said “with all of this talk of delicious, I am surprised that I have heard little about creating a decentralized social bookmark alternative” which is a good thought and echo’s a slightly more complex proposal that marn made on the topic the other day.

In short, leaders of the contemporary revolution in federated network technology (“web services should be federated and exportable so that everyone’s data doesn’t get hosed when one website closes down, and people can share data with friends who uses different services”) are saying “let’s not make this mistake again, if we’re going to build an alternative, let’s make it better.”

And I think, in typical me-fashion, I’m saying: is social bookmarking still a thing? Have twitter and identi.ca replaced our need for this kind of service? The problem I think lays in the fact that there’s so much Internet that services like delicious very quickly become “lets catalog everything” and as a result “bookmarking” as a concept isn’t very useful for sharing links and exposing information to new audiences.

Having said that, I think the notion of writing a tool to use identi.ca or a status.net installation to replace delicious functionality seems like a great idea. Similarly, while it’s not federated pinboard.in has been around for a while and while it’s not federated, (or open source, particularly,) I like just about everything they’ve done.

Ponder that!

Still Knitting

Fri, 17 Dec 2010 00:00:00 +0000

This used to be a knitting blog. No really, it did. Then I got a technology job, move east, and started singing and dancing constantly, and didn’t really have a lot of spare time to knit. Compound this with the fact that I came of age as a knitter in Southern Wisconsin, and the heavy sweaters and thick socks that were essential there were pretty much unwearable in everywhere I’ve lived since.

Which isn’t to say that I’ve stopped knitting, hardly, but I have slowed, and I’ve tended to choose particularly boring projects for their meditative quality rather than for their knitterly interest. Not that meditative knitting is a bad thing, but it means I’m less likely to be enthralled in a project in a serious way, which means they take longer. Such projects are probably also uninteresting for you all to read about. But I have been knitting and thinking about knitting things.

I gave away a bunch of yarn that I acquired during college and in my dark period after college. Yarn that I got because it was a good deal, yarn that I was given, yarn that I didn’t have a plan to use, and yarn I didn’t particularly want to knit. It felt really good, to pare down the stash to yarn that I really liked and yarn that I really wanted to knit, and I’ve never been a serious knitter when I’ve had a legitimate disposable income, so I’ve never really been in a position to say “I want to make a sweater, let me go buy the perfect yarn for it.” Now I am, and it means I need to keep a lot less “rainy day yarn.” Good feelings.

As for what I’m working on now? I have a cowl/scarf device mostly done in a ribbing that’s my “current” project. I also have a sweater that just needs sleeves, and the collar hem sown down, and I’m stalled on a Alice Starmore-inspired Aran sweater (needs sleeves and some hem’s) because I’m probably going to run out of yarn, and I don’t really know how to precede properly (it’s also going to be unwearably warm.) And some socks. I seem to almost always have a pair of socks in progress, but they’re never interesting, and I don’t wear many wool socks most of the time.

I’m not sure when I’m going to have more time, but I am hoping to clear some of the decks soon and finish some of the above lingering projects, and perhaps move on to more exciting knitting. With luck!

The Magic of Vacation

Thu, 16 Dec 2010 00:00:00 +0000

I’ve been on vacation this week, if that wasn’t clear to you all. There’s been a lot of “doing things,” notably this website, a nifty lunch at a big tech company you’ve heard of, family stuff, bumming around New York City with R., and writing seemingly thousands of emails regarding projects, events, and possible meetings that I will probably not have time to commit to or maintain once I return to my regular routine. But it feels good to be able to pay attention to all sorts of work that I’m forced to neglect most of the time.

I really like this new wiki, and I really like the new publishig work flow, the way that people have interacted with the site and the ability to blog casually again. Here are some exciting links/thoughts:

Willie Taylor, an English/Irish folk song, sung in this version by a couple of friends at Bard College (Jeremy Carter-Gordon and Ben Bath). It’s nice to know that I’m not the only person in my generation who’s into this kind of music
pinboard.in, a delicious (rip) like, link-sharing/storing service. The thing I like most about Pinboard (I have an account, but I don’t really use it.) is that they charge a nominal fee for creating an account. This supports the site and controls who gets an account. While I’m not sure I like a world where content/services on the web are all pay-to-play, I really don’t like the world where no one is willing to pay for anything digital, because there’s an expectation that things ought to be free.
I wrote about this more than a year ago, but I think that halvm (an implementation of Haskell that runs on bare Xen) is really cool, and probably an indicator of things to come, but it’ll be a while before this (and other tools like it) are really ready for general use. Which is probably a good thing because engineers and IT people need some time to adapt to thinking about software in this way.

Keep warm!

A Modest Blogging Proposal

Tue, 14 Dec 2010 00:00:00 +0000

I’m going to present this post somewhat out of order. Here’s the proposal.

I want to think about moving this blog (or starting another?) to be a blog/wiki hybrid, and at the very least, moving forward I’d like the “discussion” or comment’s link to link to a wiki page rather than a comments thread.

I’ve been thinking about the blog recently. I really enjoy writing posts, and there are days when I rely on writing a blog post to get me thinking and moving in the morning (or afternoon!) and kinds of projects that I don’t think I would be able to work on if it weren’t for having the space to write and the opportunity to have conversations with you all.

I’ve done some work recently to streamline and simplify the publishing process, which does a lot to make me more likely to post on the fly, but I’m not sure that the tone of this site, or the current design, or my own habits would really support a different kind of publishing schedule.

As an aside, I think the technological shift that made blogs possible were “content management systems” and website building tools that made updating a website with new content incredibly simple. While blogging has come to mean many other things and is defined by a number of different features, having the ability to publish on very short notice has a large effect on the way people write blog content.

So here’s the thing about blog comments: I don’t think that they’re used particularly well, and there are some important flaws in so many of the options around. First, the best systems, like the one used on LiveJournal, IntenseDebate, and Disqus (which I use on this site) are all proprietary systems that are depending on an external service to function. The worse systems all have independent authentication methods, often lack proper threading (which most comm enters aren’t terribly good at using anyway,) and it’s very difficult to prevent all these systems from being filled with spam.

What’s more, people don’t really comment that much. At least for most blogs.

better comment systems, better discussions.
catering to people who want to write a lot. in comments.
allowing the conversation to grow from comments, in productive rather than purely discursive ways.

And once you’ve moved comments into a wiki, why not move the rest of the blog as well? My preferred engine, ikiwiki, has support for blog-like content so while there would be some work involved, it wouldn’t be a major hassle to manage. And the worst case scenario is that the old content remains in the old system, which might not be a bad thing in the end.

Anyone out there in reader land have any thoughts on the subject? While I’ll probably make some sort of revision to the way I blog/maintain tychoish.com, any such change is probably a month or two in the future.

Against Open Stacks

Mon, 13 Dec 2010 00:00:00 +0000

I have misgivings about Open Stack. Open Stack is an open source “Cloud” or infrastructure/virtualization platform, that allows providers to create on-demand computing instances, as if “in the cloud,” but running on their own systems. This kind of thing is generally refereed to as “private clouds,” but as all things in the “cloud space,” this is relatively nebulous concept.

To disclose, I am employed by a company that does work in this space, that isn’t the company that is responsible for open space. I hope this provides a special perspective, but I am aware that my judgment is very likely clouded. As it were.

Let us start from the beginning, and talk generally about what’s on the table here. Recently the technology that allows us to virtualize multiple instance on a single piece of hardware has gotten a lot more robust, easy to use, and performant. At the same time, for the most part the (open source) “industrial-grade” virtualization technology isn’t particularly easy to use or configure. It can be done, of course, but it’s non trivial. These configurations and the automation to glue it all together--and the quality therein--is how the cloud is able to differentiate itself.

On some level “the Cloud” as a phenomena is about the complete conversion of hardware into a commodity. Not only is hardware cheap, but it’s so cheap that we can do most hardware in software, The open sourcing of this “OpenStack” pushes this barrier one step further and says, that the software is a commodity as well.

It was bound to happen at some point, it’s just a curious move and probably one that’s indicative of something else in the works.

The OpenStack phenomena is intensely interesting for a couple of reasons. First, it has a lot of aspects of some contemporary commercial uses of open source: the project has one contributor and initial development grows out of the work of one company that developed the software for internal use and then said “hrm, I guess we can open source it.” Second, if I’m to understand correctly, OpenStack isn’t software that isn’t already open source software (aside from a bunch of glue and scripts), which is abnormal.

I’m not sure where this leads us, and I’ve been milling over what this all means for a while, and have largely ended up here: it’s an interesting move, if incredibly weird and hard to really understand what’s going on.

Sys Admin Legacies in Free Software

Thu, 09 Dec 2010 00:00:00 +0000

I was writing a blog post about the ideological tendencies in free software and I found myself on a side note that I think really wants to be it’s own post. Funny how that happens. I was thinking about the role and importance of system administrators in free software communities and development. This post is part of an ongoing thread on dialectical futurism about systems administration and its implications.

One might think, because free software communities produce software, that free software communities are communities of software developers. Programmers make software, free software is software, ergo… But I really think that a significant differentiating factor between free and non-free software is that free software tends to be created, shaped, and designed by people who are systems administrators by trade and inclination rather than people who are primarily software developers.

This is a somewhat difficult argument, and one that requires us to presume that the boundary between developers and administrators is impermeable (it isn’t,) but I think people who are programmers first and administrators second approach technological problems. Systems administrators write code. Lots of code. Any system that must be administered (deployed, modified, and maintained) “by hand” is probably a broken system. Scripting and automation are the keys to making systems maintainable in the long run. And the boundary between writing a few (dozen (dozen)) scripts and writing an operating system is probably not that great in the grand scheme of things.

And knowing how operating systems work is probably a key to making them work. In this case, we can imagine that systems administrators like open source operating systems (i.e. Linux and GNU based ones) because their inner workings are knowable, so system administrators are likely to reap much larger benefits from free software than other classes of users.

I’ve often found that understanding the richness and complexity of the “value” of Free Software and Open Source software is important for understanding why free software continues to exist and may be worth adopting. The value of free software comes from: the fact that there are no licensing costs, the freedom you have to modify the software to your needs and potentially largely ameliorate development costs, finally free software is valuable because it creates smarter users by virtue of its origins in academia and the way that it promotes user independence and community involvement.

Business decision makers might like the fact that the initial cost of free software is minimal and controlled, but most software related costs are probably support related and free software may not do much to minimize those costs. Developers may like that free software could make their jobs easier, but they may also suffer from “not invented here” syndrome, and be resistant to working on projects that (potentially) suffer from design decisions that they disagree with. Systems administrators may have preferences regarding certain pieces of software, but will generally like the additional control that free software offers them.

I wonder about the inverse: has the involvement of systems administrators in free software had an affect on the shape of that software? Do we use Linux and BSD-Unix because they’re easier to administer? Does this extend to the network protocols and technologies that get used more frequently?

Let’s file this under “questions I wish I knew how to answer…”

Onward and Upward!

Welcome to Tychoish Rhizomatics

Wed, 01 Dec 2010 00:00:00 +0000

Welcome to this new little blog/wiki project. I made a post to the ‘blog about reorganizing the way I did my blog comments and blogging called a modest blogging proposal.

I wrote this post a few weeks ago. I was so inspired by the prospect of reorganizing the blog, and also so frustrated by the less-than-useful structure of my blogging that, I did the “big reorganization” of the blog the very same day that I published the post that started it all.

This proves, it seems, the neccessity of these changes, which are:

Moving the “dialectical futurism” blog to my “Critical Futures” domain, as an “essay” blog. I’m planning to post a new essay here every Tuesday. I might also add a second day a week for some other feature (podcast? contributed essay?)
Move the “wikish” wiki that I’d been hosting on the “tychoish.com” domain to be tychoish.com, and add a more short-form blog to the “blog/wiki.” I’m calling it “Rhizomatics,” and this is the first post. I’ll be writing here pretty regularly, and with less intense posts.
Do all of the above in a way that doesn’t break all of the existing links.

And I think I’ve done it all. I’ll be doing some tweaking in the next few days and weeks to make it all work a bit better, but suggestions are always welcome.

Stay tuned!

Ideology and Systems Administration

Tue, 30 Nov 2010 00:00:00 +0000

I do some work as a systems administrator, both personally and for friends. And I work with a lot of admins, but I don’t really think of myself as a sys admin. Though you may feel free to argue the point. Nevertheless, I spend a lot of time trying to figure out the way systems administrators think and work. This makes sense: as my professional work is written for entry level systems administrators and I work with a bunch of admins. But I think it’s probably bigger than that. This post is part of an ongoing thread on dialectical futurism about systems administration and its implications.

The best systems administrators are unnoticed and unremarkable. When a system is working smoothly, it works and no one has reason to think about who is maintaining the system. Thus, to be a better systems administrator you have to become confident in your abilities (leading to a somewhat grounded stereotype in arrogance) and you have to be resistant to change.

For example, take this slide deck of a systems administration problem. It presents a thorny sysadmin problem where the chmod utility (which is used to render files executable) has been marked unexecutable. The presentation goes through a number of different methods of fixing this, however (spoiler alert) the final solution is “the easy fix is to reboot the machine and fix it then (or something), and the machine’s running so there isn’t a problem.” While this is a funny example, I think it’s also largely a true example of the way systems administrators approach and resolve problems.

I’ve seen this kind of “well it' may not be perfect, but it works,” logic as well as the “is it worth building something new and different that might be better?” reasoning at work, and I think it’s probably apparent in all sorts of free software and other discussion forums where sys admins discuss things.

Thus, I wonder: Does this ideology extend beyond the administration of systems and into other spheres of life and thinking? About technology? About politics and economics? I’m not sure, though I’m of course inclined to say yes, and I think it’s something that requires some deliberation, and further thinking.

I look forward to hearing your thoughts, and figuring out the best way to answer this question.

Onward and Upward!

Bitlbee, The Wrong Solution that Works

Tue, 23 Nov 2010 00:00:00 +0000

About a week ago, time of writing, I switched all of my instant messaging to a little program called Bitlbee. Basically this is a program that runs locally as an IRC server and connects to various instant messaging and “presence” protocols and exposes them to the end user client as if they were IRC. Weird.

This is, emphatically, the wrong solution to the problem of finding a sane technological solution to consuming real-time information (e.g. instant messaging, twitter, xmpp, etc.) Previously, I’d been using an XMPP-only client and running jabber-to-IM transports on the server, which I think is more of a right solution. Why then did I switch?

I wanted to use irssi, which I think one of the most cleverly designed and useful pieces of software out there.
Transports that allow XMPP to interact with other services are an ideal solution and I think the inclusion of transports in the design of the XMPP protocol is a major selling point for the XMPP technology. At the same time the most stable transports aren’t terribly stable and while there could be transport widgets for all sorts of things there are only a few general purpose transports.

Practically speaking the jabber-to-AIM transport that I had been using, had a habit of dying without cause once or twice a week, and it used a lot of system resources for something that could (should?) have been much simpler.
The truth is that while XMPP is a nifty technology, and I really enjoy using it, I’m starting to think that it’s not ideal to expect that XMPP replace IRC, as both accomplish different things for their users. So while I always saw bitlbee as “giving into IRC” it’s really just an interface. And frankly IRC clients do IM better than IM clients do IRC.
Bitlbee works really well as a client for Facebook chat (which is a weird XMPP flavor) and is a functional twitter client. With the delight of using irssi, I’m able to really interact on these networks without having to spend too much brain power sifting through crud.

So here I am. Switched. The buddy list on bitlbee leaves something to be desired (but I have a particularly large buddy list) and I’ve yet to get used to the syntax for creating and administering group chats inside of bitlbee, but other than that? It’s pretty rocking.

Onward and Upward!

Writing and Growing Professionally

Tue, 16 Nov 2010 00:00:00 +0000

I spent a lot of formative time in high school and college listening to writing teachers and would be mentors tell me that I was too sloppy or too disorganized to write effectively. They were probably right. Furthermore, this is probably not something that I think I’ve been able to keep secret from anyone who has read my blog for any measurable period of time. (Though I do think most of my more recent entries are better than nearly all of my early entries.) What no one really dared to tell me, are probably the most important things I’ve learned as a writer:

First, that editors are not only essential to the writing process, but that there’s something fundamentally wrong if something leaves the original author and is handed to final readers without passing through at least one editor, and often more.
Second, the skill of writing isn’t necessarily being able to write artful sentences, or being able to perfectly apply all of the rules of grammar (which, aren’t detrimental to the craft of writing). No, writing is about being able to get things written. Writing is pobably about being able to do research while keeping in mind the parameters of the project and ending up with a few paragraphs on a given topic that make sense and enlighten more than they confuse. That is considerably more rare.

The problem, and I wish I had a solution for this, is that there is no real way to teach people to write and to love writing. Exposing people to lots of examples of writing (i.e. literature) is helpful in teaching people to read and cherish the practice of reading. Unfortunately, I think reading and writing pull on vastly different skills. And while readers have a useful and required prospective on the text, readers who don’t write often provide ambiguous and difficult to assimilate feedback.¹

Fundamentally, I think, readers live on the plane of words, and writers--at least writer’s like me--live on the plane of paragraphs. And then there’s the whole issue of confusing a love of reading with a need or desire to write, but that’s another story for another time.

The way, I think, to learn how to write better is to write a lot of crappy stuff and learn how to “fail” better and more gracefully. Blogging has and is a great tool for me in this regard, but it’s not a cure-all, and I think integrating blogging in the writing curriculum is a difficult project that requires a very nuanced view of blogging, and the right set of learning objective. Beyond this, the project of learning to write and learning to write “better” is one that I’m not sure how to properly facilitate in myself or in others.

At the end of the day, I think it’s important to realize that growth as a writer isn’t the kind of thing that happens quickly. Being a writer is a life-long project with slow and steady improvement, minor regressions, stunning breakthroughs, dashed hopes, and tactical successes.

Onward and Upward!

Readerly feedback often comes in the form of thinking that large swaths of text need to be rewritten, when the addition of a single sentence clarifies the required point. Similarly, I think readers aren’t as prone to thinking about texts and paragraphs as things that can be reordered above the level of the word. ↩︎

Why Email Still Matters

Thu, 11 Nov 2010 00:00:00 +0000

There are so many sexy topics in computing and information technology these days. In light of all this potential excitement, I’m going to write about email. Which isn’t sexy or exciting.

This isn’t we should be clear, to say that email doesn’t matter, because it seems that email still matters a great deal. Rather that email is still a relevant and useful paradigm. What’s more, the email system (i.e. SMTP and associated tools) remains in many ways superior to all of the technologies and platforms that have attempted to replace email.

The Good

Email works. The servers (e.g. Postfix, Exim, Sendmai, but most Postfix) are stable, known, and very functional. While there are flaws in a lot of email clients, there are a lot of tools that exist for processing and dealing with email, and that makes it possible for everyone to interact with their email on their own terms, in a variety of contexts that make sense the them. And email is such that we can all use it and interact with each other without requiring that we all participate in some restrictive platform or interface.

In short, email is open, decentralized, standard, lightweight, push-based, and multi-modal.

Compare this to the systems that threaten to replace email: Facebook and social networking utilities, twitter, text messaging, real-time chat (i.e. IRC, IM, and Jabber). The advantages of email on these crucial, I think, dimensions are pretty clear.

The Bad

The problem, of course, with email is that it’s terribly difficult to manage to keep current with one’s email. Part of this problem is spam, part of the problem is “bacon,” or legitimate (usu sally automated) email that doesn’t require attention or is difficult to process, and it’s undeniable that a big part of of it is that most end user email clients are inefficient to use. And there’s the user error factor: most people aren’t very good at using email effectively.

It Gets Better

No really it does. But I don’t think we can wait for a new technology to swoop in and replace email. That’s not going to happen. While I’m not going to write a book on the subject, I think there are some simple things that most people can do to make email better:

1. Do use search tools to make the organization of email matter less. Why file things carefully, when you can quickly search all of your email to find exactly what you need.

2. Filter your email, within an inch of it’s life. Drop everything you can bare to. Put email lists into their own mail boxes. Dump “work” or “client” email into its own folders. Successful filtering means that almost nothing gets to your “inbox.”

3. Use your inbox as a hotlist of things that need attention. Move email that needs responses to your inbox, and move anything that got through your filters to where it ought to be.

4. Use multiple email addresses that all redirect to a single email box. You only want to ever have to check one email system, but you probably want multiple people in multiple contexts to be able to reach you via email. This makes email filtering easier, and means that you just spend time working rather than time switching between email systems and wondering where messages are.

5. When writing emails, be brief and do your damnedest to give the people you’re writing with something concrete to respond to. Emails that expect responses but are hard to respond to are among the worst there are, because you have to say something there’s nothing worth saying.

6. Avoid top posting (i.e. responding to an email with the quoted material from previous exchanges below your respone.) When appropriate interleave your responses in their message to increase clarity and context without needing to be overly verbose.

7. Email isn’t real time. If you need real time communication use some other medium. Don’t feel like you need to respond to everything immediately. Managing expectations around email is a key to success.

That addresses most of the human problem. The technological problem will be solved by addressing spam, by building simpler tools that are easier to use effectively and support the best kind of email behaviors.

Why Email will Improve

1. Email is great in the mobile context. It’s not dependent upon having a net connection which is good when you depend on wireless.

2. Email is a given. Having email is part of being a digital citizen and we mostly assume that everyone has an email. The largest burden with most new technologies is often sufficient market share to make a “critical mass” rather than some sort of threshold of innovation.

3. Email is both push-based (and delivery times are pretty fast) and asynchronous. Though this doesn’t sound sexy, there aren’t very many other contemporary technologies that share these properties.

Onward an Upward!

Persistent SSH Tunels with AutoSSH

Tue, 09 Nov 2010 00:00:00 +0000

Rather than authenticate to a SMTP server to send email, which is fraught with potential security issues and hassles, I use a SSH tunnel to the machine running my mail server. This is automatic, easy to configure both for the mail server and mail client, and incredibly secure. It’s good stuff.

The downside, if there is one, is that the tunnel has to be active to be able to send email messages, and SSH tunnels sometimes disconnect a bit too silently particularly on unstable (wireless) connections. I (and others, I suspect) have had some success with integrating the tunnel connection with pre- and post- connection hooks, so that the network manager automatically creates a tunnel after connecting to the network. but this is a flawed solution that produces uneven results.

Recently I’ve discovered this program called “AutoSSH,” which creates an SSH tunnel and tests it regularly to ensure that the tunnel is functional. If it isn’t, AutoSSH recreates the tunnel. Great!

First start off by getting a copy of the program. It’s not part of the OpenSSh package, so you’ll need to download it separately. It’s in every pacakge management repository that I’ve tried to get it from. So installation, will probably involve one of the following commands at your system’s command line:

apt-get install autossh
pacman -S autossh
yum install autossh
port install autossh

When that’s done, you’ll issue a command that resembles the following

autossh -M 25 -f tychoish@foucault.cyborginstitute.net -L 25:127.0.0.1:25

Really, the important part here is the “autossh -M 25” part of the command. This tells autossh to watch (“monitor”) port number 25 on the local system for a tunnel. The rest of the command (e.g. “-f -L 127.0.0.1:25:127.0.0.1:25 mailserver@tychoish.com -N") is just a typical call to the ssh program.

Things to remember:

If you need to create a tunnel on a local port with numbered lower than 1000, you’ll need to run the autossh command as root.
SSH port forwarding only forwards traffic from a local port to a remote port, through an SSH connection. All traffic is transmitted over the wire on port 22. Unless you establish multiple tunnels, only traffic sent to the specific local port will be forwarded.
Perhaps it’s obvious, but there has to be some service listening on the specified remote end of the tunnel, or else the tunnel won’t do anything.
In a lot of ways, depending on your use case autossh, can obviate the need for much more complex VPN setups for a lot of deployments. Put an autossh command in an @reboot cronjob, with an account that has ssh keys generated, and just forget about it for encrypting things like database traffic and the like.

Onward and Upward!

On Romance

Tue, 02 Nov 2010 00:00:00 +0000

I don’t read romance literature.

It’s not my thing, which isn’t saying much: there’s a lot of literature that I don’t tend to consider “my thing,” for one reason or another. I don’t really read fantasy, or horror, and I’m even picky within science fiction. There are enough books out there and there is only so much time. At least that’s what I tell myself.

Nevertheless, Susan Groppi wrote a great post about coming out as a reader of romance that I found useful. I’m also reminded of comments that N. K. Jemison made about the in progress merging of the fantasy and romance genres (sorry if I’ve miss-cited this), and I’ve been thinking about how I view Romance fiction, and perhaps a bit more generally about genre fiction ghettos.

In general, I think Romance has merit, both because it’s entrancing and I think fiction which captures people’s imaginations and interest I worthwhile and important to not dismiss because it’s commercial, or the readership/writers are largely women. There are potential problems with romance, at least insofar as we typically envision it: with strong hetero tendencies, an idealization of monogamy as a social practice and marriage as an institution, and the potential to accept a very conventional conceptualization of gender. I’m sure some romance literature has been able to engage and trouble these troupes productively, but I think it’s a potential concern.

Having said that, I’m not sure that Romance has a lot of future as a genre. This is to say that I think many of the elements of romance--female characters, and an engagement with sexuality and relationships--will increasingly merge into other genres. Romance as an independent genre will linger on, but I think the “cool stuff happening in the Romance field,” will probably eventually move out into corners of other genres: thriller, fantasy, maybe science fiction.

Actually, as I think about this, it’s probably backwards. I think it’s less that Romance doesn’t have a future, as it is that the future of most popular literature lies in engaging with romance-elements and other aspects of romance stories the context of non-romance specific styles. This kind of thing is happening, and I think it’ll probably continue to happen.

I wish I could speak with greater certainty about the reasons why romance literature enjoy higher readership, or what elements of romance stories can be transplanted to other genres, but I think these are probably questions which are beyond the scope of this post. Thanks for reading!

Knitting Resumed

Wed, 27 Oct 2010 00:00:00 +0000

So I’ve started knitting again.

Shocking.

I’ve not been knitting very much in recent months, because I’ve had less time, because I’ve been focusing my energy on other projects: keeping my head above water, writing, dancing, singing, etc. It’s a shame too, because knitting is a great deal of fun, it’s pretty rewarding, and it’s something that I’m incredibly good at.

I suppose at one point there were a lot of knitters who read this blog, but I suspect many of them don’t so much any more. Anyway, I hope this post won’t alienate everyone who reads this.

First up, for some project review:

I have a cabled sweater in progress that I’m working on the first sleeve at the moment. I’m afraid I’m going to run out of yarn (but I have a plan!) and frankly cables have never really been “my thing,” but this is the only thing that remains that I was working on when I left the Midwest, so there’s nostalgia and I do want to finish it. The biggest problem, I think, is that unless I move north a lot, I’m not going to live in a place where I can really wear something like this.
I have a sweater in jumper-weight Shetland (i.e. fingering weight) wool that I’m knitting for a friend. I have the collar and the sleeves to do, but it’s very plain and a very straightforward knit. I just need to do it. This is probably the next thing on my list. It’s been on hiatus since May.
I’m making socks. As I write this, I have just completed a pair of socks that I started in May. They’re simple, and plain (which is how my socks tend to be) using Dyebolical Yarn. I also got to use a set of Blackthron Needles that my mother got for my birthday. Both are quite wonderful. I think I’ve discovered how much knitting can be done on a commute, and I do expect to do a lot more commute knitting, but I need to find a way to balance knitting with writing and reading on the train. Perhaps some sort of morning/night split.

Aside from finishing the socks and immediately casting another pair, I’ve been doing a lot of “yarn stash” reorganization and trimming. This last week or so I’ve gotten inspired to reevaluate all of the stuff that I have to see what I really need in my life and what I’m just keeping because it’s there. I’ve been through my clothing, the book collection, and the yarn.

Although I’ve done “stash culls” before I felt like my collection of yarn had a lot of stuff in it that I got without any intention of a project, or for any reason other than “I might like to make something with it some day.” I’ve never really been a knitter when I’ve had a real budget for hobbies and entertainment, nor have I ever knitted at such a moderate pace. So I made the decision to not keep yarn around just for insulation, and just get the yarn that I really want to knit with rather than what I feel like I ought to knit with because it’s in the bin. It’s been quite liberating.

I’m also, as I sit knitting, thinking about the overlap between what I do professionally (documenting technical solutions and systems administration practices) and pattern writing for knitters.

There’s a lot of overlap in how I write and think about both, enough to inspire me to think about doing more knitting related writing. *As if I didn’t have enough projects already.

In any case, I don’t know that I’ll blog regularly about knitting as I continue to knit more, but it might come up from time to time. You have been warned.

Onward and Upward!

Phone Torched

Thu, 21 Oct 2010 00:00:00 +0000

I mentioned in a recent update post, that I had recently gotten a new cell phone, which given who I am and how I interact with technology means that I’ve been thinking about things like the shifting role of cell phones in the world, the way we actually use mobile technology, the ways that the technology has failed to live up to our expectations, and of course some thoughts on the current state of the “smart-phone” market. Of course.

I think even two years ago quasi-general purpose mobile computers (e.g. smart phones) were not nearly as ubiquitous as they are today. The rising tide of the iPhone has, I think without a doubt, raised the boat of general smart phone adoption. Which is to say that the technology reached a point where these kinds of devices--computers--are of enough use to most people that widespread adoption makes sense. We’ve reached a tipping point, and the iPhone was there at the right moment and has become the primary exemplar of this moment.

That’s probably neither here nor there.

With more and more people connected in an independent and mobile way to cyberspace, via either simple phones, (which more clearly matches Gibson’s original intentions for the term,) or via smart phones I think we might begin to think about the cultural impact of having so many people so connected. Cellphone numbers become not just convenient, but in many ways complete markers of identity and person-hood. Texting in most situations overtakes phone calls as the may way people interact with each other in cyberspace, so even where phone calls may be irrelevant SMS has become the unified instant messaging platform.

As you start to add things like data to the equation, I think the potential impact is huge. I spent a couple weeks with my primary personal Internet connection active through my phone, and while it wasn’t ideal, the truth is that it didn’t fail too much. SSH on Blackberries isn’t ideal, particularly if you need a lot from your console sessions, but it’s passable. That jump from “I really can’t cut this on my phone,” to “almost passable” is probably the hugest jump of all. The series of successive jumps over the next few years will be easier.

Lest you think I’m all sunshine and optimism, I think there are some definite short comings with contemporary cell phone technology. In brief:

There are things I’d like to be able to do with my phone that I really can’t do effectively, notably seamlessly sync files and notes between my phone and my desktop computer/server. There aren’t even really passable note taking applications.
There are a class of really fundamental computer functionality that could theoretically work on the phone, but don’t because the software doesn’t exist or is of particularly poor quality. I’m thinking of SSH, of note taking, but also of things like non-Gmail Jabber/XMPP functionality.
Some functionality which really ought to be more mature than it is (e.g. music playing) is still really awkward on phones, and better suited to dedicated devices (e.g. iPods) or to regular computers.

The central feature in all of these complaints is software related, and more an issue of software design, and an ability to really design for this kind of form factor. There are some limitations: undesirable input methods, small displays, limited bandwidth, unreliable connectivity, and so forth. And while some may improve (e.g. connectivity, display size) it is also true that we need to get better at designing applications and useful functionality in this context.

My answer to the problem of designing applications for the mobile context will seem familiar if you know me.

I’d argue that we need applications that are less dependent upon a connection and have a great ability to cache content locally. I think the Kindle is a great example of this kind of design. The Kindle is very much dependent upon having a data connection, but if the device falls offline for a few moments, in most cases no functionality is lost. Sure you can do really awesome things if you assume that everyone has a really fat pipe going to their phone, but that’s not realistic, and the less you depend on a connection the better the user experience is.

Secondly, give users as much control over the display, rendering and interaction model that their software/data uses. This, if implemented very consistently (difficult, admittedly,) means that users can have global control over their experience, and users won’t be confused by different interaction models between applications.

Although the future is already here, I think it’s also fair to say that it’ll be really quite interesting to see what happens next. I’d like a chance to think a bit about the place of open source on mobile devices and also the interaction between the kind of software that we see on mobile devices and what’s happening in the so-called “cloud computing” world. In the mean time…

Outward and Upward!

Running Multiple Emacs Daemons on a Single System

Wed, 13 Oct 2010 00:00:00 +0000

Surely I’m not the only person who’s wanted to run multiple distinct instances of the emacs daemon at once. Here’s the use case:

I use one laptop, but I work on a number of very distinct projects many of which involve having a number of different buffers open, most of which don’t overlap with each other at all. This wouldn’t be a huge problem except that I’ve easily gotten up to two hundred buffers open at once. It can get a bit confusing. Particularly since I never really need to touch my work related stuff when I’m writing blags, and my blogging and website design buffers never intersect with fiction writing.

If I weren’t using emacs in daemon mode (that is, invoked with the “emacs --daemon” command) I’d just open separate instances of emacs. The problem with that is, when X11 crashes (as it is so wont to do) the emacs instances crash too and that’s no good. Under normal conditions if you start emacs as a daemon, you can only run one at a time, because it grabs a socket and the emacsclient program isn’t smart enough to be able to decide which instance of emacs you want. So it’s a big ball of failure.

Except I figured out a way to make this work.

In your .emacs file, at the very beginning, put the following line:

(setq server-use-tcp t)

In the default configuration, the emacs daemon listens on a UNIX/system socket. However, in emacs can also, with the above option set, can also listen for connections over TCP. I’ve not yet figured out how to create the required SSH tunnel to make this particularly cool, but it makes this use case possible.

Now, when you start emacs, use commands in the following form:

emacs --daemon=tychoish
emacs --daemon=work

Each server process creates a state file in the “~/.emacs.d/server/” folder. If you are using version control on this file, you may want to consider explicitly ignoring this folder to avoid confusion.

To open an emacs client (i.e. an emacs frame attached to the emacs daemon,) use commands in the following form

emacsclient --server-file=tychoish -c -n
emacsclient --server-file=work -c -n

You may append a file name to open a specific client with one of these emacsclient invocations, or use any of the other emacsclient options. Although these commands are long, I have integrated them into my default zsh configuration as aliases, and as key shortcuts in my window manager. So opening a frame on a specific emacs instance isn’t particularly difficult.

And that’s it. It just works. I have the following two lines in my user’s crontab file:

@reboot    emacs --daemon=tychoish
@reboot    emacs --daemon=work

These lines ensure that the usual (and expected) named emacs daemons are started following reboot. More generally, the @reboot cronjob is great for making the “my computer just rebooted, and now I have to fuss over it for ten minutes before I can work” problem seem much less daunting.

In conclusion I’d like to present one piece of unsolicited advice, and ask a question the assembled.

Advice: Even though it’s possible to create a large number of emacs instances, and on modern systems the required RAM is pretty low, avoid this temptation. The more emacs instances you have to juggle the greater the chance that you’ll forget what buffers are open in what instance. Undesirable.
Question: Is there a way to get the value of server-name in into emacs lisp so that I can if things against it? Haven’t figured this one out yet, but it seems like it would be nice for conditionally loading buffers and things like org-mode agenda. Any ideas?

Onward and Upward!

Sing the Shapes

Tue, 05 Oct 2010 00:00:00 +0000

I wrote a bit about sacred harp singing for a few months about a year ago, when I was really starting to get into it, and then I mostly stopped. I’ve had a few singing related experiences recently that I think are worth recounting, even if they’re a bit disjointed. So I’ll just hop in and hope that it adds up to something in the end. Also, if you’re not familiar with Sacred Harp Singing, I’m sorry if there isn’t a lot of sub tittling. Thanks for reading!

I was hanging out with R.F. and we were flipping through my copy of the sacred harp, and he was trying to get how the relative pitching thing works (having more formal experience singing with choirs and what not, and a sense of pitch that’s way more closely tied to a piano than mine.) and he said something like “so this one would start ‘here?'” I think it was 300, and I have no clue how “right,” I was or what inspired this, but his pitch was about a step and a half (I think,) high, and so I gave something that was more or less where I thought the song was supposed to sit. We sang through a little bit of it, and it seemed to work.

I’ve never really had a lot of interest in being able to offer pitches to a class of Sacred Harp singers, beyond the very selfish ability to lead signings without needing to make sure that someone who can offer keys in attendance.

I’m working on memorizing the book--strategically, of course-- as I can. This makes signings more fun because you can look at people, while singing rather than having your nose in a book the whole time. While there aren’t songs that I can safely leave the book closed for the shapes, I know the tunes (mostly bass parts) and words to most of the common ones (e.g. 178, 155, 89, 312b, 355, 300, 146, 148, 153, 112, 422, 209, 189, 186,) save a few middle verses that are sung rarely. I don’t think of my memory as being particularly good for this kind of information, but it’s nice to have reality prove you wrong.

One of the things that made Sacred Harp “click” for me when I really started to get into it was that I had the good sense to sing bass. My voice is pretty low, so this seems to fit, and I think staying in one section for a long time helped solidity my sense of the music.

Since March/April, or thereabouts, I’ve started singing tenor (the lead/melody) a bit. It’s a stretch for my voice, and I’m slightly more prone to loosing track of the key when singing higher notes (a not uncommon problem,) but it’s good for my brain, and I think it makes me a better singer and leader. I’ve mostly done this at local singings, and smaller signings when there are enough basses, or for a few songs at a bigger singing when the mood strikes.

I’m thinking of doing this more often, and at more singings, as part of an effort to become a better singer.

I think it’s easy (at least for me) particularly in accounts like this to focus on the singing, the technical aspects of the music, and the texts used. And all of these components contribute to what makes singing so great: its a gestalt experience, but I think its easy to gloss over the best part of being a singer. Which is, of course, all the other singers.

Being a “community guy,” I think it might be easy for me to wax poetic about how great sacred harp singings are--and they are--but I think there’s something deeper and specific about singing communities that make them more accepting, more engaged, more inclusive than other communities (dancing, writing, professional,) that I’ve been involved in

Maybe it’s that singing is a more transcendent experience that the focal points of other communities to begin with so people are willing to connect a bit more. Maybe the fact that singings are sometimes (often?) held in people’s homes is a factor. Maybe the extreme inclusiveness combined with the somewhat substantial learning curve creates the right environment to foster a strong and self selecting community. Perhaps all of the travel to all day singings and conventions, combined with the effort to arrange socials, unifies the community.

I’m not sure, but I’ve met a bunch of great people singing, and people with whom I share more than just sufficient common interest in a shared activity. I’m not sure every singing community is like this, but the conversations and connections I’ve had with other singers have been depthy, interesting, and have expanded beyond singing.

Key Git Concepts

Thu, 30 Sep 2010 00:00:00 +0000

Git is a very… different kind of software. It’s explicitly designed against the paradigm for other programs like it (version control/source management) and to make maters worse most of it’s innovations and eccentricities are very difficult to create metaphors and analogies around. This is likely because it takes a non-proscriptive approach to workflow (you can work with your collaborators in any way that makes sense for you) and more importantly it lets people do away with linearity. Git makes it possible, and perhaps even encourages, creators to give up an idea of a singular or linear authorship process.

That sounds great (or is at least passable) in conversation but it is practically unclear. But even when you sit down and can interact with a “real” git repository, it can still be fairly difficult to “grok.” And to make matter worse, there are a number of very key concepts that regular users of git acclimate to but that are still difficult to grasp from the ousted. This post, then, attempts to illuminate a couple of these concepts more practically in hopes of making future interactions with git less painful. Consider the following:

The Staging Area

The state of every committed object (i.e. file) as of the last commit is the HEAD. Every commit has a unique identifying hash that you can see when you run git log.

The working tree, or checkout, is the files you interact with inside of the local repository. You can checkout different branches, so that you’re not working in the “master” (default or trunk) branch of the repository, which is mostly an issue when collaborating with other people.

If you want to commit something to the repository, it must first be “staged” or added with the git add command. Use git status to see what files are staged and what files are not staged. The output of git diff generates the difference between the HEAD plus all staged changes, and all unstaged changes. To see the difference between all staged changes and HEAD use the “git diff --cached”.

The staging area makes it possible to construct commits in very granular sorts of ways. The staging area makes it possible to use commits, less like “snapshots” of the entire tree of a repository, and rather as discrete objects with that contain a single atomic change set. This relationship to commits is enhanced by the ability to do “squash merges” and squash a series of commits in a rebase, but it starts with the staging area.

If you’ve staged files incorrectly you can use the git reset command to reset this process. Used alone, reset is a non destructive operation.

Branches

The ability to work effectively in branches is the fundamental function of git, and probably also the easiest to be confused by. A branch in git, fundamentally, is just a different HEAD in the same repository. Branches within a single repository allow you to work on specific sets of changes (e.g. “topics”) and track other people’s changes, without needing to make modifications to the “master” or default branch of the repository.

The major confusion of branches springs from git’s ability to treat every branch of every potentially related repository as a branch of each other. Therefore it’s possible to push to and pull from multiple remote branches from a single remote repository and to push to and pull from multiple repositories. Ignore this for a moment (or several) and remember:

A branch just means your local repository has more than one “HEAD” against which you can create commits and “diff” your working checkout. When something happens in one of these branches that’s worth saving or preserving or sharing, you can either publish this branch or merge it into the “master” branch, and publishes these changes.

The goal of git is to construct a sequence of commits that represent the progress of a project. Branches are a tool that allow you to isolate changes within tree’s until you’re ready to merge them together. When the differences between HEAD and your working copy becomes to difficult to manage using git add and git reset, create a branch and go from there.

Rebase

Rebasing git repositories is scary, because the operation forces you to rewrite the history of a repository to “cherry pick” and reorder commits in a way leads to a useful progression and collection of atomic moments in a project’s history. As opposed to the tools that git replaces, “the git way” suggests that one ought to “commit often” because all commits are local operations, and this makes it possible to use the commit history to facilitate experimentation and very small change sets that the author of a body of code (or text!) can revert or amend over time.

Rebasing, allows you to take the past history objects, presumably created frequently during the process of working (i.e. to save a current state) and compress this history into a set of changes (patches) that reflect a more usable history once the focus of work has moved on. I’ve read and heard objects to git on the basis that it allows developers to “rewrite history,” and individuals shouldn’t be able to perform destructive operations on the history of a repository. The answer to this is twofold:

Git, and version control isn’t necessarily supposed to provide an consistently reliable history of a projects code. It’s supposed to manage the code, and provide useful tools to managing and using the history of a project. Because of the way the staging area works, sometimes commits are made out of order or a “logical history object” is split into two actual objects. Rebasing makes these non-issues.
Features like rebasing are really intended to happen before commits are published, in most cases. Developers will make a series of commits and then, while still working locally, rebase the repository to build a useful history and then publish those changes to his collaborators. So it’s not so much that rebasing allows or encourages revisionist histories, but that it allows developers to control the state of their present or the relative near future.

Bonus: The git stash

The git stash isn’t so much a concept that’s difficult to grasp, but a tool for interacting with the features describe above that is pretty easy to get. Imagine one of the following cases:

You’re making changes to a repository, you’re not ready to commit, but someone writes you an email, and says that they need you to quickly change 10 or 12 strings in a couple of files (some of which you’re in the middle of editing,) and they need this change published very soon. You can’t commit what you’ve edited as that might break something you’re unwilling to risk breaking. How do you make the changes you need to make without committing your in-progress changes?

You’re working in a topic branch, you’ve changed a number of files, and suddenly realized that you need to be working in a different branch. You can’t commit your changes and merge them into the branch you need to be using that would disrupt the history of that branch. How do you save current changes and then import them to another branch without committing?

Basically invoke git stash which saves the difference between the index (e.g. HEAD) and the current state of the working directory. Then do whatever you need to do (change branches, pull new changes, do some other work,) and then invoke git stash pop and everything that was included in your stash will be applied to the new working copy. It’s sort of like a temporary commit. There’s a lot of additional functionality within git stash but, that’s an entirely distinct bag of worms.

Onward and Upward!

It's A Great Time To Be Alive

Wed, 22 Sep 2010 00:00:00 +0000

… when you’re a tycho.

I’ve not been blogging very much. There were even two weeks where I totally forgot to post anything to the blog. I’m not sure if there’s been a hiatus of that length at any other point in the last three years. Strange, but not bad. I’ve been writing--not as much as I’d like, but enough--but nothing has really managed to filter through to the blog. Seems, then, like a good time for a general “what the hell have you been up to and what are you working on post?”

I’ve been plugging away on the novel project for a while I’m in the home stretch: two and a half chapters to write. I know I can write a chapter in a couple of weeks if I put my mind to it, and have a good weekend day to do nothing other than sit around and write. I’ve discovered some things about the story that have made it much easier and clear for me to write. A character that I thought was the main is a horrible point-of-view character, I have a more clear idea of what I’ve been trying to get after the entire story, I’ve started to really like the project again. The problem now, is just finding time to finish it.

I moved nearly two months ago: it was a good thing indeed. I’m closer to friends and activities that I find fulfilling. Although I’m not in the heart of my city, I have a train pass and am a block away from a train station that will get me into the city pretty much whenever I want. Train passes are a fantastic innovation, and my only real complaint (on this note) is that for varied historical reasons Tran service in America is woefully pathetic. But it’s great to be in a place where public transit is a viable option for most commuting.

I’ve recently started dating someone, which is a terribly atypical experience for me given how independent and quirky I am in “default mode.” Nifty though, and utterly unintended: but there’s something delightful about the whole deal. Who am I to argue?

And by “recently started” I mean almost three months ago. Right.

I got a new phone. After a long time with a Blackberry Bold, I took an upgrade and got the new Blackberry Torch. I had been strongly pondering an Android phone, but the one I wanted most (HTC EVO) is on a network that I’d like to avoid, and the other offerings seem lackluster. Blackberries have never (rarely?) been exciting, but they have worked, and there are some features that make sense to me (massively configurable ring/tone behavior, central messages feed that all applications can create entries in.) Having said that, I’m not wild about being on a closed platform, though I think we shouldn’t delude ourselves into thinking that from the user’s perspective that Android is an open platform.

With a web browser that is basically mobile Safari, and applications to do the major things I need to do (IM, email, calendar, GPS, web browsing, address book, twitter, alarm clock, ebook reading, emergency ssh,) and a device that is build wonderfully, I couldn’t be happier. I wish there was a better solution for writing/note taking/outlining for the Blackberry, but these things happen.

It’s probably a symptom of the commute as much as anything but I’ve had a hard time figuring out how to write on the train. I did ok for a while, and then I’ve had a horrible time with it for the past few weeks. The new plan is to do a little reboot of my organizational system and to switch back to using the laptop as my only computer.

The first part, the organizational reboot, is important because I’m not used to thinking about my available time as existing in the “shape” that it currently exists. If I’ve learned one thin in the past couple of years, its that staying organized and on top of ones personal project is almost entirely dependent upon being able to successfully break apart big projects into “doable” pieces that you can tackle in the time you have. That sounds easy, but it’s pretty hard to know what your project will require and what your near-future is going to hold enough to be able to get everything to line up. Sometimes it requires little reboots.

The second, much more technological remedy, is mostly that while the separation between my work system (and tasks) and my personal system (and tasks) has been nice, in practice it doesn’t work that well. The context switch between the train and the office is too hard, and not being able to just unplug and shut down, means that the start up/shut down costs for the context switches are simply too high and I’m the one who feels the impact. So re-merging my systems seems to be the way to win this game, at least for the moment.

I think I’m going to ultimately invest in a hot spare laptop that’s mostly identical to my current laptop to reduce some of the anxiety regarding the “what happens if my machine croaks,” worry. Which is always a fear.

Wikis are not Documentation

Tue, 14 Sep 2010 00:00:00 +0000

It seems I’m writing a minor series on the current status (and possible future direction?) of technical writing and documentation efforts. Both in terms of establishing a foundation for my own professional relevancy, as well as in and for itself because I think documentation has the potential to shape the way that people are able to use technology. I started out with Technical Writing Appreciation and this post will address a few sore points regarding the use of wikis as a tool for constructing documentation.

At the broadest level, I think there’s a persistent myth regarding the nature of the wiki and the creation of content in a wiki that persists apart from their potential use in documentation projects. Wiki’s are easy to install and create. It is easy to say “I’m making a wiki, please contribute!” It is incredibly difficult to take a project idea and wiki software and turn that into a useful and vibrant community and resource. Perhaps these challenges arise from the fact that wiki’s require intense stewardship and attention, and this job usually falls to a very dedicated leader or a small core of lead editors. Also, since authorship on wikis is diffuse and not often credited, getting this kind of leadership and therefore successfully starting communities around wiki projects can be very difficult.

All wikis are like this. At the same time, I think the specific needs of technical documentation makes these issues even more prevalent. This isn’t to say that wiki software can’t power documentation teams, but the “wiki process” as we might think of it, is particularly unsuited to documentation.

One thing that I think is a nearly universal truth of technical writing is that the crafting of texts is the smallest portion of the effort of making documentation. Gathering information, background and experience in a particular tool or technology is incredibly time consuming. Narrowing all this information down into something that is useful to someone is a considerable task. The wiki process is really great for the evolutionary process of creating a text, but it’s not particularly conducive to facilitating the kind of process that documentation must go through.

Wikis basically “here’s a simple editing interface without any unnecessary structure: go and edit, we don’t care about the structure or organization, you can take care of that as a personal/social problem.” Fundamentally, documentation requires an opposite approach, once a project is underway and some decisions have been made, organization isn’t the kind of thing that you want to have to manually wrestle, and structure is very necessary. Wikis might be useful content generation and publication tools, but they are probably not suited to supporting the work flow of a documentation project.

What then?

I think the idea of a structured wiki, as presented by twiki has potential but I don’t have a lot of experience with it. My day-job project uses an internally developed tool, and a lot of internal procedures to enforce certain conventions. I suspect there are publication, collaboration, and project management tools that are designed to solve this problem, but I’m not particularly familiar with anything specific. In any case, it’s not a wiki.

Do you have thoughts? Have I missed something? I look forward to hearing from you in comments!

Technical Writing Appreciation

Thu, 09 Sep 2010 00:00:00 +0000

I’m a technical writer. This is a realization that has taken me some time to appreciate and understand fully.

Technical writing is one of those things that creators of technology, a term that I will use liberally, all agree is required, but it’s also something that’s very difficult to do properly. I think this difficulty springs from the following concerns: What constitutes describing a technology or process in too much detail? Not enough? Are all users of a technology able to make use of the same level of documentation sets? If users are too diverse, what is the best way to make sure that their needs are addressed: do we write parallel documentation for all users, or do we try and equalize less advanced users so that the core documentation is useful to everyone?

The answers to these questions vary of course with the needs of the product being documented and the use cases, but I think resolving these concerns presents a considerable challenge to any kind of technical documentation project, and the way that the documentation developers resolve these issues can have a profound effect not only on the documentation itself but the value and usefulness of the documentation itself. As I’ve been thinking about the utility and value of technical writing, a professional hazard, I’ve come up with a brief taxonomy of approaches to technical writing:

First, there’s the document everything approach. Starting with a full list of features (or even an application’s source) the goal here is to make sure that there’s no corner unturned. We might think of this as the “manual” approach, as the goal is to produce a comprehensive manual. These are great reference materials, particularly when indexed effectively, but the truth is that they’re really difficult for users to engage with, even though they may have all the answers to a users questions (e.g. “RTFM.") I suspect that the people who write this kind of documentation either work closely with developers or are themselves developers.
Second, there’s what I think of as the systems or solutions document, which gives up comprehensiveness, and perhaps even isolation to a single tool or application, and documents outcomes and processes. They aren’t as detailed, and so might not answer underlying questions, but when completed effectively they provide an ideal entry point into using a new technology. In contrast to the “manual” these documents are either slightly more general interest or like “white papers.” This class of documentation, thus, not simply explains how to accomplish specific goals but illuminates technical possibilities and opportunities that may not be clear from a function-based documentation approach. I strongly suspect that the producers of this kind of documentation are very rarely the people who develop the application itself.
In contrast to the above, I think documentation written for education and training purposes, may appear to be look either a “manual” or a “white paper,” but have a fundamentally different organization and set of requirements. Documentation that supports training is often (I suspect) developed in concert with the training program itself, and needs to impart a level of deeper understanding of how a system works (like the content of a manual,) but doesn’t need to be comprehensive, and needs mirror the general narrative and goals of the training program.
Process documentation finally, is most like solution documentation, but rather than capture unrealized technological possibilities or describe potentially hypothetical goals, these kinds of documents capture largely institutional knowledge to more effectively manage succession (both by future iterations of ourselves, and our replacements). These documents have perhaps the most limited audience, but are incredibly valuable both archival (e.g. “How did we used to do $*?") and also for maintaining consistency particularly amongst teams as well as for specific tasks.

I think the fundamental lesson regarding documentation here isn’t that every piece of technology needs lots and lots of documentation, but rather that depending on the goals for a particular technology development program or set of tools, different kinds of documentation may be appropriate, and more useful in some situations.

As a secondary conclusion, or direction for more research: I’d be interested in figuring out if there are particular systems that allow technical writers (and development teams) to collect multiple kinds of information and produce multiple documentation for different organizations. Being able to automatically generate different wholes out of documentation “objects” if we may be so bold.

I must look into this. Onward and Upward!

Covered In LaTeX

Tue, 31 Aug 2010 00:00:00 +0000

Although I haven’t used LaTeX much in the past few years, it was one of the primary tools that hastened my shift to using GNU/Linux full time. Why? I’d grown sick of fighting with document preparation and publishing systems (e.g. Microsoft Word/Open Office), and had started using LaTeX on my Mac to produce all of my papers and documents that needed to be output to paper-formats. Why switch? Because after a certain point of having every tool you use be Free software (because it’s better!), it becomes easier and more cost effective to just jump the gun and by commodity hardware and use a system that’s designed to support this kind of software (managing a large selection lots of free software packages on OS X can become cumbersome).

So why LaTeX? What’s the big deal? Why do I care now? Well…

LaTeX is a very usable front-end/set of macros for the TeX typesetting engine. Basically, you write text files in a particular way, and then run LaTeX (or pdflatex) and it generates the best looking PDF in the world of your document. You get full control over things that matter (layout, look and feel) and you don’t have to worry about things that ought to be standard (titles, headlines, citations with BibTeX, page numbering, hyphenation). The best part, however, is that once you figure out how to generate a document correctly once, you never have to figure it out again. Once you realize that most of the things you need to output to paper are in the same format, you can use the same template and be able to generate consistently formated documents automatically. There’s a “compile” step in the document production process, which means changes aren’t often immediately recognizable, but I don’t think this is a major obstacle.

Word processing and document preparation is a critical component of most common computer users. At least, I’d assume so, though I don’t have good numbers on the subject. In any case, I think it might be an interesting project to see how teaching people how to use LaTeX might both improve the quality of their work, and also the way that they’re able to work. It’s advanced, and a bit confusing at first, but I’d suspect that once you got over the initial hump LaTeX presents a more simple and consistent interface: you only get what you tell it to give you and you only see the functionality that you know about. This might make the discovery of new features more difficult, but it doesn’t limit functionality.

I’m not sure that this post is the right space to begin a lesson or series on getting started with LaTeX, but I think as a possible teaser (if there’s interest) that the proper stack for getting started with LaTeX would consist of:

A TeXlive distribution. You need the basic tool kit including pdflatex, TeX, Metafont, LaTeX, and BibTeX.
A Text Editor with LaTeX support: emacs, TextMate, etc. Plain text can be difficult and cumbersome to edit unless you have the right tools for the job, which include a real text editor.
Some sort of macro or snippet expansion system. TeX is great. But it’s also somewhat verbose, and having an easy way to insert text into your editing environment, both for templates but also for general operations (emphasis, notes, etc.) is incredibly useful, and reduces pain greatly.
A template management system. This probably needn’t be a formal software system, but just something to organize and store the basic templates that you will use.

And the rest is just learning curve and practice. Onward and Upward!

Console Use for the Uninitiated

Fri, 13 Aug 2010 00:00:00 +0000

I have a computer, an old laptop, that I mostly use as the foundation of my stereo system. It’s just a basic system with a few servers (a web server and the music player daemon), and it doesn’t have a running window manager. This configuration usually doesn’t bug me: I connect remotely and the computer sits under the couch, but since my recent move I’ve not had a network connection at home and I’ve defaulted to playing music and managing the system from the console.

This works just fine for me. The virtual terminals aren’t noticeably different from the terminal I get over ssh (as you would expect/hope), except now I have to walk across the room. The people who listen to music with me haven’t yet been other terminal geeks, and so I’ve taken on the role of stereo whisperer. Until a friend looked over my shoulder and wanted to change the track. Using the console is sometimes (often) a slippery slope.

I realized immediately that this situation was much more conducive to learning to use the console than the kinds of introductions to using the console that I’ve typically written. The commands we used were very limited: the mpc program that acts as a simple command-line client to the music player daemon (mpd) and grep. We also used the pipe operator.

There are thousands of commands on most Linux/UNIX systems and remembering all of them can be a bit challenging. The console is a limiting environment basically you can do one thing at a time with it, and you don’t have a lot of leeway with common errors. At the same time, there are a great number of programs and commands that a beginner has no way of knowing about or knowing when to use. Legitimately, the console is both too limiting and expansive to be quickly accessible to the uninitiated. Starting with a very limited selection of commands is way to break through this barrier.

The terminal environment is also very “goal-oriented.” Enter a command to generate some sort of output or result and then repeat. At the end your system will have done something that you needed it to, and/or you’ll learn something that you didn’t already know. When you’re just trying to learn, all of the examples seem fake, contrived, and bothersome because many people already have an easy way of accomplishing that task using GUI tools. Learning how the terminal works, thus, needs a real example, not just a potentially realistic example.

The great thing, I think, is that once you have a need to learn command line interaction, it makes a lot of sense even to people who aren’t die-hard geeks: Commands all have a shared structure that is fairly predictable and inconsistencies are apparent. Perhaps most importantly the command line’s interaction model is simple: input a command and get a response. Advanced users may be able to bend the interaction model a bit, but it is undeniably parsimonious.

It seems, in conclusion, that the command-line is easy to learn for the new user for the same reason it is beloved by the advanced. Ongoing questions, include:

If this kind of realization were to catch on, how might it affect interaction design in the long run? Might “simple to design” and “easy to use” move closer together?

Is there a way to build training and documentation to support users who are new to this kind of interaction style?

Collaborative Technology

Thu, 05 Aug 2010 00:00:00 +0000

I agreed to work on an article for a friend about the collaborative technology “stuff” that I’ve been thinking about for a long time. I don’t have an archive that covers this subject, but perhaps I should, because I think I’ve written about the technology that allows people to make things with other people a number of times, though I have yet to pull together these ideas into some sort of coherent post or essay.

This has been my major post-graduation intellectual challenge. I have interests, even some collected research, and no real way to turn these half conceptualized projects into a “real paper.” So I’ve proposed working with a friend to collect and develop a report that’s more concrete and more comprehensive than the kind of work that I’ve been attempting to accomplish on the blog. Blogging is great, don’t get me wrong, but I think it leads to certain kinds of thinking and writing (at least as I do it,) and sometimes other kinds of writing and thinking are required.

Regarding this project, I want to think about how technology like “git” (a distributed version control system) and even tools like wiki’s shape the way that groups of people can collaborate with each other. I think there’s an impulse in saying “look at the possibilities that these tools create! This brave new world is entirely novel, and not only changes the way I am able to complete my work, but how I look at problems, and make it so much easier for me to get things done..” At the same time, the technology can only promote a way of working it doesn’t necessarily enforce a way of working, nor does any particular kind of technology really remove the burdens and challenges of “getting things done.” More often perhaps new kinds of technology, like distributed version control, is responsible for increasing the level of abstracting and allowing us (humans) to attend to higher order concerns.

Then, moving up from the technology, I think looking at how people use technology in this class allows us to learn a great deal about how work is accomplished. We can get an idea of when work is being done, an idea of how quality control efforts are implemented. Not only does this allow us to demystify the process of creation, but having a more clear idea of how things are made could allow us to become better makers.

The todo list, then, is something like:

Condense the above into something that resembles a thesis/argument.
Become a little more familiar with the git-dm (“data mining”) tool that the Linux Foundation put together for their “state of Kernel development.”
Develop some specific questions to address. I think part of my problem above and heretofore has been that I’m saying “there’s something interesting here, if we looked,” rather than. “I think w kind of projects operate in x ways, where y projects will operate in z ways.”
Literature review. I’ve done some of this, but I’ve felt like I need to do even more basic methodological and basic theory reading. And even though an unread Patterns of Culture is on my bookshelf, I don’t need to read that to begin reading articles.

That’s a start. Feedback is always useful. I’ll keep you posted as I progress.

Saved Searches and Notmuch Organization

Fri, 30 Jul 2010 00:00:00 +0000

I’ve been toying around with the Notmuch Email Client which is a nifty piece of software that provides a very minimalist and powerful email system that’s inspired by the organizational model of Gmail.

Mind you, I don’t think I’ve quite gotten it.

Notmuch says, basically, build searches (e.g. “views”) to filter your email so you can process your email in the manner that makes the most sense to you, without needing to worry about organizing and sorting email. It has the structure for “tagging,” which makes it easy to mark status for managing your process (e.g. read/unread, reply-needed), and the ability to save searches. And that’s about it.

Functionally tags and saved searches work the way that mail boxes in terms of the intellectual organization of mailboxes. Similarly the ability to save searches, makes it possible to do a good measure of “preprocessing.” In the same way that Gmail changes the email paradigm by saying “don’t think about organizing your email, just do what you need to do,” not much says “do less with your email, don’t organize it, and trust that the machine will be able to help you find what you need when the time comes.”

I’ve been saying variations of the following for years, but I think on some level it hasn’t stuck for me. Given contemporary technology, it doesn’t make sense to organize any kind of information that could conceivably be found with search tools. Notmuch proves that this works, and although I’ve not been able to transfer my personal email over, I’m comfortable asserting that notmuch is a functional approach to email. To be fair, I don’t feel like my current email processing and filtering scheme is that broken, so I’m a bad example.

The questions that this raises, which I don’t have a particularly good answers for, are as follows:

Are there good tools for the “don’t organize when you can search crew,” for non-email data? And I’m not just talking about search engines themselves (as there are a couple: xapian, namazu), or ungainly desktop GUIs (which aren’t without utility,) but the proper command-line tools, emacs interfaces, and web based interfaces?
Are conventional search tools the most expressive way of specifying what we want to find when filtering or looking for data? Are there effective improvements that can be made?
I think there’s intellectual value created by organizing and cataloging information “manually,” and “punting to search” seems like it removes the opportunity to develop good and productive information architectures (if we may be so bold.) Is there a solution that provides the ease of search without giving up the benefits that librarianism brings to information organization?

Gear Mongering

Tue, 27 Jul 2010 00:00:00 +0000

At the end of the day (or even the beginning,) I’m just another geek, and despite all of the incredibly (I’d like to think at least) reasoned ways I think about the way I use technology, I occasionally get an old fashioned hankering for something new. We’ve all had them. Perhaps my saving graces are that I do occasionally need new things (computers wear out, cellphones are replaced, needs change), and the fact that I’m both incredibly frugal and task oriented (rational?) about the way I use technology.

But I’m still a geek. And gear is cool. Thoughts, in three parts.

Part One, Phones and the HTC EVO

I’ve been using a Blackberry for 18 months, and I’ve come to two conclusions: blackberries are great and they have the “how ot integrate messaging into a single usable interface.” I was skeptical at first, and it’s very simple, and I’ve never quite gotten my email to function in an ideal way, largely because I think the Blackberry works really well for email. It’s everything else that I might want to do with my phone that I can’t, and I’d probably like to: I have an SSH client and it’s nearly usable. Nearly. I have IM clients, that are nearly functional. Nearly.

When I got the Blackberry, is and was the most important communication I was doing. I worked for a very email-centric company, and I wanted to be able to stay in the email-loop even when I was off doing something else. These days, IRC and XMPP are a far more central feature of my digital existence, and I tend to think that it’s not an Internet connection if I can’t open SSH. I’m also switching on a much longer public-transit focused commute in the next few week, and being able to do research for writing projects will be nice. I’m not sure what the best solution is exactly, though the HTC EVO is a pretty swell phone.

As the kids these days say, “Do want.”

Part two, Infrastructural Computing and Home Servers

I’ve fully adopted an infrastructural approach to technology, at least with regards to my own personal computing. That was a mouthful. Basically, although I work locally on the machine that’s in front of me (writing, email, note taking, collaboration,) much of the “computing” that I interact with isn’t actually connected to the machine I interact with directly. In some ways, I suppose this is what they meant when they said “cloud computing,” but the truth is that my implementation is somewhat more… archaic: I use a lot of SSH, cron, and a little bailing wire to distribute computing tasks far and wide, and the process of moving everything in my digital world from a laptop that I carried around with me everywhere (college,) to a more sane state of affairs has been a long time coming.

Right.

The long story short is that aside from a machine (my old laptop) that’s at capacity powering my “stereo,” I don’t have any computer’s at home aside from my laptop, and I tend to take it everywhere with me, which makes it unideal for some sorts of tasks. Furthermore, without an extra machine setting around, file storage, some kinds of backups, are somewhat more complicated than I’d like. So, I’m thinking about getting some sort of robust server-type machine to stick in a corner in my apartment.

Not exactly sure what the best option is there. I’m burdened by: frugality, sophisticated tastes, and the notion that having quality hardware really does matter.

More thinking required.

Part three, More Laptops

So I might have a laptop related illness. Back in the day, laptops always seemed like a frivolity: underpowered, never as portable as you wanted, awkward to use, and incredibly expensive. Now, laptops are cheap, and even the Atom-based “netbooks,” are functional for nearly every task. I tend to buy used Thinkpad Laptops, and as I think about it, I’ve probably spent as much on the three Thinkpads, all of which are still in service, as I did on any one mac laptop.

The thing about my current laptop is that when you think about it, it’d make a decent home server: the processor has virtualization extensions, the drive is fast (7200 rpm) and it can handle 4 gigs of ram (and maybe more.) What more could I want? And I distributed things correctly, the “server” laptop could be pressed into service as a backup/redundant laptop, in case something unforeseen happened.

Or I could dither about it for another few months, and come to some other, better, fourth solution.

Onward and Upward!

Creating Useful Archives

Thu, 22 Jul 2010 00:00:00 +0000

I’ve done a little tweaking to the archives for dialectical futurism recently, including creating a new archive for science fiction and writing and being who I am this has inspired a little of thought regarding the state and use of archives of blogs.

The latest iteration of this blog has avoided the now common practice of having large endless lists of posts organized by publication month or by haphazardly assigned category and tag systems. While these succeed at providing a complete archive of every post written, they don’t add any real value to a blog or website. I’m convinced that one feature of successful blogs moving forward will be archives that are curated and convey additional value beyond the content of the site.

Perhaps blogs as containers for a number of posts will end up being to ephemeral than I’m inclined to think about them, and will therefore not require very much in the way of archives, Perhaps, Google’s index will be sufficient for most people’s uses. Maybe. I remain unconvinced.

Heretofore, I have made archives for tychoish as quasi-boutique pieces: collections of the best posts that address a given topic. This is great from the perspective of thinking about blog posts as a collection of essays, but I’ve started to think that this may be less less useful if we think of blogs as a collection of resources that people might want to have access to beyond it’s initial ephemeral form.

Right now my archives say “see stuff from the past few months, and several choice topics on which I wrote vaguely connected sequences of posts.” The problem with the list of posts from the last few months is that beyond date, there’s not a lot of useful information beyond the title and the date. The problem with the topical archives is that they’re not up to date, their not comprehensive even for recent posts, and there’s little “preview” of a given post beyond it’s title. In the end I think the possibility of visiting a topical archive looking for a specific post and not finding it is pretty large.

In addition to editorial collecting, I think archives, guides, or indexes of a given body of information ought to, provide some sort of comprehensive method for accessing information. There has to be some middle ground.

I think the solution involves a lot of hand mangling of content, templates, and posts. I’m fairly certain that my current publication system is probably not up for the task without a fair amount of mangling and beating. As much as I want to think that this is an problem in search of the right kind of automation, I’m not sure that’s really the case. I’m not opposed to editing things by hand, but it would increase the amount of work in making any given post significantly.

There is, I suspect, no easy solution here.

In Favor of Simple Software

Tue, 13 Jul 2010 00:00:00 +0000

I’ve spent a little bit of time addressing some organizational and workflow angst in the past few weeks, and one thing I’d been focusing on had been to update and fine tune my emacs (text editor) and irssi (irc/chat) configuration. Part of my goal had been to use irssi-xmpp to move all of my chat/synchronous communication into one program; unfortunately I’ve not been able to get irssi-xmpp to build and function in a fully stable way. This is probably because I’m hard on software and not because of anything specific to the software itself.

In any case, this lead me to come to the following conclusion about these programs, as they are probably the two most central and most heavily used applications in my arsenal, and without a doubt are the applications that I enjoy using the most. I scribbled the following note a few days ago in preparation for this entry:

In many ways the greatest advance or feature that these programs provide isn’t some killer feature, it’s a simple but more powerful abstraction that allows users to interact with their problem domain. Emacs is basically a text-editing application framework, and provides users with some basic fundamentals for interacting with textual information, and a design that allows users to create text editing modalities or paradigms which bridge the divide between full blown applications and custom configurations. By the same token, Irssi is really a rather simple program that’s easy to script, and contains a number of metaphors that are useful for synchronous communication (chat.)

And we might be able to expand this even further: these are two applications that are not only supremely functional, but are so usable because they are software projects that really only make sense in context of free software.

I want to be very careful here: I don’t want to make the argument that free software isn’t or can’t be commercial, because that’s obviously not the case. At the same time, free software, like these applications needn’t justify itself in terms of “commercial features,” or a particular target market in order to remain viable. It’s not that these programs don’t have features, it is that they have every feature, or the potential for every feature, and are thus hard to comprehend and hard to sell. Even if it only takes a use over a short period of time for users to find them incredibly compelling.

The underlying core extensibility that both of these “programs” have is probably also something that is only likely to happen in the context of open source or free software. This isn’t to suggest that proprietary software doesn’t recognize the power or utility of extensible software, but I don’t think giving users so much control over a given application makes sense from a quality control perspective. Giving users the power to modify their experience of software in an open ended fashion, also gives them the power to break things horribly, and that just doesn’t make sense from a commercial perspective.

There’s probably also some hubris at play: free software applications, primarily these two, are written by hackers, with a target audience of other hackers. Who needs a flexible text editing application framework (e.g. emacs), but other programmers. And the primary users of IRC for the past 8-10 years have largely been hackers and developers and other “geek” types. irssi is very much written for these kinds of users. To a great extent, I think it’s safe to suggest that when hackers write software for themselves, this is what it looks like.

The questions that must linger is: why isn’t other software like this? (Or is it, and I’m missing it in my snobbishness,) and where is the happy medium between writing software for non-hackers and using great software (like these) to “make more hackers.”

Onward and Upward!

Focus and Context Switching

Thu, 08 Jul 2010 00:00:00 +0000

This post is inspired by Cory Doctorow’s Inventory about the technology he uses for his work and ongoing personal angst and gadget lust.

For the past six months or so I collapsed my entire computing existence into one, single, computer. It’s a nifty Thinkpad x200, a small laptop with just the right balance of usable screen space, sufficient computing power, and great build quality. And when I say “everything,” I mean it. I hook it up to a monitor and keyboard at the office and do my day-job (technical writing/sys admin stuff) on this system, I have a similar “desktop” situation at home, and I write fiction, do all of my email, write and blog posts, off of this system. I even have development web servers running here.

In a number of ways it’s great, and I wouldn’t trade this for the world. Everything just works the way I want it to, and I never have to worry that I’ve left some important edits to a file on a system elsewhere. Everything is always with me.

Now to be fair, I have additional computers. My old(der) desktop at home keeps backups of files, plays music, and does a number of other tasks. I have (and use) the server that this and other websites run on for some tasks, and I have another instance at the office that manages some work functions, but despite their varying physical distance from me at any given point, my interactions with these computers is always as if they’re remote. When I use a computer, it’s this one.

Now there isn’t a real problem here, except that from a workspace and mindspace perspective the context switching can be somewhat complicated and frustrating. While I’ve got most of the kinks worked out of the docking (monitors and keyboards) process figured out, it takes me a few moments to settle into or out of “laptop-mode” or “workdesk-mode” or “homedesk-mode.” While not having to worry if my files are all up to date, it’s also somewhat distracting for all my different projects to be open all the time. The article I’m working on for work is always open and a few key presses away from the novel I’m working on or the latest in-progress blog post.

Again, this isn’t a really huge issue, but it means that when I get somewhere and want to begin working I have to take a deep breath, and spend a moment or two getting going again. These do fit into the category of “first world problems,” and I’m not sure if there’s a really easy solution. I’ve toyed with a number of resolutions to this angst:

Distribute my existing machines such that I have a machine that lives at work and a machine that lives at home so that I can just sit down at a desk and start working without shenanigans. Given the available hardware, this might mean that I’d spend most of my time using systems that I’m not particularly fond of.
I’ve toyed with having a “tycho writing laptop” that wouldn’t have a web browser installed, for more distraction free writing. I’ve got my old laptop set up to do this, but I’m not likely to take it anywhere (i.e. for the commute,) so it might make sense to get a little netbook for the train for this function, but that seems like overkill.
I might redistribute more of my workload to servers rather than doing everything on the laptop. I’m thinking about having the terminal sessions that I use for email primarily live in screen sessions elsewhere.
I’ve thought about getting a second laptop, (like my main system at the moment,) both for redundancy and to help reduce the cost of switching between various contexts.

As I’ve been toying with this, I’ve been making a number of tweaks to my work flow to help address the pain of context switching, most of which are too trivial and too specific to outline here. Mostly I’ve been tweaking some customizations, improving how I use virtual desktops. While these tweaks have improved things greatly, better internal system management doesn’t solve the underlying issues: it takes time to reconnect to networks, to close tabs in the web browser, to get to the relevant files open in emacs, and navigate to the proper desktop. All the while other contexts (other files, other virtual desktops,) lurk nearby.

And figuring out how to solve this problem involves a certain amount of “head game” for me: avoiding having “the old laptop” be the primary computer for a given task, making sure that I don’t need a network connection for essential tasks. Assorted other weirdnesses.

If anyone recognizes features of this angst that in their own work they’ve managed to resolve, I’d love to hear about your setup.

Fan Fiction is Criticism

Tue, 06 Jul 2010 00:00:00 +0000

Thanks to `Shaun Duke <http://skiffyandfanty.wordpress.com>`_ for inspiring this little rant.

I must confess that I’m mostly uninvolved in the world of fan fiction these days, though I have traveled in “fanish” circles at various points in my past. It’s not because I don’t think fans have interesting things to say about literature and media, or that I don’t think what’s happening in fandom important and fascinating. No, I’m mostly withdrawn because I have too much on my plate and participating in fandom doesn’t really contribute to the specific goals I have at this moment. But I sometimes feel that way about social science.

In any case, I’d like to put forth the following arguments for viewing fan fiction as a form a literary criticism rather than a literary attempt in it’s own right:

Fan fiction is a form of literary criticism. Sure it’s casual, sure it’s written in the forum of a story, but the fan fictioner and the critic both write from the same core interest in interpreting texts and using varying readings of texts to create larger understandings of our world.
The fact that fan fiction looks like a story, is mostly distracting to what’s happening in these texts. Fan fiction, has always been written in communities. The people who read fan fiction are largely the people who write fan fiction. Fan fiction inspires
The quality of fan fiction is also largely irrelevant to the point of whether fan fiction is worthwhile. More so than other forms of writing, fan fiction is less about the technical merits of the text, and more about the discursive process under which the texts are created. Better quality writing makes better fan fiction, certainly but I don’t think fan fiction centers on those kinds of values.
Copyright, and the “intellectual property” status of fan fiction is also sort of moot. It’s true that if we’re being honest fan fiction impinges upon the copyright of the original author. At the same time, fan fiction doesn’t really hurt creators: people aren’t confused that fan fiction is “real fiction,” fan fiction by and large doesn’t divert sales from “real fiction,” and so forth. Sure, it’s a bit weird for some others to find other people playing in their sand boxes, but the truth is that authors have never had a great deal of control over what happens to their work post-publication, so it’s fair.

Additionally, I think that fan fiction accomplishes something that are incredibly powerful and worthwhile that “normal” fiction cannot accomplish. Writing fan fiction can be, I’d wager, an incredibly effective educational experience for new writers, particularly genre fiction writers. By providing a very fast feedback loop with an audience of readers and writers (and lovers of literature and story telling.) Not to mention the fact that because fan fiction tends to be somewhat ephemeral and there’s a wealth of inspiration and impetus for fiction, fan writers can write a lot, and if they choose in a very productive sort of way.

And that is almost certainly a good thing.

The Overhead of Management

Thu, 01 Jul 2010 00:00:00 +0000

Every resource, every person, every project, every machine you have to manage comes with an ongoing cost. This is just as true of servers as is it is of people who work on projects that you’re in charge of or have some responsibility for, and while servers and teammates present very different kinds of management challenges, working effectively and managing management costs across contexts is (I would propose) similar. Or at least similar enough to merit some synthetic discussion.

There’s basically only one approach to managing “systems administration costs,” and that’s to avoid it as much as possible. This isn’t to say that sys admins avoid admining, but rather we work very hard to ensure that systems don’t need administration. We write operating systems that administer themselves, we script procedures to automate most tasks as much as possible (the Perl programing language was developed and popularized for use of easing the administration of UNIX systems,) and we use tools manage larger systems more effectively.

People, time, and other resources cannot be so easily automated, and I think in response there are two major approaches (if we can create a somewhat false dichotomy for a moment:)

On the one hand there’s the school of thought that says “admit and assess management costs early, and pay them up front.” This is the corporate model in many ways. Have (layers upon layers of) resources dedicated to managing management costs, and then let this “middle management” make sure that things get done in spite of the management burden. On servers this is spending a lot of time choosing tools, configuring the base system, organizing the file system proactively, and constructing a healthy collection of “best practices.”

By contrast, the other perspective suggests that management costs should only be paid when absolutely necessary. make things, get something working and extant and then if something needs to be managed later, do it then and only as you need. On some level this is inspiring philsophy behind the frequent value of favoring “working code” over “great ideas” in the open source world.¹ Though I think they phrase it differently, this is the basic approach that many hacker-oriented start ups have taken, and it seems to work for them. On the server, this approach is the “get it working,” approach, and these administrators aren’t bothered by having to go in every so often to “redo” how things are configured, and I think on some level this kind of approach to “management overhead” grows out of the agile world and the avoidance of “premature optimizations.”

But like all “somewhat false dichotomies,” there are flaws in the above formulation. Mostly the “late management” camp is able to delay management most effectively by anticipating their future needs (either by smarts or by dumb luck) early and planning around that. And the “early management” camp has to delay some management needs or else you’d be drowned in overhead before you started: and besides, the MBA union isn’t that strong.

We might even cast the “early management” approach as being “top down,” and the “late management” camp as being “bottom up.” If you know, we were into that kind of thing. It’s always, particularly in the contemporary moment to look at the bottom-up approach and say “that’s really innovative and awesome, that’s better,” and view “top-down” organizations as “stodgy and old world,” when neither does a very good job of explaining what’s going on and there isn’t inherent radicalism or stodginess in either organization. But it is interesting. At least mildly.

Thoughts? Onward and Upward!

Alan Cox’s Cathedrals, Bazaars and the Town Council ↩︎

Jekyll Publishing

Tue, 29 Jun 2010 00:00:00 +0000

I wrote about my efforts to automate my publishing workflow a couple of weeks ago, (egad!) and I wanted to follow that up with a somewhat more useful elucidation of how all of the gears work around here.

At first I had this horrible scheme setup that dependent on regular builds triggered by cron, which is a functional, if inelegant solution. There’s a lot of tasks that you can give the appearance of “real time,” responsiveness by scheduling more brute tasks regularly enough. The truth is, however, that its not quite the same, and I knew that there was a better way.

Basically the “right way” to solve this problem is to use the “hooks” provided by the git repositories that I use to store the source of the website. Hooks, in this context refer to a number of scripts which are optionally run before or after various operations in the repositories that allow you to attach actions to the operations you perform on your git repositories. In effect, you can say “when I git push do these other things” or “before I git commit check for these conditions, and if they’re not met, reject the commit” and so forth. The possibilities can be a bit staggering.

In this case what happen is: I commit to the tychoish.com repositories a script that synchronizes the appropriate local packages runs and publishes changes to the server. It then sends me an xmpp message saying that this operation is in progress. This runs as the post-commit hook, and for smaller sites could simply be “git push origin master”. Because tychoish is a large site, and I don’t want to be rebuilding it constantly, I do the following:

#!/bin/bash

# This script is meant to be run in a cron job to perform a rebuilding
# of the slim contents of a jekyll site.
#
# This script can be run several times an hour to greatly simplify the
# publishing routine of a jekyll site.

cd ~/sites/tychoish.com/

# Saving and Fetching Remote Updates from tychoish.com
git pull >/dev/null &&

# Local Adding and Committing
git checkout master >/dev/null 2>&1
git add .
git commit -a -q -m "$HOSTNAME: changes prior to an  slim rebuild"  >/dev/null 2>

# Local "full-build" Branch Mangling
git checkout full-build >/dev/null 2>&1 &&
git merge master &&

# Local "slim-bild" Branch Magling and Publishing
git checkout slim-build >/dev/null 2>&1 &&
git merge master &&
git checkout master >/dev/null 2>&1 &
git push --all

# echo done

Then on the server, once the copy of the repo on the server is current with the changes published to it (i.e. the post-update hook), the following code is run:

#!/bin/bash
#
# An example hook script to prepare a packed repository for use over
# dumb transports.
#
# To enable this hook, make this file executable by "chmod +x post-update".

unset GIT_DIR
unset GIT_WORKING_TREE

export GIT_DIR
export GIT_WORKING_TREE

cd /path/to/build/tychoish.com
git pull origin;

/path/to/scripts/jekyll-rebuild-tychoish-auto-slim &

exit

When the post-update hook runs, in runs in the context of the repository that you just pushed to, and unless you do the magic (technical term, it seems) the GIT_DIR and GIT_WORKING_TREE variables are stuck in the environment and the commands you run fail. So basically this is a fancy git pull, in a third repository (the one that the site is built from.) The script jekyll-rebuild-tychoish-auto-slim looks like this:

#!/bin/bash
# to be run on the server

# setting the variables
SRCDIR=/path/to/build/tychoish.com/
DSTDIR=/path/to/public/tychoish/
SITENAME=tychoish
BUILDTYPE=slim
DEFAULTBUILD=slim

build-site(){
 cd ${SRCDIR}
 git checkout ${BUILDTYPE}-build >/dev/null 2>&1
 git pull source >/dev/null 2>&1

 /var/lib/gems/1.8/bin/jekyll ${SRCDIR} ${DSTDIR} >/dev/null 2>&1
 echo \<jekyll\> completed \*${BUILDTYPE}\* build of ${SITENAME} | xmppipe garen@tychoish.com

 git checkout ${DEFAULTBUILD}-build >/dev/null 2>&1
}

build-site;

This sends me an xmpp message when the build has completed. And does the needful site rebuilding. The xmppipe command I use is really the following script:

#!/usr/bin/perl
# pipes standard in to an xmpp message, sent to the JIDs on the commandline
#
# usage: bash$ `echo "message body" | xmppipe garen@tychoish.com
#
# code shamelessly stolen from:
# http://stackoverflow.com/questions/170503/commandline-jabber-client/170564#170564

use strict;
use warnings;

use Net::Jabber qw(Client);

my $server = "tychoish.com";
my $port = "5222";
my $username = "bot";
my $password = ";
my $resource = "xmppipe";
my @recipients = @ARGV;

my $clnt = new Net::Jabber::Client;

my $status = $clnt->Connect(hostname=>$server, port=>$port);

if (!defined($status)) {
  die "Jabber connect error ($!)\n";
}
my @result = $clnt->AuthSend(username=>$username,
password=>$password,
resource=>$resource);

if ($result[0] ne "ok") {
  die "Jabber auth error: @result\n";
}

my $body = '';
while (<STDIN>) {
  $body .= $_;
}
chomp($body);

foreach my $to (@recipients) {
 $clnt->MessageSend(to=>$to,
 subject=>",
 body=>$body,
 type=>"chat",
 priority=>10);
}

$clnt->Disconnect();

Mark the above as executable and put it in your path somewhere. You’ll want to install the Net::Jabber Perl module, if you haven’t already.

The one final note. If you’re using a tool like gitosis to manage your git repositories, all of the hooks will be executed by the gitosis user. This means that this user will need to have write access the “build” copy of the repository and the public directory as well. You may be able to finesse this with the +s “switch uid” bit, or some clever use of the gitosis user group.

The End.

The Meaning of Work

Tue, 22 Jun 2010 00:00:00 +0000

I’ve started to realize that, fundamentally, the questions I’m asking of the world and that I’m trying to address by learning more about technology, center on work and the meaning and process of working. Work lies at the intersection of the all the things that I seem to revisit endlessly: interfaces, collaboration technology, cooperatives and economics institutions, and open source software development. I’m not sure if I’m interested in work because it’s the unifying theme of a bunch of different interests, or this is the base from which other interests spring.

I realize that this makes me an incredibly weird geek.

I was talking to caroline about our respective work environments, specifically about how we (and our coworkers) relocated (or didn’t) for our jobs, and I was chagrined to realize that this novel that I’ve been working at (or not,) for way too long at this point spends some time revolving around these questions:

How does being stuck in a single place and time constrain one agency to effect the world around them?
What does labor look like in a mostly/quasi post-scarcity world?

Perhaps the most worrying thing about this project is that I started writing this story in late August of 2008. This was of course before the American/Financial Services economic crash that got me blogging and really thinking about issues outside of technology.

It’s interesting, and perhaps outside the scope of this post, but I think it’s interesting how since graduating from college, my “research” interests as they were, all work them into fiction (intentionally or otherwise.) I suppose I haven’t written fiction about Free Software/open source, exactly, but I think there’s a good enough reason for that.¹

I’m left with two realizations. First, that this novel has been sitting on my plate for far too long, and there’s no reason why I can’t write the last 10/20 thousand words in the next few months and be done with the sucker. Second, I’m interested in thinking about how “being an academic” (or not) affects the way I (we?) approach learning more about the world and the process/rigor that I bring to those projects.

But we’ll get to that later, I have writing to do.

I write fiction as open source, in a lot of ways, so it doesn’t seem too important to put it in the story as well. ↩︎

There's Always Something New to Learn

Thu, 17 Jun 2010 00:00:00 +0000

Now that I’m fairly confident in my ability to do basic Linux systems administration tasks: manage web and email servers, maintain most Linux systems, convince desktop systems that they really do want to work the way they’re supposed to, I’m embarking on a new learning process. I’ve been playing around with “real” virtualization on my desktop, and I’ve been reading a bunch about systems administration topics that are generally beyond the scope of what I’ve dealt with until now. Here is a selection of the projects I’m playing with:

Getting a working xen setup at home. This requires, learning a bit more about building working operating systems, and also in the not to distant future buying a new (server) computer.
Installing xen on the laptop, because it’ll support it, I have the resources to make it go, and it’ll be awesome.
Learning everything I can about LVM, which is a new (to me) way of managing partitions and disk images, that makes backups, disk snapshots, and other awesomeness much easier. It means, some system migration stuff that I have yet to tinker with, as none of my systems currently support LVM.
Doing package development for Arch Linux, because I think that’s probably within the scope of my ability, because I think it would add to my skill set, and because I appreciate the community, and I want to be able to give back. Also I should spend some time editing the wiki, because I’m really lazy with that.

I guess the overriding lesson of all these projects is a more firm grasp of how incredibly awesome, powerful, and frankly stable Arch Linux is (or can be.) I mean there are flaws of course, but given how I use systems, I’ve yet to run into something show stopping. That’s pretty cool.

The Old Projects Project

Wed, 16 Jun 2010 00:00:00 +0000

Before a road trip a, by now, a couple of months ago, I installed a copy of nginx on my laptop on the hope of doing some web development and working on other projects when I was in the car. For the uninitiated (you mean you don’t all write technical documentation for web developers and systems administrators?!?) nginx is an incredibly powerful web server. As of June 11th, foucualt the server that hosts the Cyborg Institute and tychoish.

This is, almost always, I think, a loosing proposition.

I never get any sort of substantial (or insubstantial) work done during my road trips up-and-down the north east corridor. Not that that’s a bad thing, but I also expect that there’ll be more awake-time when I’m not driving or gossiping.

And there never is.

So the web server sat unused for a long time on my laptop, but recently I’ve been playing with it a bit and I’ve finally gotten a number of cool things set up. I have a local “git web” instance which makes it easier to track progress on local and private projects that are stored in git. Perhaps more importantly, I have set up quick local ikiwiki instances for a number of projects. They’re easy to configure, quick to setup, and while I suppose I could hack something together in nifty for myself, there’s something nifty about being able to take an alternate view of some content and also being able to really preview changes to you work before publishing them.

Also, and the real reason for this post, is that by virtue of this development, I have revisited a few projects that had been lingering in the home directory of my computer for far too long. Which has been a powerful and useful exercise.

By which I mean, it’s been painful.

Besides “the novel,” which has been the lingering and dragging front burner project for a year, there are a number of quasi-serial stories that have lingered in some state of incompleteness for a couple of years now. I’m kind of amazed both at how foreign these stories seem to me both in terms of the style (good to know that I’m a better writer than I was a few years ago,) and also how quickly I can fall right back into the story and tell you every little thing about the world, situation, and moment where I left off.

The mind is, indeed, an amazing thing.

Where my strategy for the past year has been to “plow through and finish the novel,” I think my tactic this summer will be to move all of my projects forward in some way. Small daily writing goals for the novel, combined with somewhat less regular (but more specific) goals with regards to other projects. In the next two months I want to have a fairly active and varied writing schedule worked out that isn’t based around the monthly (or so) weekend binges that I’ve been using for most of the last year.

That’s the plan at any rate.

Jekyll and Automation

Tue, 08 Jun 2010 00:00:00 +0000

As this blog ambles forward, albeit haltingly, I find that the process of generating the site has become a much more complicated proposition. I suppose that’s the price of success, or at least the price of verbosity.

Here’s the problem: I really cannot abide by dynamically generated publication systems: there are more things that can go wrong, they can be somewhat inflexible, they don’t always scale very well, and it seems like horrible overkill for what I do. At the same time, I have a huge quantity of static content in this site, and it needs to be generated and managed in some way. It’s an evolving problem, and perhaps one that isn’t of great specific interest to the blog, but I’ve learned some things in the process, and I think it’s worthwhile to do a little bit of rehashing and extrapolating.

The fundamental problem is that the rebuilding-tychoish.com-job takes a long time to rebuild. This is mostly a result of the time it takes to convert the Markdown text to HTML. It’s a couple of minutes for the full build. There are a couple of solutions. The first would be to pass the build script some information about when files were modified and then have it only rebuild those files. This is effective but ends up being complicated: version control systems don’t tend to version mtime and importantly there are pages in the site--like archives--which can become unstuck without some sort of metadata cache between builds. The second solution is to provide very limited automatically generated archives and only regenerate the last 100 or so posts, and supplement the limited archive with more manual archives. That’s what I’ve chosen to do.

The problem is that even the last 100 or so entries takes a dozen seconds or more to regenerate. This might not seem like a lot to you, but the truth that at an interactive terminal, 10-20 seconds feels interminable. So while I’ve spent a lot of time recently trying to fix the underlying problem--the time that it took to regenerate the html--when I realized that the problem wasn’t really that the rebuilds took forever, it was that I had to wait for them to finish. The solution: background the task and send messages to my IM client when the rebuild completed.

The lesson: don’t optimize anything that you don’t have to optimize, and if it annoys you, find a better way to ignore it.

At the same time I’ve purchased a new domain, and I would kind of like to be able to publish something more or less instantly, without hacking on it like crazy. But I’m an edge case. I wish there were a static site generator, like my beloved jekyll that provided great flexibility, and generated static content, in a smart and efficient manner. Most of these site compilers, however, are crude tools with very little logic for smart rebuilding: and really, given the profiles of most sites that they are used to build: this makes total sense.

I realize that this post comes off as pretty complaining, and even so, I’m firmly of the opinion that this way of producing content for the web is the most sane method that exists. I’ve been talking with a friend for a little while about developing a way to build websites and we’ve more or less come upon a similar model. Even my day job project uses a system that runs on the same premise.

Since I started writing this post, I’ve even taken this one step further. In the beginning I had to watch the process build. Then I basically kicked off the build process and sent it to the background and had it send me a message when it was done. Now, I have rebuilds scheduled in cron, so that the site does an automatic rebuild (the long process) a few times a day, and quick rebuilds a few times an hour.

Is this less efficient in the long run? Without a doubt. But processors cycles are cheap, and the builds are only long in the subjective sense. In the end I’d rather not even think that builds are going on, and let the software do all of the thinking and worrying.

Organize Your Thoughts More Betterly

Wed, 02 Jun 2010 00:00:00 +0000

I’ve been working with a reader and friend on a project to build a tool for managing information for humanities scholars and others who deal with textual data, and I’ve been thinking about the problem of information management a bit more seriously. Unlike numerical, or more easily categorized information data, how to take a bunch of textual information--either of your own production or a library of your own collection--is far from a solved problem.

The technical limitation--from a pragmatic perspective--is that you need to have an understanding not only of the specific tasks in front of you, but a grasp of the entire collection of information you work with in order to effectively organize, manage, and use the texts as an aggregate.

“But wait,” you say. “Google solved this problem a long time ago, you don’t need a deterministic information management tool, you need to brute force the problem with enough raw data, some clever algorithms, and search tools,” you explain. And on some level you’d be right. The problem is of course, you can’t create knowledge with Google.

Google doesn’t give us the ability to discover information that’s new, or powerful. Google works best when we know exactly what we’re looking for, the top results in Google are most likely to be the resources that the most people know and are familiar. Google’s good, and useful and a wonderful tool that more people should probably use but Google cannot lead you into novel territory.

Which brings us back to local information management tools. When you can collect, organize, and manipulate data in your own library you can draw novel conclusions, When the information is well organized, and you can survey a collection in useful and meaningful ways, you can see holes and collect more, you can search tactically, and within subsets of articles to provide. I’ve been talking for more than a year about the utility of curation in the creation of value on-line. and fundamentally I think the same holds true for personal information collections.

Which brings us back to the ways we organize information. And my firm conclusion that we don’t have a really good way of organizing information. Everything that I’m aware of either relies on search, and therefore only allows us to find what we already know we’re looking for, or requires us to understand our final conclusions during the preliminary phase of our investigations.

The solution to this problem is thus two fold: First, we need tools that allow us to work with and organize the data for our projects, full stop. Wiki’s, never ending text files, don’t really address all of the different ways we need to work with and organize information. Secondly we need tool tools that are tailored to the way researchers who deal in text work with information from collection and processing to quoting and citation, rather than focusing on the end stage of this process. These tools should allow our conceptual framework for organizing information to evolve as the project evolves.

I’m not sure what that looks like for sure, but I’d like to find out. If you’re interested, do help us think about this!

(Also, see this post `regarding the current state of the Cyborg Institute <http://www.cyborginstitute.com/2010/06/a-report-from-the-institute/>`_.)

The Schedule

Wed, 26 May 2010 00:00:00 +0000

Wow. Hello blog.

I’m pretty busy. It even seems sort of cliche to complain about such things on ones blog, but I think being busy has coincided with a somewhat larger reevaluation of nearly everything.

Wait, no. I’m not quitting blogging.

I’m actually really proud of the Knowing Mars launch, and it feels really good to have that project “done,” even if I think it needs a major revision, and I have a lot more fiction on my plate that I don’t want to just “let go” like that.

I’m also somewhat displeased with the kind of blog posts that I’ve been writing recently. It seems that I’ve been writing about some basic ideas: my disdain for the way the web functions as a user interface, some general work flow topics, some basic cyber-culture topics, and half way through most of these blog posts I mostly loose interest, and I suspect you have as well.

I’ve had a post in my “write this soon list,” about digging in deeper and striving for a more rich engagement with the topics I try and cover here, and I’ve pretty much failed with that. In any case, this post was supposed to be more about the things that are on my schedule:

I’ve been doing a lot offline these past few weeks. It’s May and that means it’s Morris Dancing season. I seem to have joined an interesting phenomena called “Maple Morris,” (more reflection on that when I’ve processed a bit more,) the usual Mayday festivities, Midwest Morris Ale. And then there are a bunch of singing conventions, which are a great deal of fun and fulfilling, and then there are contra dancing things, but none of these things transmit to quiet weekends alone writing. Or even quiet evening around writing. At least very often. Some highlights of the recent past and near future:

Maple Morris; A my-generation Morris dancing event, last weekend in Boston. A bunch of Morris dancers in my general age rage got together to dance some really challenging dances and to sing great songs. I was totally overwhelmed.
The Midwest Morris Ale; My regular annual Morris dancing ale. This is my 9th consecutive ale (and my 10th anniversary of dancing Morris.)
Since last September, I’ve gone to an all- or mulit- day Sacred Harp singing convention most months, since last September, and there’s one on my calender every month between now and this September.
I’m going to “Youth Dance Weekend” in Vermont in September, which I’ve never been to, but I think it’ll be a a great deal of fun, and I’m very much looking forward to it. I’ve not been contra dancing as much, but that’s not a huge problem for me.
I’m moving to Philadelphia in the summer, which means a drastically longer commute, but an easier to orchestrate social life, and a better work/life balance. This means apartment hunting and all that jazz.

While this means less writing time and time for taking care of my own projects, it doesn’t mean that I don’t have any writing time. Sure writing takes time, but the largest challenge as a writer is in using the time I/we already have effectively, and getting the most out of those opportunities.

It also, I think, means finding a way to develop a writing (and blogging) habit that:

Doesn’t revolve around a fixed daily publication schedule. I still want to write essays, but I need to write essays when I have a compelling argument for an essay, rather than around the same core of ideas that I’ve been running around for the last year.
I need to be able to put the blog on the back burner while I focus on things like writing fiction, or hacking projects, or Cyborg Institute stuff. The blog is great, and I love writing the blog, but It’s far to easy for me to fall into a pattern where the blog becomes the project, rather than the journal in support of the project.
I need to organize my projects and tasks into clumps of work that are easier to manage in shorter periods of time. This is probably a reorganization problem that needs to mostly occur within my head.

So where does that leave us? I have a few posts piled up that I’ll parcel out over the next few weeks, though on the whole there will probably be less posting by me around here. I’m probably going to do more posts along the lines of “here’s what I’ve been up to, go read my work elsewhere.” There will be some guest posts and I’ve already begun working with some writers for that. Beyond that, I guess we’ll both be able to be surprised.

Knowing Mars, a Novella

Wed, 19 May 2010 00:00:00 +0000

I don’t know about this. But here it goes, anyway.

I’m pleased to announce the complete publication of my novella “`Knowing Mars <http://tychogaren.com/mars/>`_” on `tychogaren.com <http://tychogaren.com/>`_.

“Knowing Mars” is an important story for me. I wrote it after I graduated from college, after I didn’t go to graduate school the first time, and in a lot of ways it was the project that got me started down the path of being a “real writer,” post-graduation. I’d written fiction before college, and mostly avoided writing fiction in college, and then right as I was finishing college I started writing stories again. It was strange for a while, but it was delightful to be able to tell stories and be so much better at it than I was the first time.

I find that this is fundamentally a recurring issue. When I started writing “Knowing Mars,” I felt like I was starting out light years ahead of what I had written four or five years before. Now, I feel like the stuff I’m working on now is light years ahead of “Knowing Mars.” This is probably an encouraging sign.

I suppose you’d like to know a bit more about the story. I’ll leave most of the details to the reading, but basically it’s a sort of superhero/cyberpunk story that explores themes related to diaspora, political organization, historical narrative, and gender. I don’t know if I’ve ever described it as such before.

The story is available in multiple formats. Each chapter is available in full HTML as part of tychogaren.com and in a plan Markdown formatted plain text. Furthermore the complete text of the novella is available in both of these formats, and a simple un-styled HTML version that should be ideal for conversion to various electronic reading platforms. If you want to read the novella but would find another format easier to process, talk to me about it and I’ll get something pulled together for you.

All “full html” versions of the text have comments enabled using the same system as the blog. I look forward to your comments. Thanks for reading, and stay tuned for more fiction.

Onward and Upward!

In Favor of Unpopular Technologies

Tue, 18 May 2010 00:00:00 +0000

This post ties together a train of thought that I started in “The Worst Technologies Always Win” and “Who Wants to be a PHP Developer” with the ideas in the “Ease and the Stack” post. Basically, I’ve been thinking about why the unpopular technologies, or even unpopular modes of using technologies are so appealing and seem to (disproportionately) capture my attention and imagination.

I guess it would first be useful to outline a number of core values that seems to guide my taste in technologies:

Understandable

Though I’m not really a programmer, so in a lot of ways it’s not feasible to expect that I’d be able to expand or enhance the tools I use. At the same time, I feel like even for complex tasks, I prefer using tools that I can have a chance of understanding how they work. I’m not sure if this creates value in the practical sense, however, I tend to think that I’m able to make better use of technologies that I understand the fundamental underpinnings of how they work.

Openness and Standard

I think open and standardized technologies are more useful, in a way that flows from “understandable,” I find open source and standardized technology to be more useful. Not in the sense that open source technology is inherently more useful because source code is available (though sometimes that’s true), but more in the sense that software developed in the open tends to have a lot of the features and values that I find important. And of course, knowing that my data and work is stored in a format that isn’t locked into a specific vendor, allows me to relax a bit about the technology.

Simple

Simpler technologies are easier to understand and easier--for someone with my skill set--to customize and adopt. This is a good thing. Fundamentally most of what I do with a computer is pretty simple, so there’s not a lot of reason to use overly complicated tools.

Task Oriented

I’m a writer. I spend a lot of time on the computer, but nearly everything I do with the computer is related to writing. Taking notes, organizing tasks, reading articles, manipulating texts for publication, communicating with people about various things that I’m working on. The software I use supports this, and the most useful software in my experience focuses on helping me accomplish these tasks. This is opposed to programs that are feature or function oriented. I don’t need software that could do a bunch of things that I might need to do, I need tools that do exactly what I need. If they do other additional things, that’s nearly irrelevant.

The problem with this, is that although they seem like fine ideals and values for software development, they are, fundamentally unprofitable. Who makes money selling simple, easy to understand, software with limited niche-targeted feature sets? No one. The problem is that this kind of software and technology makes a lot of sense, and so we keep seeing technologies that have these values that seem like they could beat the odd and become dominant, and then they don’t. Either they drop task orientation for a wider feature set, or something with more money behind it comes along, or the engineers get board and build something that’s more complex, and the unpopular technologies shrivel up.

What to do about it?

Learn more about the technologies you use. Even, and epically if you’re not a programmer.
Develop simple tools and share them with your friends.
Work toward task oriented computing, and away from feature orientation.

The Worst Technologies Always Win

Thu, 13 May 2010 00:00:00 +0000

This post is the culmination of two things:

1. Who wants to be a PHP Developer?

2. An ongoing conversation I’ve had with a number of coworkers about the substandard technologies that always seem to triumph over the “better” options.

The examples of the success of inferior technologies are bountiful. MySQL’s prevalence despite some non-trivial technical flaws (around clustering, around licensing as highlighted by the Oracle merger); PHP as the de facto glue language of the web despite the fact that every other language in it’s class is probably a better programming language (e.g. Python, Perl); VHS and Beta Max; BlueRay (which are proprietary and a physically less durable media) and HD-DVD; and so forth.

The factors are (of course) multiple:

Marketing.

People have to know about technologies at some stage in their development if the technology is to take off. I’m not sure what that crucial point is, and frankly marketing is something that not only I don’t understand, but I don’t really think anyone understands. Having said that, I think it’s clear that technologies don’t compete simply on their technical merits, and this is in recognition of that.

Timing.

Technologies that appear at the right time, with regards to availability of alternatives and the needs/interests in the market on those technologies matter a great deal, and can sometimes tip the balance between competing technologies. Arguably MySQL beat PostgreSQL not because it was better, but because it existed in a firm way a little bit earlier. Linux “won” market share over BSD, because BSD wasn’t quite fully free/open source (or available) in 1990-1992 when Linux was taking off.

Momentum.

A project that doesn’t look like it has energy and a large team behind it is probably doomed to fail on some level, not because it’s a bad technology, but potential users of a technology need to feel confident that it’s going to stick around. If no one is excited about it, then it’ll never win, even if it’s superior in the final analysis.

There’s nothing, really, to be done. I think the truly superior technologies might benefit by paying attention to these factors when it matters, but then the developers of said technologies are probably less interested in marketing and proper timing than the competition. Which is perhaps as it should be.

I guess the lingering questions that I’ll leave you with relate to thinking about the ways that open source and free software relate to other technologies. It strikes me that while there’s a pretty good balance between open source and proprietary technologies in the examples I provided above, all of the open source technologies were commercialized very early on and very intensely in a way that none of their competitors really did. Are “wining technologies,” a mystification of the proprietary technology world? Can community-based open source and free software technologies innovate and “win” in relation to their competitors?

I look forward to sorting out the answers in the comments. Onward and Upward!

Who wants to be a PHP Developer?

Tue, 11 May 2010 00:00:00 +0000

So PHP is this programming language that’s widely used, and often reviled by systems administrators and people who fancy theme selves “real programmers.” And yet, I think, while the “real programmers” were busy being “real,” PHP got something very fundamental right that explains its success despite the disdain.

I should interject with some context. First, I think this is another in my ongoing series of posts regarding linguistic relativism and computer programming. Second, for those of you who don’t spend your days in this space PHP is a programming language designed specifically for use in the context of the web, and it has only comparatively recently emerged as a possibility for “general programming tasks,” in contrast to other languages in “the space” (ruby, python, perl, etc.) which started as general purpose languages that have become common for use for web programming. Also as a computer language, there’s nothing particularly innovative about PHP, which earns it no small amount of ire.

So here’s the thing. PHP is easy. It’s designed to be easy. The syntax is familiar to people who are know even a little Perl or other C-like languages. Although the language has had object oriented support for several years, most PHP applications aren’t written in an object oriented manner and in a number of contexts that makes things a bit easier to understand.

And here’s the thing that I seem to notice in the context of administration: compared to other languages and frameworks, PHP is dead simple to deploy. Sure, everything under big loads becomes complex, and sure PHP applications consume more server resources than perhaps they should, but basically you configure a web-server to process PHP code, and then you write your code, inside of your page, and the web server generates what you need it to, and it just works. You don’t have to screw around with writing boilerplate CGI stuff, you don’t have to screw around with cgi-bins/ and script aliases which were never intuitive, you don’t need special servers, it just works.

And I already know that, someone is going to tell me that there’s a Perl module that lets you use perl in the same way, or that Python and Ruby don’t make you write CGI boilerplate either (or that there’s a Perl module to write the CGI boilerplate). And I know these things, but I’m not sure that it matters anymore. PHP, as a language is written around the needs of web development, and there’s merit in that.

I’m not saying, “go forth and write your next application in PHP:” I don’t even know if dynamic web applications are worth writing anymore. I am saying that despite all of the dreck in the PHP space, there are some things that are incredibly worthwhile that the current generation of web developers may miss.

That is all. For now. Onward and Upward!

Ease and The Stack

Thu, 06 May 2010 00:00:00 +0000

As if I needed a new project, this post introduces a new project that’s floating around in my mind. I was having a conversation with a friend about how I use the computer, I realized that while I’ve talked about various elements of how I use computer’s (the short story: peculiarly), I’ve not really talked about the holistic experience. As I started to talk about the various components and how they connect and work together, I realized that with out an example it was about as clear as mud.

So, in light of this, I’ve decided to make a “tychoish stack,” which won’t be anything particularly novel, but a repackaging of the software--mostly configurations and little bits here and there--that I use on a daily basis. My stumpwm configuration. The highlights of my Emacs configuration, and a few install scripts to make it all work together. An SSH configuration file that will make your life much easier, a list of packages that you’ll want to install on common operating systems (Debian/Ubuntu and Arch Linux), and--because I am who I am--a fair piece of writing about best practices and how to these tools effectively.

I was talking with Chris the other day in one of our never ending conversations regarding is ever changing choices of desktop operating systems. “Windows just feels more polished to me right now,” he said after a stint with the latest Ubuntu or one of its derivatives.

To which I said, “of course it does,” they pay bunches of people lots of money to make sure that Windows is polished and it’s a high priority given the failure of Vista and the direction of the market. The reason I use Linux full time is not because I want a better more polished experience, I use Linux full time, because I want something very specific: a window manager that stays out of my way and doesn’t distract me with “chrome,” emacs buffers that run in the way that I expect them to, package management tools that allow my system to work and function day in and day out, the ability to customize all of these functions to suit the evolving needs of my work, and nothing else that I have to mess around with.

This is something that I can only get from a UNIX system, and the more I play with different systems, the more I’m inclined to think that the only way to get this is with Arch Linux. But that’s just me. I’ve played with a bunch of different operating systems (that’s an aspect of my day job) and I’ve spent time using OS X, and it just doesn’t work for me. I don’t need smooth, I don’t want polish, I just want something that lets me work.

Different computer systems make sense for different people. There’s no problem with that assertion, I think.

The problem of course, that my setup doesn’t scale. I can go from a bare arch installation to a working version of my system in a few hours by using rsync to copy over my home directory, updating a few git repositories, installing a list of packages, and creating a half dozen symbolic links, but building this from the bottom up would take a long time.

The goal of this project then, is to make that process easier. I’ve done a bunch of work to get a setup that does what I need it to do, I know which applications work, I know how to plug everything together to make it easier to manage. I want to build a stack so that you all can take it, learn from what I’ve done, and spend the time customizing it to what you do, rather than going through the trouble of building it up yourselves.

How’s that sound?

The Internet in Real Life

Tue, 04 May 2010 00:00:00 +0000

In many ways, I think you could say, I live and work in a bubble of the technical future that, as Gibson said “isn’t evenly distributed,” yet. I have developed a set of tools and work flows that enable me to work nearly anywhere and on a moment’s notice. I work for a company which great and open internal infrastructure that allows us to securely communicate and collaborate in whatever way we think will best serve the projects we’re working on. And I know enough to be able to automate the boring parts of my technological experience. In all, pretty good.

Here’s the thing though: despite all of this technological infrastructure, all this know how and frankly awesome connectivity: all of the tools we use to collaborate technologically: chat rooms, wikis, paste-bins, version control systems, instant messages and email all work better when you’re in the same room. Examples:

I was sitting in a talk with a coworker, and we were both logged into an IRC room from our laptops, where we were able to share some useful examples, links, and other commentary without being (very) disruptive. In day to day work, I (and my coworkers) spend a lot of time using chat rooms to communicate and share information with people who are only a few feet away, and in the end we get a lot done.

There are probably a lot of reasons why this is the case: digital relationships are almost always supported by real life relationships, there’s a level of hard to document interstitial and context setting that we do in real life that is difficult to efficiently create digitally, but that can be accomplished without second thought *in real life, and so forth. But, having made this realization, I think there are a few conclusions to be drawn about collaboration technology:

It helps to centralize information flow. So much collaboration technology is “pull” based, and there’s no good way to ensure that people know you’ve done something that they might consider without pushing information to them in some manner. Even so, create one place where the people you’re working with can see what you’re working on.

Use something like an IRC channel, or an xmpp MUC room, combined with a service like notifixious or something similar. In a lot of ways, the incessant emails Facebook sends achieve the same goal.
Communities come together to work on something specific and concrete, but inevitably they bond and endure for other reasons and other kinds of conversations. While creating “off topic” silos is awkward, creating the space for people to get to know each other is essential to making people work together well (and thus use collaborative technology better.)
Focus most of your attention on “getting things done,” and less attention on the “how things are done.” There are so many technological solutions, so many options, and so many different contexts that it doesn’t really matter how things get done as long as they do get done. The right and preferred tools will arise and present themselves when needed, and as long as things are getting done using the right tool doesn’t matter much.

Onward and Upward!

Strategies for Organizing Wiki Content

Fri, 30 Apr 2010 00:00:00 +0000

I’ve been trying to figure out wikis for a long time. It always strikes me that the wiki is probably the first truly unique (and successful) textual form of the Internet age. And there’s a lot to figure out. The technological innovation of the wiki is actually remarkably straightforward,¹ and while difficult the community building aspects of wikis are straightforward.² The piece of the wiki puzzle that I can’t nail down in a pithy sentence or two is how to organize information effectively on a wiki.

That’s not entirely true.

The issue, is I think that there are a number of different ways to organize content for a wiki, and no one organizational strategy seems to be absolutely perfect, and I’ve never been able to settle on a way of organizing wiki pages that I am truly happy with. The goals of a good wiki “information architecture” (if I may be so bold) are as follows:

Clarity: It should be immediately clear to the readers and writers of a wiki where a page should be located in the wiki. If there’s hierarchy, it needs to fit your subject area perfectly and require minimal effort to grok. Because you want people to focus on the content rather than the organization, and we don’t tend to focus on organizational systems when they’re clear.
Simplicity: Wikis have a great number of internal links and can (and are) indexed manually as needed, so as the proprietor of a wiki you probably need to do a lot less “infrastructural work” than you think you need to. Less is probably more in this situation.
Intuitive: Flowing from the above, wikis ought to strive to be intuitive in their organization. Pages should answer questions that people have, and then provide additional information out from there. One shouldn’t have to dig in a wiki for pages, if there are categories or some sort of hierarchy there pages there shouldn’t be overlap at the tips of various trees.

Strategies that flow from this are:

In general, write content on a very small number of pages, and expand outward as you have content for those pages (by chopping up existing pages as it makes sense and using this content to spur the creation of new pages.
Use one style of links/hierarchy (wikish and ciwiki fail at this.) You don’t want people to think: Should this be a camel case link? Should this be a regular one word link? Should this be a multiple word link with dash separated words or underscore separated words? One convention to rule them all.
Realize that separate hierarchies of content within a single wiki effectively create separate wikis and sites within a single wiki, and that depending on your software, it can be non-intuitive to link between different hierarchies.
As a result: use as little hierarchy and structure as possible. hierarchy creates possibilities where things can go wrong and where confusion can happen. At some point you’ll probably need infrastructure to help make the navigation among pages more intuitive, but that point is always later than you think it’s going to be.
Avoid reflexivity. This is probably generalizable to the entire Internet, but in general people aren’t very interested in how things work and the way you’re thinking about your content organization. They’re visiting your wiki to learn something or share some information, not to think through the meta crap with you. Focus on that.
Have content on all pages, and have relatively few pages which only serve to point visitors at other pages. Your main index page is probably well suited as a traffic intersection without additional content, but in most cases you probably only need a very small number of these pass through pages. In general, make it so your wikis have content everywhere.

… and other helpful suggestions which I have yet to figure out. Any suggestions from wiki maintainers?

There are a number of very simple and lightweight wiki engines, including some that run in only a few lines of Perl. Once we had the tools to build dynamic websites (CGI, circa 1993/1994), the wiki became a trivial implementation. ↩︎
The general Principal of building a successful community edited wiki is basically to pay attention to the community in the early stages. Your first few contributors are very important, and contributions have to be invited and nurtured, and communities don’t just happen. In the context of wikis, in addition to supporting the first few contributors, the founders also need to construct a substantive seed of content. ↩︎

A Git of One's Own

Tue, 27 Apr 2010 00:00:00 +0000

My most sincere apologies to Virginia Woolf for the title.

We use a lot of git at work, and I’ve earned a bit of a reputation as “the git guy,” both at work and amongst the folks who read the blog. So, I suppose it should come as no surprise that a coworker (hi stan!) said “You should write something about using git when it’s just one person.” And I said “well sure, but it’s not nearly as interesting as you think it’s going to be.” He didn’t seem to mind, so here I am.

Lets back up for a second.

Git is a tool that programmers use to facilitate collaboration. It stores versions of computer code (and associated file) and save incremental sets of changes to those files, so that programmers can easily experiment with changes without destroying code, and so that teams of programmers (sometimes even large teams) can all work and develop on a single code base without stepping on eachothers toes, or duplicating efforts because you end up working on different versions of the code.

Git makes a number of innovations that make version control with git much preferable (at least in my experience) to other tools, but fundamentally that’s what git does. Git has all sorts of innovations that make it awesome: it’s fast, it can take “diverged branches” and merge them together painlessly and almost automatically. It’s great and mind bending, and I think really forces us to rethink all sorts of assumptions about authorship, and the coherency of “texts” in general.

But I’m famous for using git all alone, with just me. Here are the lessons and conclusions that I’d draw from my experiences over the past… two or three (or so) years of using git:

Use fewer features. Git can do all sorts of funny stuff with branches and merges, but the truth is that when you’re working alone you don’t really want to have to much with branches. Because they’re the really novel feature (at least in terms of their usability) in git, everyone wants to use them but they add complexity, and there are other approaches to managing files and content with git that are probably preferable.
Resist the temptation to store binary files in git. It’ll work, but you won’t be really happy with it.
Even though you don’t need to have a remote repository to push your git repositories to, keep an off site repository in almost every case. You get incremental backups for free with git, and remote back ups are nearly free.
Use a tool like gitosis (but it’s probably in your distribution’s repository and you should use that version) to manage repositories. It’s overkill for your use-case, but it makes things easier in terms of creating repositories. Perhaps consider something like girocco if you want even more overkill, and more web-based interface.
There are git tools for most text editors and graphical tools that you may choose to use, but don’t, at least until you understand what’s going on behind the scenes. Learn git commands, and do stuff from the command line, as you’ll be much better (in the long term,) at fixing things as issues come up.
If you need to maintain multiple machines, think of each machine as a collaborator, and it’s probably easiest to have a centralized group of repositories that you can push to in order to keep these machines up to date.
If you’re using git to manage configuration files (which is great) I strongly recommend having a “sub-home” directory in a git repository with your configuration files with symbolic links pointing to the files in the repository. This strikes me as towing the balance between utility and control, without being a total pain the ass. As it were.

And that’s about it.

I want ZFS in the Kernel

Wed, 21 Apr 2010 00:00:00 +0000

The background:

Sun Microsystems developed this file system called “ZFS,” which is exceptionally awesome in it’s capabilities and possibilities. The problem, is that it was developed and released as part of the Open Solaris project which has a licensing incompatibility with the Linux Kernel. Both are open source, but there is a technical (and not all together uncommon) conflict in the terms of the license that makes it possible to combine code from both licenses in a single executable.

Basically the GPL, under which the Linux Kernel is distributed, says if you distribute a binary (executable) under the terms of the GPL, the source code is all files that you used to make that binary. By contrast ZFS’s license says “here are all the files that we used to make this binary, if you change them when you make your binary and give that binary to other people you have to give them, but if you add additional files, you don’t have to give those out to people.”

Apparently the idea behind the ZFS license (i.e. the CDDL, and the MPL from whence it originated,) is that it allows for easier embedding of ZFS (and other technologies) in proprietary code because the resulting binary isn’t list covered by the CDDL in most cases. Even though the CDDL is incredibly confusing, apparently it’s more “business friendly,” but I diverge from my original point.

And so if Linux users want to run ZFS, they have to run it as a user-space process (i.e. not in the kernel,) which is suboptimal, or they have to run Solaris in a vitalized environment (difficult,) or something. There’s also a ZFS-like file system called “btrfs,” which can be included in the kernel (interestingly, developed by Oracle who of course now own ZFS itself,) but it is not production ready.

What I’m about to propose is an end run around the GPL. Because it seems to me that combining the source code violates neither license, distributing source code violates no license. Compiling the source code for your own use violates no license. I mean it’s annoying and would require a bit of bootstrapping to get a Linux+zfs system up and running, but this is the kind of thing that Gentoo Linux users do all the time, and isn’t a huge technological barrier.

It feels a bit creepy of course. but I think it works. The logic has also been used before. We’ll call it the “PGP loophole.”

PGP is an encryption system that’s damn good so good in fact, that when it was first released, there were export restrictions on the source code because it qualified as military-grade munitions in America. Source code. How peculiar. In any case there were lawsuits, and PGP source was released outside of America by printing it in a book. Which could be disassembled and scanned into a computer and then compiled. Books were never and--as far as I know--are not classified as munitions, and so they could be exported. Of course I’m not a lawyer, but it strikes me that linux+zfs and PGP in the 90’s may be in analogous situations.

And I think, because this proposal centers around the distribution of source code and only source code this kind of distribution is fully within the spirit of free software. Sure it’s pretty easy, even for the “good guys,” to run a foul by distributing a binary, but this would be easy to spot, and there are already suitable enforcement mechanisms in place, for the Linux kernel generally, and Oracle’s legal department which we can assume will take care of itself.

Or Oracle could release ZFS under GPL. Either solution works for me.

Why Open Source Matters

Tue, 20 Apr 2010 00:00:00 +0000

A reader (hi grandma!) asked me to write a post about why I’m so interested in open source, and who am I to refuse. In fact, I tend to do requests pretty well, so if there’s a subject you’d like to see me cover here, just ask and I’ll see what I can do. In any case, I’ve been involved, for varying definitions of involved, in Free Software and open source for a few years now. On a personal level, I use this software (almost exclusively) because I can make it do exactly what I need it to do, because it’s very stable, and because from an architecture perspective I understand how these systems work and that’s useful for me. Having said that, I think open source is important and worth considering for reasons beyond the fact that I (and people like me) find it to be the most important tool for the work we do.

When folks get together and say “I’m going to work on an open source project,” I think some interesting things happen. First, they’re making a number of interesting economic decisions about their work. There are business models around open source, but they are more complex than “I make software, you give me money for software,” and thus require people to think a little bit more widely about the economic impact of their work. I think the way that people view the implications of their labor is incredibly important, and free software presents an interesting context to think about these questions.

The second, and perhaps larger reason I’m interested in open source is the community. Open source developers often know that the things they want to create are beyond the scope of their free time and personal ability, so they collaborate with other people to make something of value and worth. How this collaboration happens: what motivates developers, how they create tools and technologies to support this kind of work flow, how the “intellectual property” is negotiated (particularly in projects that don’t use the GNU GPL,) how leaders are selected and appointed, how decisions are made as a community, and how teams are organized and organize themselves. These are intensely fascinating.

And these phenomena matter, both in and for themselves, but also as they impact and connect with other questions and phenomena in the world. For instance:

I think that the decision making process in free software projects is instructive for thinking about how all sorts of communities can reach a “decision making fork” resolve it somehow and then continue with their work. Some open source projects have formal structures, and that is easier to understand from the outside, but most make decisions in an informal way, and that decision making process is potentially novel, I’d argue. In what other context do people have to construct projects outside of work.
While leaders in the open source community are rarely elected (aside from a number of notable examples; the Debian Project Leader springs instantly to mind) most projects are very democratic. But this requires that we keep in mind a fairly broad definition of democracy. Because there isn’t a lot of voting, and sometimes decisions aren’t discussed thoroughly before people start doing things, it doesn’t look democratic. But everything is volunteer based, and leaders I think have a sense of responsibilities to their constituencies, which is meaningful.
The tools that open source developers use are, unsurprisingly open source, and are often picked up and used by teams that aren’t making free software. I’m interested in thinking about how “ways of working,” proliferate out of open source and into other spheres. Is non-open source developed differently if the developers are familiar with and use open source tools?
Similarly, I think I’m interested in thinking about how the architecture of Linux and Unix give rise to a thought about APIs and open standards in a way that doesn’t necessarily happen on closed platforms. After a certain point, I think I’m forced to ask: is GNU/Linux the leading free software/open source platform because it just happens to be, or because it’s UNIX. Is there something special about the design of UNIX that leads to openness, and the practices of openness? To what extent does the limitations of the environment (operating system here) the social conventions that are built on it?

And then beyond the specific questions--which are terribly important in and of themselves--open source present a terribly exciting subject for the study of these issues. There is so much data on the ground concerning open source: version control history, email logs, IRC logs, and so forth. Not only are the issues important but the data is rich, and I think has a lot to tell us if we (I?) can bother to spend some time with it.

Ritual, Velocity, and Getting Things Done

Thu, 15 Apr 2010 00:00:00 +0000

I finished chapter eight of the novel that I’ve been working on for… Oh? Way. Too. Fucking. Long. And you want to know how I’m even comfortable asserting that I’m done with the initial draft of Chapter 8? I wrote an entire scene from Chapter 9 without saying “wow, I need another scene in Chapter 8 so that the story works out.” Because I’ve finished Chapter 8 at least three times, but this time I’m pretty sure. There are a couple of interesting, or at least quasi interesting factors that I think are worth some attention.

For starters, I made some progress on the novel. I know that I don’t have endless time to write fiction, and in addition to a day job that requires a bunch of my time and brain cycles, I write the blog, and work on other side projects: fiction, programming related things, the Cyborg Institute, dancing, singing, and occasionally sleeping. These are all projects that are important to me, and I think create value for me (and I hope in their own ways, you as well) so I don’t want to sound as if I’m complaining about being too busy and overextended (perhaps I am), but as a result I think I’m being pragmatic to accept a slower pace of development.

At the same time, damn I need to finish this thing. It’s good, I’m finally back in a place where I don’t hate the story, but at the same time I’m very aware that I’ve learned a lot.

And perhaps that’s the problem with taking so long to finish novels. Not so much that writing is a race, but if you aren’t able to pull it off in a reasonable period of time, say 12 months, or so by the time you get to the end, you know so much more about the way you work about how to put together stories, and how to write, that creating a cohesive work becomes an actual challenge. At least for me, the thing I probably want more than anything right now, is a chance to work on other fiction projects, to take the lessons that I’ve learned from writing this story and apply them to writing other projects. I have a great idea for a new story laying around in a text file, but I’m not touching it yet.

The two most important things about being a writer, as far as I’m concerned, are actually writing things (done!) and finishing things (at which I think I get a middling B). So just starting new projects at whim, isn’t exactly an option either. So in light of all this, what’s my strategy? Fairly simple…

I’ve set a recurring task in my org system to write 100 words a day on the novel. Just 100 words. And if I know it’s not going to happen I can mark the task as “skipped” or do it “late.” But the truth is that 100 words is the kind of thing I can do in only a few moments, so it’s not only a regular reminder to write, but also an eminently reasonable goal. Not only do 100 word segments add up (in a way that 0 word segments never do,) but the real trick is that in my mind I’m not trying to write very much, just enough to get started. If I don’t, at least I’ve made a little progress. If I do, then all the better.

In addition to the regular writing task for the fiction project, I’ve also started keeping a journal using this method. I’ve also created a recurring tasks for keeping the journal, and I find this method tends to have a positive effect on my productivity. To-do lists are great for remembering and prioritizing tasks when you have a lot of balls up in the air, but they often fail at tracking real life in a reasonable way. The journal provides a good way to keep track of, and recognize the importance of all the things that we spend time doing, but that don’t often have an opportunity to be captured into the to-do list before they get done. I think of it as a sort of inverse-to-do list.

It doesn’t always work, of course. There are days when I don’t get to either one of these tasks, and there are some days where I catch up on one or the other of them. But it’s a good practice, and I focus on the things that are important: actually producing something and then also building and maintaining a habit.

Because I don’t know how else things get done. Not that I’d be unwilling to listen if you have a better solution. See you in comments…

Objective Whatsis

Tue, 13 Apr 2010 00:00:00 +0000

Ok, confession time, I don’t really get object orientation. So in an effort to increase my understanding, I’m going to write some overview and discussion in an effort to understand things a bit better. Hopefully some of you will find this helpful. I’ve tinkered with programing for a long time. I read a huge chunk of a popular Python introductory text, and I’ve read a chunk of Practical Common Lisp, but people start talking about objects and I loose track of everything. I think there’s something slightly unconventional--at least initially--about object orientation.

We understand procedural programming pretty easily. There’s a set of steps that you need to perform, and you tell the computer what they are, and then data gets handed off to the program, it runs, and the steps are performed at the end. Or, conversely, someone calls your program and says “I want data” (perhaps implicitly) and the program says “ok, to go get data, I need to do these things,” and then it runs and at the end you see data.

Object orientation turns this sort of sideways and says “lets build a representation of our data (objects) then write code that says what happens to those objects.” Ok, that almost makes sense, data happens to your program and you write code to provide behaviors and responses to all of the things that will happen when your program runs. So you feed objects (data) into your program it does its thing in response to those objects and different data (probably) exists on the other end.

I hope I haven’t lied yet! To continue…

The thing that always confused me, given my utter lack of background is this whole “methods,” and “classes,” thing that programmer types launch into almost immediately. To overview:

Objects as I said above, are just another way to think about data. It’s a “thing,” that the program has to deal with. Classes really just represent the structure of a program. We hear “objects are instances of classes,” but this feels sort of backwards, it feels more intuitive to say that classes provide a framework for interacting with objects: they describe the loose “shape” of their objects and then create a place for behaviors to exist. Methods, then, are those behaviors. Often methods “belong” to classes (either literally in the structure of the code, or just conceptually) and they define what happens to objects as the program runs.

Thus the role of an object oriented runtime (or compiler? I think that’s the right word for the program that executes the program,) is to take data that comes in, figure out what class (or classes) the object “fits into,” and then apply the methods that belong to that class.

Whew! So, how’d I do?

Assuming my understanding is correct, allow to offer the following analysis:

By using multiple methods, in given sequence you can reuse code within a class, rather than needing to define and redefine a set of increasingly complex procedures.
At the same time there’s a much higher start up cost for object oriented code. Because we think about getting things done in programs we--or I--tend to think in terms of procedures, rather than objects, it takes a bit of extra brainpower to do the object oriented way. And for most tasks--which are pretty small--creating classes and creating methods seems like a lot of stuff to have to hold in your head when you’re figuring out what needs to done?

It sort of seems like, in order to do object orientation right, you have to already know what has to happen in the program. Otherwise, classes fail to properly describe the data/methods that you need.
Ok, so now that I, more or less, understand what’s going on here, might we be better off calling it class-oriented programming? Or “class-centered” programming?

Thoughts?

Skill and Mastery

Mon, 12 Apr 2010 00:00:00 +0000

A few years ago, at a Morris Dance weekend, I saw a woman sitting in one of the common rooms obviously struggling with a piece of knitting. I helped her figure out something, and then went back to singing or whatever it was. When I returned to my chair the experienced dancer sitting next to me assumed I was the student and said, “Getting a knitting lesson, eh?”

And I chuckled in between choruses and said “Not really, I’m an even better knitter than I am a Morris dancer.”

Impressed he said “Wow, and you’re a pretty good Morris Dancer.”

Which was an utterly delightful thing to hear, particularly from this chap. I’m an ok Morris dancer, and--particularly then--I tend to use youth as a compensation for skill. And I don’t knit as much now I as used to, and while I don’t know all there is to know about knitting, not by a long shot, I’m pretty good. I’ve been knitting for 8 years, or so and the thing that’s keeping me from knitting these days is time, and the fact that I live in a pretty warm climate at the moment. I never see something knitted and think “wow that’s too hard for me.” In a lot of ways, I think I’ve mastered knitting.

In some ways that’s kind of cool. It’s nifty to be able to think about something and say, “yes! I can do this.” And at the same time, I can’t help but have a little bit of regret for the fact that I spent so much time figuring out how to do something that I can’t really use most of the time: I don’t live in a climate that really calls for woolens most of the time, I don’t have any real interest in being a knitwear designer, and I consistently have trouble finding time to knit amongst all of the other things that I find myself committed to.

Speaking of which, I’ve certainly committed myself to other things. The dancing, I’ve spent a lot of time dancing and learning how to be a better dancer. And it’s sort of paid off. I’m not great, and I fake my way though far too many things, but I feel competent, and I’ve gotten some pretty good feedback. And that’s awesome. While it’s fun and socially fulfilling--and that’s good enough--at the same time I’m not sure if it gets me anywhere in particular. I don’t really want to play music, and I’m not a very good teacher of dance.

I am incredibly grateful that I started dancing when I was in high school. I don’t know if I’d have been able to pick things up as easily without those experiences. As with knitting in college, the early and intense experiences gave confidence and enough base-skill to make mastery a possibility. Had I started not when I did, I don’t know that I would have had the patience, confidence, or persistence as a(n albeit-young) adult to learn the things that I do automatically. I can’t really fathom it.

Which brings me to writing. And I don’t even have a clue what to say there. I’m pretty good--I mean, I definitely have the persistence nailed--but I have no delusions that I’m a great writer. I’ve been writing forever, and I’ve definitely been in situations where I’ve had to edit other people’s work and thought “wow, my prose may be flawed, but at least it isn’t that.” I wonder why writing is different then, at least for me. Am I a better dancer/knitter than I am a writer? Is it simply easier to critique writing than it is to critique a sweater or ones ability to dance socially? Is the fact that one some level the production of text has economic baggage in a way that a waltz so rarely does?

My own angst aside, I was talking with a longtime reader of the blog about employment and changing careers and about figuring what your skills and assets are. Because the transition from “I know how to knit sweaters and write things and dance,” into “I have a job writing about Linux-based systems administration,” isn’t the kind of thing that makes sense immediately.

But when you dig, I think it does: it turns out that writing knitting patterns in a the narrative/Elizabeth Zimmerman-inspired way that I do/did, is very much like process for writing about systems administration tasks. And I think dancing gives you the ability to be nimble and quick, not just physically (which may be of limited use more generally,) but also in social situations. Fixing a contra dance line that’s gone awry in the middle of the line has transferable skills. We hope, at least.

The challenge is in making those connections is difficult, and figuring out how to calculate the value that these skills might provide to the world. And isn’t this always the case?

Planned Obsolescence and Gadgets

Thu, 08 Apr 2010 00:00:00 +0000

I’ve had a tab open for most of last week for a 128 gig solid state hard disk that would fit my laptop, and that had a list price that I was comfortable with, and a maker that I tend to consider reputable. I haven’t ordered if for a number of reasons (not ready to swap drives, I’m using the wrong kind of file systems, given that my current drive works fine it’s a bit more than I want to spend at the moment.) But the fact that these things are “affordable,” and cost half what I thought they did, I’ve been thinking about my current stable of gear. This is, as those of you who think about technology in a way similar to me, not surprising. Here’s what’s on my mind in terms of new gadgets:

Netbooks, the small laptops are awfully sexy and I kind of want one. This isn’t logical: my real laptop is both small (12 inches,) has a full sized keyboard, decent resolution, and is totally functional: I use it for everything from the writing and production of this blog, to most of my day job, to all of my entertainment computing. I take my laptop everywhere, so a notebook doesn’t make a lot of sense. Also the relative (and ongoing!) cost of maintain a full “tycho-stack” on more than one machine is difficult and not something I want to get into. This doesn’t mean that I don’t find all the netbooks extremely cute and desirable but I don’t really need one. I suppose the interest in the SSD is part of an effort to make the existing laptop into more of a netbook.

Also on the stack I’m approaching the time in my cell phone contract when it’s time to upgrade again. I’ve been debating the Nokia N900 for some time. Though I’ve not played around with the n900 I think all of my “I wish my phone could do X,” problems could be solved by using a phone that actually uses Debian. Except I’ve learned in the last week that the screen on the n900 is resistive, like all of the old Palm Pilots (say,) and not capacitive like the iPhone/iPod Touch/Android phones. Big bummer. Not only is the phone not subsidized, and expensive, but it uses outmoded technology for the most important component. Fail.

At the same time it looks like Android phones are finally starting to make it to ATT. I’m not a good candidate for the iPhone (not being a reliable Mac User) and while I’m not sure about the enduring possibilities of the Android Platform, it seems like the best option at the moment.

I like my blackberry, really, but the replacement for the blackberry I have at the moment? The same phone. Really. Well, I think it has a different model number and a slightly different trackball, but otherwise, it’s the same. I mean… Throw me something here? Upgrades are supposed to be upgrades. I wonder if Blackberry smart-phones are today’s Nokia S60.

In any case, dithering about cell phones, netbooks, and SSDs aside, nothing else in my technology stable needs changing really. I geeked out a few weeks ago, and reorganized some of my mail config (but not much of it!), and I think if I had free time, I might redeploy the server to use nginx and prosody which are a bit more lightweight that my current stack, but that’s all minor and will totally wait. It’s a good place to be.

Content Management Beyond Wordpress

Wed, 07 Apr 2010 00:00:00 +0000

As a follow up to my, surprisingly popular post on the Limitations of Wordpress, and also I suppose my post on the current status of tumblelogs, I wanted to ruminate on “where Wordpress” is as a piece of software (and a platform,) and what the whole content management space looks like today.

In a lot of ways Wordpress won. Wordpress does what it does very well. And the thing it does, powering blogs, is in point of fact what most people need. The wordpress plug-in and theme ecosystems are vibrant and powerful and add a great deal of value to the system. Concerns about performance are largely solved by Wordpress Super Cache, and even though I’m squidgy about MySQL and PHP as a platform, for the job at hand it works.

The limitations that I spoke to a year ago are--I think--largely still relevant: exceedingly few (and fewer) innovative websites and blogs will be started that use Wordpress. This is, I think mostly because the prescribed form (“blog” as seen by Wordpress,) has become very cemented in our minds, and it becomes harder and harder to break from that form. In a lot of ways the largest limitation of Wordpress is not the software itself but the habits we have developed as users of Wordpress.

Indeed this is the general problem behind most content management systems: all sites that use X-platform tend to look very much like all other sites that use X-platform. Content management systems that purport to be web development frameworks (Rails/Django) are a bit better in this regard, but the problem remains. But this post is about the future, not about the histories or even the stale-present.

Some predictions and forward looking trends are thus in order:

Content management systems will increasingly manage work-flow rather than content presentation. Every site needs to be built and constructed, and in a lot of ways building a site and creating content are fixed costs no matter what system you use. The work flow that you use to maintain content is highly variable and can be pragmatically managed in more effective ways. Stay tuned for this.
The “built-in” feature set, or default configuration of a content management system will become less important than the possibilities of the platform. While I don’t think frameworks and CMS’s will merge in the next few years, they’ll get closer.
Smart static generation is still the future. Most things don’t need fully dynamic content, and the intense caching that we have to do to offset the dynamic overhead of contemporary systems isn’t the real solution to this problem.
Content management systems are, at the moment, at the center of a web site stack/deployment, and thus are huge all encompassing programs. I think increasingly content management systems can be designed to be much smaller applications, and only manage content and content work-flows rather than entire websites. We might call this the API-ization or the Unix-ification of web development.

Anything else? Am I totally off my rocker? Onward and Upward!

Installing Mcabber .10-rc3 on Debian Lenny

Mon, 05 Apr 2010 00:00:00 +0000

mcabber is console based XMPP or Jabber client. It runs happily within a screen session, its lightweight, and it does all of the basic things that you want from an IM client without being annoying and distracting. For the first time since I started using this software a year or two ago, there’s a major release that has some pretty exciting features. So I wanted to install it. Except, there aren’t packages for it for Debian Lenny, and I have a standing policy that everything needs to be installed using package management tools so that things don’t break down the line.

These instructions are written for Debian 5.0 (Lenny) systems. Your millage may vary for other systems, or other versions of Ubuntu. Begin by installing some dependencies:

apt-get install libncurses5-dev libncursesw5 libncursesw5-dev pkg-config libglib2.0-dev libloudmouth1-dev

The following optional dependencies provide additional features, and may already be installed on your system:

apt-get install libenchant-dev libaspell-dev libgpgme-dev libotr-dev

When the dependencies are installed, issue the following commands to download the latest release into the /opt/ directory, unarchive the tarball, and run the configure script to install mcabber into the /opt/mcabber/ folder so that it is easy to remove later if something stops working.

cd /opt/
wget http://mcabber.com/files/mcabber-0.10.0-rc3.tar.gz
tar -zxvf mcabber-0.10.0-rc3.tar.gz
./configure --prefix=/opt/mcabber

When that process finishes, run the following:

make
make install

Now copy the following /opt/mcabber-0.10-rc3/mcabberrc.example file into your home directory. If you don’t already have mcabber configured, you can use the following command to copy the file to your home directory.

cp /opt/mcabber-0.10-rc3/mcabberrc.example ~/.mcabberrc

If you do have an existing mcaber setup, then use the following command to copy the example configuration file to a non-overlapping folder in your home directory

cp /opt/mcabber-0.10-rc3/mcabberrc.example ~/mcabber-config

Edit the ~/.mcabberrc or ~/mcabber-config as described in the config file. Then start mcabber with the following command, if your config file is located at ~/.mcabberrc:

/opt/mcabber/bin/mcabber

If you have your mcabber config located at ~/mcabber-config start mcabber with the following command:

/opt/mcabber/bin/mcabber -f ~/mcabber-config

And you’re ready to go. Important things to note:

If something gets, as we say in the biz “fuxed,” simply “rm rf /opt/mcabber/” and reinstall.
Check mcabber for new releases and release candidates. These instructions should work well once there’s a final release, at least for Debian Lenny. The release files are located here.
Make sure to stay up to date with new releases to avoid bugs and potential security issues. If you come across bugs, report them to the developers there is also a MUC for the mcabber community here: xmpp:mcabber@conf.lilotux.net.
If you have an additional dependency that I missed in this installation do be in touch and I’ll get it added here.
Debian Lenny ships with version 0.9.7 of mcabber. If you don’t want to play with the new features and the magic in 0.10, then go for it. If you just want a regular client, install the stable mcabber with the “apt-get install mcabber” command and ignore the rest of this email.

Overheard

Fri, 02 Apr 2010 00:00:00 +0000

One of my favorite meme’s on twitter is the “OH:” meme, where folks post little snippets of things they’ve heard in the world that are (usually) hilarious. This post will be, I think, a collection of the best little quotes I’ve heard, heard about, or seen recently.

“Chicken is easily divisible”

“If you’re hand is one space off on your keyboard and you start typing server, you start typing awesome. servers are awesome.”

“I will be eagerly awaiting the New York Times style piece on the growing trend of the ‘ZOMG WE’RE NOT EVEN DATING GUYS’ rings.”

“The emacs makes the text, I am but a humble servant.”

“We should get facebook married so everyone would know its the fakes.”

“If [company] were a musical, there’d be a song here. Thankfully it’s not.”

“Caffeine is like liquid naps.”

“For epic lulz you should switch your keyboard [with blank keys] to Dvorak.”

“wat.”

Enterprise Linux Community

Wed, 31 Mar 2010 00:00:00 +0000

Ok. I can’t be the only one.¹

I look at open source projects like OpenSolaris, Alfresco, Resin, Magento, OpenSuSE, Fedora, and MySQL, among others, and I wonder “What’s the community around these projects that people are always talking about.” Sure I can download the source code under licenses that I’m comfortable with, sure they talk about a community, but what does that mean?

What, as a company, does it mean to say that the software you develop (and likely own all the rights to,) is “open source,” and “supported by a community?”

If I were sensible, I’d probably stop writing this post here. From the perspective of the users of and participants in open source software, this is the core question, both because it dictates what we can expect from free software and open source and more importantly because it has been historically ill defined.

There are two additional, but related, questions that lurk around this question, at least in my mind:

1. Why are new open source projects only seen as legitimate if the developers are able to build a business around the project?

2. What does it mean to be a contributor to open source in this world, and what do contributors in “the community,” get from contributing to commercial projects?

There are of course exceptions to this rule: the Debian Project, the Linux Kernel itself, GNU packages, and most open source programming languages among others. I’d love to know if I’ve missed a class of software in this list--and there’s one exception that I’ll touch on in a moment--but the commonality here is that that these projects are so low level that it seems too hard to build businesses around directly.

When “less technical” free software projects began to take off, I think a lot of people said “I don’t know if this open source thing will work when the users of the software aren’t hackers,” because after all what does open source code do for non-hackers? While it’s true that there are fringe benefits that go beyond the simple “free as in beer” quality of open source for non-hacker users, these benefits are not always obvious. In a lot of ways the commercialization around open source software helps add a buffer between upstreams and end users. This is why I included Debian in the list above. Debian is very much a usable operating system, but in practice it’s often an upstream of other distributions. Ubuntu, Maemo, etc.

The exception that I mentioned is, to my mind, projects like Drupal and web development frameworks like Ruby on Rails and Django. These communities aren’t sponsored or driven by venture capital funded companies. Though the leader of the Drupal community has taken VC money for a Drupal-related start up. I think the difference here is that the economic activity around these projects is consulting based: people use Drupal/Django/Rails to build websites (which aren’t, largely open source) for clients. In a lot of ways these are much closer to the “traditional free software business model,” as envisioned in the eighties and nineties, than what seems to prevail at the moment.

So to summarize the questions:

What, as a company, does it mean to say that the software you develop (and likely own all the rights to,) is “open source,” and “supported by a community?”
What does it mean to participate in and contribute to a community around a commercial product that you don’t have any real stake in?
How does the free software community, which is largely technical and hacker centered, transcend to deal with and serve end users?
How do we legitimize projects that aren’t funded with venture capital money?

Onward and Upward!

I think and hope this is the post I meant to write when I started writing this post on the work of open source ↩︎

Analyzing the Work of Open Source

Tue, 30 Mar 2010 00:00:00 +0000

This post covers the role and purpose (and utility!) of analysts and spectators in the software development world. Particularly in the open source subset of that. My inspirations and for this post come from:

In the video Coté says (basically,) open source projects need to be able to justify the “business case” for their project, to explain what’s the innovation that this project seeks to provide the world. This is undoubtedly a good thing, and I think we should probably all be able to explore and clearly explain and even justify the projects we care about and work on in terms of their external worth.

Project leaders and developers should be able to explain and justify the greater utility of their software clearly. Without question. At the same time, problems arise when all we focus on is the worth. People become oblivious to how things work, and become unable to successfully participate in informed decisions about the technology that they use. Users, without an understanding of how a piece of technology functions are less able to take full advantage of that technology.

As an aside: One of the things that took me forever to get used to about working with developers is the terms that they describe their future projects. They use the imperative case with much more ease than I would ever consider: “the product will have this feature” and “It will be architected in such a way.” From the outside this kind of talk seems to be unrealistic and grandiose, but I’ve learned that programmers tend to see their projects evolving in real time, and so this kind of language is really more representative of their current state of mind than their intentions or lack of communications skills.

Returning for a moment to the importance of being able to communicate the business case of the projects and technology that we create. As we force the developers of technology to focus on the business cases for the technology they develop we also make it so that the only people who are capable of understanding how software works, or how software is created, are the people who develop software. And while I’m all in favor of specialization, I do think that the returns diminish quickly.

And beyond the fact that this leads to technology that simply isn’t as good or as useful, in the long run, it also strongly limits the ability of observers and outsiders (“analysts”) to be able to provide a service for the developers of the technology beyond simply communicating their business case to outside world. It restricts all understanding of technology to journalism rather than the sort of “rich and chewy” (anthropological?) understanding that might be possible if we worked to understand the technology itself.

I clearly need to work a bit more to develop this idea, but I think it connects with a couple of previous arguments that I’ve put forth in these pages one regarding Whorfism in Programming, and also in constructing rich arguments.

I look forward to your input as I develop this project. Onward and Upward!

The Successful Failure of OpenID

Mon, 29 Mar 2010 00:00:00 +0000

Just about the time I was ready to call OpenID a total failure, something clicked and, if you asked how I thought “OpenID was doing,” I’d have to say that it’s largely a success. But it certianly took long enough to get here.

Lets back up and give some context.

OpenID is a system for distributing and delegating authentication for web services to third party sites. Basically to the end user, rather than signing into a website with your username and password, you sign in with your profile URL on some secondary site that you actually log into. The site you’re trying to log in, asks the secondary site “is this legit,” the secondary site prompts you (usually just the first time, though each OpenID provider may function differently here.) then you’re good to go.

Additionally, and this is the part that I really like about Open ID is that you can delegate the OpenID of a given page to a secondary host. So on tychoish.com you’ll find the following tags in the header of the document:

<link rel="openid.server" href="http://www.livejournal.com/openid/server.bml" />
<link rel="openid.delegate" href="http://tychoish.livejournal.com/" />

So I tell a third party site “I wanna sign in with http://tychoish.com/ as my OpenID,” it goes and sees that I’ve delegated tychoish.com’s OpenID to LiveJournal (incidentally the initiators of OpenID if memory serves,) and LiveJournal handles the authentication and validation for me. If at some point I decide that LiveJournal isn’t doing what I need it to, I can change these tags to a new provider, and all the third party sites go talk to the new provider as if nothing happened. And it’s secure because I control tychoish.com and contain a provider-independent identity server, while still making use of these third party servers. Win.

The thing is that OpenID never really caught on. Though managing a single set of authentication credentials, and a common identity across a number of sites has a lot of benefits to the users, it never really caught on. Or I should say, it took a very long time to be taken seriously. There are a number of reasons for this, in my understanding:

1. Third party vendors wanted to keep big user databases with email addresses. OpenID means, depending on implementation that you can bypass the traditional sign up method. This isn’t a technological requirement but can be confusing in some instances. By giving up the “traditional” value associated with sponsoring account creation, OpenID seemed like a threat to traditional web businesses. There were ways around this, but it’s confusing and as is often the case a dated business model trumped an inspiring business model.

2. There was and is some fud around security. People thought if they weren’t responsible for the authentication process that they wouldn’t be able to ensure that only the people who were supposed to were able to get into a given account. Particularly since the only identifying information associated with an account was a publicly accessible URL. Nevertheless it works, and I think people used these details to make people feel like the system isn’t/wasn’t secure.

3. There are some legitimate technological concerns that need to be sorted out. Particularly around account creation. This is the main confusion cited above. If someone signs up for an account with an OpenID, do they get a username and have to enter that, or do we just use the OpenID URL? Is there an email address or password associated with the account? What if they get locked out and need to get into the account but there’s no email? What if they need to change their OpenID provider/location at some point. These are legitimate concerns, but they’re solvable problems.

4. Some users have had a hard time groking it. Because it breaks with the conventional usage model, and it makes signing into sites simple it’s a bit hard to grok.

What’s fascinating about this is that eventually it did succeed. More even than joy at the fact that I get to use OpenID, finally, I think OpenID presents an interesting lesson in the eventual success of emergent technological phenomena. Google accounts, flickr accounts, and AIM accounts all provide OpenID. And although “facebook connect” is not using OpenID technology, it’s conceptually the same. Sites like StackOverflow have OpenID only authentication, and it’s becoming more popular.

OpenID succeeded not because the campaign to teach everyone that federated identity vis a vis OpenID was the future and the way we should interact with web services, but rather because the developers of web applications learned that this was the easier and more effective way to do things. And, I suspect in as much as 80% or 90% of cases when people use OpenID they don’t have a clue that that’s the technology they’re using. And that’s probably an ok thing.

The question that lingers in my mind as I end this post is: is this parallel any other optimistic technology that we’re interested in right now? Might some other “Open*” technology take away a strategic lesson from the tactical success of OpenID? I’d love to see that.

Onward and Upward!

Common Lisp, Using ASDF Install With SBCL

Thu, 25 Mar 2010 00:00:00 +0000

So I, like any self respecting geek trying to learn Common Lisp started to read the cliki, which is a wiki that supports Common Lisp projects. Nifty right? Right. It’s full of stuff, and between it and Common-Lisp.net, you can be pretty sure that if it exists in the common Lisp world it’ll appear on one of those two sites. And for every cool lisp thing, rather than usable instructions for installing the software it would just say “use asdf install and have fun.” Which is good if you know what asdf is or what it’s supposed to, and how to use it.

But, there’s a decent chance that you’re like me, and were completely clueless.

Turns out asdf-install is the common lisp equivalent of the CPAN shell or Ruby gems, or the Debian project’s dpkg, with some lisp-centric variations. This post provides an overview and a “quick start guide” in case you want to get started. The directions I provide are in line with “the way I like to keep my file system organized (e.g. ~/) and center around the Arch Linux and SBCL system that I use. However, this should hold true (more or less) for any distribution of Linux with SBCL and possibly to other lisps. Feel free to add your own modifications in comments or in the lisp page on wikish.

Begin by getting to a CL REPL. If you have emacs and “slime” installed get to a REPL using “M-x slime” otherwise just type sbcl at a system prompt. Installing slime, emacs, and sbcl are beyond the scope of this post, but in general use the packages designed for your platform and you should be good. MacPorts for OS X users and the package managers for most prevalent Linux-based operating systems should have what you need.

At the REPL do the following:

(require 'asdf)
(require 'asdf-install)

(asdf-install:install '[package-name])

Remember to replace the [package-name] with the dependency or package that you want to install. asdf will ask you if you want to install the package system wide, or in a user-specific user directory. I tend to install things in the user-specific directories because it gives me a bit more control over things. The user specific directory is located in ~/.sbcl if you want to poke around the files yourself. Done. That’s pretty straight forward. Lets get to the awesome parts.

Make a ~/lisp directory. I keep mine stored in a git repository. I’ve also kept my .sbcl directory inside ~/lisp and then created a symbolic link so that the computer is none the wiser. Issue the following commands to accomplish this:

cd ~/
mkdir -p ~/lisp/systems
mv .sbcl ~/lisp/
ln -s ~/lisp/.sbcl

Adjust the path as necessary. Additionally You will also want to create a ~/.sbclrc file with some code for asdf to initialize itself when SBCL runs. Do the following:

cd ~/
touch ~/lisp/.sbclrc
ln -s ~/lisp/.sbclrc

In your .sbclrc file you’ll probably want something like the following:

(require 'asdf)

(pushnew #p"/usr/share/common-lisp/systems/" asdf:*central-registry* :test #'equal)
(push #p"/usr/share/common-lisp/systems/" asdf:*central-registry*)

(pushnew #P"/home/[username]/lisp/systems/" asdf:*central-registry* :test #'equal)
(push #P"/home/[username]/lisp/systems/" asdf:*central-registry*)

This tells SBCL and asdf where all of the required lisp code is located. Alter path’s as needed. We’ve not talked very much about the ~/lisp/ directory yet. Basically it’s a directory that serves as a playground for all things lisp related. Each “project” or package should have it’s own directory, which will contain lisp code and an .asd file. To make a package accessible via asdf on your system create a symbolic link for these .asd files in your ~/lisp/system folder. Done.

So let’s set up a basic “hello world” package that we’ll call “reject,” just for grins. File one, ~/lisp/reject/reject.asd:

(defsystem "reject"
  :description "a reject program
  :version "0.1"
  :author "tycho garen"
  :licence "ISC License"
  :depends-on ("cl-ppcre")
  :components ((:file "reject")
               (:file "package")))

The dependency on cl-ppcre isn’t required, but that’s how it would work if you needed a regex engine for a reject hello world application. File two, ~/lisp/reject/package.lisp:

(defpackage reject (:use :common-lisp))

File three, ~/lisp/reject/reject.lisp:

(in-package :reject)

(defun hello-world () ()
(print "Hello World, tycho"))

(hello-world)

Once those fils are saved, issue the following commands to create the needed symbolic link:

cd ~/lisp/system/
ln -s ~/lisp/reject/reject.asd

Now, from the REPL issue the following expression to load the package:

(asdf:operate 'asdf:load-op 'reject)

And then the following expression to test that it works:

(hello-world)

And you’re set to go. As to how you’d write or package up something that might actually have value? That’s a problem I’m still wrapping my head around. But that can all happen later.

If I’ve overlooked something or you think my understanding of something here isn’t incredibly clear, do be in touch. I hope this helps!

Why Cloud Culture Isn't

Wed, 24 Mar 2010 00:00:00 +0000

I read this post by Bruce Sterling about “Cloud Culture”, and haven’t quite been able to formulate an appropriate response. It’s true that I often find myself ranting about this or that thing Bruce Sterling has written, so I’m perhaps not the most impartial

To summarize, his notion is that “the cloud,” as evidenced by a wealth of usable digital information and content, combined with an uptick in digital participation will lead to some sort of minimalistic cultural singularity that he’s calling “Cloud Culture.”

Right.

So lets attack this full on. The “Cloud” is marketing dribble. There’s nothing there, it’s just a different way of organizing the servers that provide content and services on the Internet. In a lot of way the Amazon and Rackspace “Clouds” have brute forced the server and hosting business with utterly conventional hardware, off the shelf open source software, and a bunch of [insert-favorite-scripting-language-here] scripts. Oh, and a lot of capital.

That’s the back-end, and moving upwards in the stack, we’ll find that the tools (e.g. Apache, Relational Databases, Perl) that we’re using now are for the most part the same this time around as they were ten or fifteen years ago. Sure we have new practices, and there are new frameworks, new versions of some software, but the changes are mostly evolutionary. Until you get to what the users see, and now we have applications that live in web browsers, and a more robust set of interconnected network services.

Otherwise, the cloud is just the Internet bundled up in a differed way. And it’s true that the bundling matters, but at the end of the day just because the “cloud” presents a better way to do the work of the Internet (in some situations,) it doesn’t mean that any of the fundamental principals have changed.

Another phenomena that hasn’t changed with the cloud: we’ve not quite gotten the economic system for the Internet figured out in a way that’s reliable and sustainable. New things on the Internet are supported by capital from “old-world” industries by way of venture capital and spin off subsidiaries. The leading business models are advertising (very print media) and premium subscription services (which are often contrived). So eventually it all collapses, and any gains in cultural participation run up against the hard costs of keeping power flowing to the servers in some way.

A few thoughts:

We could just as easily call this whole “cloud” thing “the matrix,” or “cyberspace,” in Gibson’s original formulation. Cloud is a horrible metaphor for most things, but particularly this.
The apparent cohesiveness of the Internet in 2010 is almost entirely due to the success of Google. As much as we might try and avoid Google, I firmly believe if Google were to go away without warning, while the Internet would still work most of the economy around it would crash and burn: people wouldn’t discover new things, the on-line advertising industry would largely collapse, a lot of the software that a lot of people inside and outside of google use would be without maintainers.
I’m not sure cultural participation is problem space that can be brute forced, and certainly the development of additional applications in the cloud is unlikely to increase participation in a direct manner. I’m not being an elitist and saying “cultural participation by the unschooled and unwashed masses isn’t worthwhile,” but rather cultural participation is about developing direct relationships and making/building things, and that requires forming relationships that don’t scale well.

Thoughts?

File System Metaphors

Thu, 18 Mar 2010 00:00:00 +0000

The file system is dead. Long live the File system.

We live in an interesting time. There are two technologies that aim to accomplish two very goals. On the one hand we have things like Amazon’s S3, Hadoop, NoSQL, and a host of technologies that destroy the file system metaphor as we know it today. The future, if you believe it, lays in storing all data in some sort of distributed key/value store-based system. And then, on the other hand we have things like “FUSE” that attempt to translate various kinds of interfaces and data systems onto the file system metaphor.

Ok, so the truth is that the opposition between the “lets replace file systems” with non-file based data stores folks and the “lets use the file system as a metaphor for everything,” is totally contrived. How data is stored and how we interact with data are very different (and not always connected) problems.

Let’s lay down some principals:

There are (probably) more tools to interact with, organize, manage, and manipulate files and file system objects than there are for any other data storage system in contemporary technology.
Most users of computers have some understanding of file systems and how they work, though clearly there are a great diversity of degrees here.
In nearly every case, only one system can have access to a given file system at a time. In these days of such massive parallel computing, the size of computer networks, (and the associated latency) this has become a rather substantial limitation.
From the average end user’s perspective, it’s probably the case that file systems provide too much flexibility, and can easily become disorganized.
There are all sorts of possible problems regarding consistency, backups, and data corruption that all data storage systems must address, but that present larger problems as file systems need to scale to handle bigger sets of data, more users, and attach to systems that are more geographically disparate.

Given these presumptions, my personal biases and outlook, and a bit of extrapolation here’s a basic feature set for “information storage system.” These features will transcend the storage engine/interface boundary a bit. You’ve been warned.
Multiple people and systems need to be able to access and edit the same objects concurrently.
Existing tools need to be able to work in some capacity. Perhaps using FUSE-like systems. File managers, mv, ls, and cp should just work, etc.
There ought to be some sort of off-network capability so that a user can loose a network connection without loosing access to his or her data.
Search indexing and capabilities should be baked into the lowest levels of the system so that people can easily find information.
There ought to be some sort of user facing meta-data system which can affect not just sort order, but also attach to actions, to create notifications, or manipulate the data for easier use.

These sorts sorts of features are of course not new ideas. My sygn project is one example, as is haven, as is this personal information management proposal.

Now all we need to do is figure some way to build it.

Why Bother With Lisp?

Tue, 16 Mar 2010 00:00:00 +0000

I’m quite fond of saying “I’m not a programmer or software developer,” on this blog, and while I don’t think that there’s a great chance that I’ll be employed as a developer, it’s becoming more apparent that the real difference between me and a “real developer” is vanishingly small. Stealth Developer, or something. In any case, my ongoing tooling around with common lisp and more recently the tumble manager project have given me opportunities to think about lisp and to think about why I enjoy it.

This post started when a friend asked me “so should I learn common lisp.” And my first response was something to the effect of “no, are you crazy?” or, alternately “well if you really want to.” And then I came to my senses and offered a more reasonable answer that I think some of you might find useful.

Let us start by asking “Should You Study Common Lisp?”

Yes! There are a number of great reasons to use Common Lisp:

There are a number of good open source implementations of the common lisp language including a couple of very interesting and viable options. They’re also stable: SBCL which is among the more recent entrants to this field is more than a decade old.
There are sophisticated development tools, notably SLIME (for emacs) which connects and integrates emacs with the lisp process, as well as advanced REPLs (i.e. interactive mode). So getting started isn’t difficult.
Common Lisp supports many different approaches to programming. Indeed, contemporary “advanced” languages like Ruby and Python borrow a lot from Lisp. So it’s not an “archaic” language by any means. Dynamic typing, garbage collection, macros, and so forth.
CL is capable of very high performance, so the chance of saying “damn, I wish I wrote this in a faster language,” down the road isn’t terribly likely. Most implementations run on most platforms of any consequence, which is nice.
You’re probably tired of hearing that “Learning Lisp will make your a better programmer in any language,” but it’s probably true on some level.

The reasons to not learn Lisp or to avoid using it are also multiple:

“Compiled” Lisp binaries are large compared to similarly functional programs in other languages. While most CL implementations will compile native binaries, they also have to compile in most of themselves.
Lisp is totally a small niche language, and we’d be dumb to assume that it’s ever going to take off. It’s “real” by most measurements, but it’s never really going to be popular or widely deployed in the way that other contemporary languages are.
Other programmers will think you’re weird.

Having said that all of I think we should still start projects in CL, and expand the amount of software that’s written in the language. Here’s why my next programing project is going to be written in lisp:

I enjoy it. I suspect this project like many projects you may be considering is something of an undertaking. Given that I don’t want to have to work in an environment that I don’t enjoy, simply because it’s popular or ubiquitous.
Although Lisp isn’t very popular, it’s popular enough that all of the things that you might want to do in your project have library support. So it’s not exactly a wasteland.
The Common Lisp community is small, but it’s dedicated and fairly close knit. Which means you may be able to get some exposure for your application in the CL community, simply because your app is written in CL. This is a question of scale, but it’s easier to stand out in a smaller niche.

Of course there are some advantages to “sticking with the crowd” and choosing a different platform to develop your application in:

If you want other people to contribute to your project, it’s probably best to pick a language that the people who might be contributing to your application already know.
While there are libraries for most common things that you might want to do with Common Lisp, there might not be libraries for very new or very esoteric tasks or interfaces. Which isn’t always a problem, but can be depending on your domain.
The binary size problem will be an issue if you plan to deploy in limited conditions (we’re talking like a 15 meg base size for SBCL, which is a non issue in most cases, but might become an issue.)
If you run into a problem, you might have a hard time finding an answer. This is often not the case, but it’s a risk.

Onward and Upward!

Where is Java Today?

Mon, 15 Mar 2010 00:00:00 +0000

A few weeks ago a coworker walked into my office to talk about the architecture of a project, complete with diagrams, numbers I didn’t grasp (nor really need to,) and the examples of potential off the shelf components that would make up the stack of the application at hand. I asked scores of questions and I think it was a productive encounter. Normal day, really. I seem to be the guy developers come to and pitch ideas to for feedback. Not sure why but I thin think that the experience of talking through a programing or design problem tends to be a productive learning experience for everyone. In any case the details aren’t terribly important

What stuck in my head is that an off the self, but non-trivial part of the system was written in Java.

We all inhaled sharply.

I don’t know what it is about Java, and I don’t think it’s just me, but the moment I find out that an application is written in Java, I have a nearly visceral reaction. And I don’t think it’s just me.

Java earned a terrible reputation in the 90s, because although it was trumped as the next big thing every user facing application in Java sucked: first you had to download a lot of software (and hope that you got the right version of the dependency) and then when you ran the app it took a while to start up and looked like crap. And then your system ground to a halt and the app crashed. But these problems have been fixed: the dependency issue is more clear with the GPLing of Java, GUI bindings for common platforms are a bit stronger, computers have gotten a lot faster, and perhaps most importantly the hopes of using Java as the cross platform application development environment have been dashed. I think it’s probably fair to say that most Java these days runs on the server side, so we don’t have to interact with it in the same sort of hands on way.

This isn’t to say that administering Java components in server operations is without problems: Java apps tend to run a bit hot (in terms of RAM,) and can be a bit finicky, but Java applications seem to fit in a bit better in these contexts, and certainly have been widely deployed here. Additionally, I want to be very clear, I don’t want to blame the language for the poor programs that happen to be written in it.

Here are the (hopefully not too leading) questions:

1. Is the “write once run everywhere,” thing that Java did in the beginning still relevant, for server-based applications? It’s a server application after all, you wouldn’t be loosing much by targeting a more concrete native platform.

2. Is the fact that Java is statically typed more of hindrance in terms of programmer time? And will the comparative worth of Java’s efficiency wear off as computers continue to get more powerful

Conventional wisdom being that while statically typed apps “run faster,” but take longer to develop. This is the argument used by Python/Perl/Ruby/etc proponents, and I don’t know how the dynamics of these arguments shift in response to the action of Moore’s Law.

3. One of the great selling points of Java is that it executes code in a “managed” environment, which provides some security and safety to the operator of the system. Does the emergence of system-level visualization tools make the sandboxing features of the JVM less valuable?

4. I don’t think my experiences are particularly typical, but all of the Java applications I’ve done any sort of administrative work with have been incredibly resource intensive. This might be a product of the problem domains. Using Java is often like slinging a sledge hammer around, and so many problems these days don’t really require a sledge hammer.

5. At this point, the amount of “legacy” Java code in use is vast. I sometimes have trouble understanding if Java current state is the result of all of the tools that have already been invested in the platform or the result of actually interesting and exciting developments in the platform. Like Clojure. Is Clojure (as an example,) popular because Lisp is cool again and people have finally come to their senses (heh, unlikely) or because it’s been bootstrapped by java and provides a more pain free coding experience for Java developers?

Anyone have ideas on these points? Questions that you think I’m missing?

Things I'm Wiking On

Fri, 12 Mar 2010 00:00:00 +0000

The Contra Purity Test: A couple friends and thought it might be cool to work on a “purity test” for contra dancing/dancers. Mostly for jokes, I said “I’ll make a wiki page,” and then I did. It is, I’d say 70% done.
Some writing and thinking about TumbleManager, tumblelog engine in response to this post. I’m writing this as a design document/spec sheet because I’m a writer and my brain works like this.
The Lessons from System Administration is the beginning of an archive of posts that I’ve written recently drawing lessons learned in system administration into other more generally useful contexts.
Sygn Networking - I’ve done some editing recently on this older project with an attempt to move it forward. Reorganization, cleaning up of text, adding new points in where they make sense. Of particular interest is, I think the data examples of what sygn profiles might look like. I hope this all helps illuminate the project a bit.
Although this only barely counts: I did some very wiki-like editing to the tubmlelog post, which I think cleaned up the text a bit. I’m, as you’re aware, not a very polished sort of writer. But I think I’m pretty good at cleaning things up after a fashion, and once I have the right distance. Or something. It’s a bit clearer now, at any rate.
Also not wiki related specifically, but I’ve been doing some rolling revisions on the Cyborg Institute website recently. I don’t want the CI project to die, but I want it to be more realistic, and useful to other people as a platform for some really awesome projects. I think by focusing the site, and pulling out some cruft we’re getting in that direction.

In addition to the above, I wanted to explore the possibility of doing more posts like this occasionally as a way of saying “Hey! I’m working on cool things like the stuff I write about here, but the stuff isn’t getting posted here. Check them out” is that a valuable use of this space? Or would this tedious?

Onward and Upward!

Input in the Next Wave

Wed, 10 Mar 2010 00:00:00 +0000

In response mostly to my own comentary of the iPad I’d like to lead a collective brainstorming of input and computer interact modalities in “the next wave.”

What’s the next wave? That thing that’s always coming “soon,” but isn’t quite here yet, the thing that we are starting to see glimpses of, but don’t really know. Accepting for a moment that things like Blackberries, netbooks, Kindles, iPads, iPhones and the like are these “harbingers” of the next wave.

The “make or break” feature of all these new and shiny things is the input method: how we get stuff from our heads into a format that a computer can do something with. While I’m a particularly… textual sort of guy, the “input question,” is something everyone who uses technology will eventually come to care about. Blackberry’s sell because they speak “messaging,” and because most of them have hardware keyboards. The iPad, with its bigger onscreen keyboard and external keyboard dock, is--to my mind--an admission that the little onscreen keyboard of the iPhone doesn’t work if you want enter more than 50 or 60 characters at any given time.

I love a good hardware keyboard. A lot, and I’m not just talking about the kind on the blackberry, but a real keyboard. The truth is I can’t even quite bring myself to justify one of the little “netbooks” on the principal that everything I do involves massive amounts of typing. And fundamentally, at the moment there doesn’t seem to be a good replacement for getting data into a computer system, that doesn’t involve a keyboard. Clearly this can’t hold out forever, and so I’d like to pose two questions:

What kind of computer interfaces will replace the command line?

So in 2010 most people interact with their computers by way of the mouse and a lot of pretty pictures. Even mobile environments like the iPhone/iPad/etc. and the Blackberry have some sort of a pointer that the user has to manipulate.

But the truth is that this kind of modality has always been inefficient: switching between the mouse and the keyboard is the greatest time sink in current user interfaces. Graphical environments require increasingly sophisticated graphics hardware, they require users to memorize interfaces in a visual way that may not be intuitive (even if we’re accustomed,) and they have incredibly high development costs relative to other kinds of software. Furthermore, most of us use a lot of text-based interfaces weather we know it or not. Google is a command line interface, as are most web browser’s address bars. And although my coworkers and I are hardly typical, we all have a handful of terminals open at any given time.

Clearly shells, (e.g. bash, zsh, and the like) are not going to be around forever, but I think they’re going to be around until we find some sort of solution that can viably replace the traditional shell. We need computer interfaces that are largely textual, keyboard driven, powerful, modern, lightweight, fast, and designed to be used interactively. I’m not sure what it looks like, but I know that it needs to exist.

What kind of interfaces will replace the keyboard for data entry?

When I was writing the iPad reflection, I thought it might be cool to have an input device that was mostly on the back of the device, so that you hold the device in both hands, your fingers make contact with some sort of sensors on the back, with your thumbs touching something on the front, and there’s some sort of on-screen interface that provides feedback to make up for the fact that you can’t see “the keys.”

I’d be inclined to think that this would be QWERTY derived, but that’s as much a habit as it is anything. I’m a pretty good touch typist, not perfect, and not the fastest, but I don’t have to think at all about typing it just happens. But I don’t know or think that the QWERTY keyboard is going to be the interface modality of the future. While I do want to learn DVORAK typing--but haven’t managed to really feel inspired enough to do that--I think its more productive to think about replacements for the keyboard itself rather than alternate layouts.

Thoughts?

If Open Source is Big Business Then Whither the Community?

Tue, 09 Mar 2010 00:00:00 +0000

I’ve been thinking recently about the relationship and dynamic between the corporations and “enterprises” which participate in and reap benefits from open source/free software and the quasi-mythic “communities” that are responsible for the creation and maintenance of the software. Additionally this post may be considered part of my ongoing series on cooperative economics.

When people, ranging from business types, to IT professionals, to programmers, and beyond, talk about open source software we talk about a community: often small to medium sized groups of people who all contribute small amounts of time to creating software. And we’re not just talking about dinky little scripts that make publishing blogs easier (or some such), we’re talking about a massive amount of software: entire operating systems, widely used implementations of nearly all relevant programing languages, and so forth. On some level the core of this question is who are these people, and how do they produce software?

On the surface the answer to these questions is straightforward. The people who work on open source software are professional programmers, students, geeks, and hacker/tinkerer-types who need their computers to do something novel, and then they write software. This works as model for thinking about who participates in open source, if we assume that the reason why people contribute to open source projects is because their individual works/contributions are too small to develop business models around. This might explain some portion of open source contributions, but it feels incomplete to me.

There are a number of software projects that use open source/free software licenses, with accessible source code, supported by “communities,” which are nonetheless developed almost entirely by single companies. MySQL, Alfresco, and Resin among others serve as examples these kinds of projects which are open source by many any definitions and yet don’t particularly strike me as “community,” projects. Is the fact that this software provides source code meaningful or important?

Conceptualizing Scale

Mon, 08 Mar 2010 00:00:00 +0000

I’ve been thinking about how ideas, projects, and ideas scale a bit in the past few weeks, and as usual, I wanted to collect a few of these thoughts. This post is generally in my series of posts of “Extrapolations from Systems Administration.” Inspirations and origins of these ideas come from, in part:

My Tumblr Killed the Tumblelog Star post.
My Google reader unread count.
Savage Minds Post on Social Enterpernurship and Anthropology post.
All the television I don’t watch.
The nginx HTTP Proxy module.
Seeing the Forest for the Brand

The internet is a big place, you don’t need me to tell you this, but I think that it’s really incomprehensibly big. Even the small corners of the internet that we (well, I at least,) inhabit contain vast amounts of information and it’s very difficult to keep your head above water, to feel like you’re in touch with what’s happening. Strategies for managing this information successfully are as concerned with “figuring out what to ignore,” as they are about figuring out how to absorb information successfully.

Scaling an idea or a concept (like a blog, or a piece of software or a web server) to be able to address problem sets (like an audience, or a given set of data, or both) of different sizes is just as difficult. It’s tough to get a web server to be able to host really large loads, its difficult to be able to write a blog that appeals to a huge audience: the this nexuses of related problems are quite large.

I think, however, we can begin to draw some conclusions. I do hope that you’ll be able to help me add to this list. Perhaps in a wikish page.

Be the biggest fish in the smallest possible pond.

The core strategy here is to avoid having to figure out how to scale up to “full speed,” by reframing the problem set. You don’t have to become the most popular or widely consumed blogger/novelist: you just have to become the most popular blogger about cyborg philosophy, or the political economics and philosophy of the open source world. You have to become the most popular post-colonial historiography space opera novelist.
Don’t participate in the proliferation of crap: only build/use what you need to.

I see lots of people say something along the lines of “I want to make a websites for all of the people interested in what I’m interested in, and we’ll need a wiki and some discussion forums, and some sort of blogs, maybe a lot of blogs, and…” This inevitably leads to a bunch of organization and building of things for their websites, and then everything is built and… no one is interested in using the crap.

This is a classic premature optimization problem. Don’t build things that you think you might need later. Build things that you need now. Or things that you really needed last week. Focus on the thing you do, and build the infrastructure as you need it, when you need it.
Work in a scalable and sustainable manner, and assume that other people will need to pick up on your projects.

While you shouldn’t expend the effort to scale before you need to, because that could end in failure, it’s common sense to approach your projects with the assumption that other people might have to finish them for you, if things take off and you need to delegate later you’ll be ready for them. Consider the possibility that you might need to scale a project when you’re in the initial planning stages and avoid getting backed into a corner by a decision.
Ignore everything you can possibly stand to.

There are so many things that you could be doing with your time. There are so many distractions. Email lists, RSS feeds, the work of other people in your field. Charity projects of one sort or another. All of these things are important and you should participate fully in the communities that surround your work, but be fully aware that humans as individuals don’t scale well, and succeeding at [your project] is dependent upon your ability to ignore everything that you can stand to.
Consume information on your terms, in the formats that make the most sense to you.

As a corollary to the above, the way to successfully engage and manage everything that you can’t possibly stand to ignore is to as much as possible engage on your terms. Figure out what your terms are first, and then work to consume content in these terms.
Use technology and media to build relationships rather than accumulate information.

Too often, I think, the geekier among us (and I count myself among this number) are interested in technology because it’s cool, and we’re tempted to solve technological problems and learn about the inner workings of stuff because they interest us. And that’s okay, as a hobby: in the pursuit of doing work, technology is only useful insofar as it allows you to get things done. And in most cases, the core function that technology provides is to enable relationships. So focus on that, and fiddle with the technical underpinnings, only when you must.

Onward and Upward!

Distance from the Divine

Fri, 05 Mar 2010 00:00:00 +0000

When I talk about Sacred Harp singing with my friends from college, they all look at me like I’m crazy. “Right, I go sing 18th century hymns set to music in the 19th century (and later,) with my hippie and queer friends in quasi-archaic harmonies. It’s a blast!” This isn’t my tradition, both in the sense that I don’t come from a sacred harp singing family, and in the sense that I come from a particularly unobservant Jewish family.

There’s something about Sacred Harp, that I think is hard to describe. Everyone comes to it for different worlds, and there are a lot of people who grew up with it and/or identify as Christian in the community. While I think most people who sing Sacred Harp find it a deeply fulfilling experience, for many its as much about the shared experience, the guttural experience of the chords, and the “sacred/special space” as it is about the texts. This is something that I think is pretty difficult to explain to someone who hasn’t felt a singing.

I mentioned a while ago my interest in writing hymns in the tradition of the sacred harp, and I don’t have anything to present to you yet, or even some example of what I’m playing with, as I haven’t actually constructed anything. I do have the following observation.

The texts and aesthetics of the Sacred Harp is encapsulated with what I think is probably a not very contemporary view of the divine. In my reading, the texts display a relationship to the divine that is distant and detached. I might even say reserved and tentative. The speakers in the text are not individuals but more collectively constituted persons--this fits well with the way the music is sung--and the speakers generally do not interact with or speak to the divinity directly, except possibly to give thanks or appreciation. Like the harmonies themselves, the texts are spare and stark; indeed I suspect their simplicity makes the tunes easier to sing (words are almost always easier to sing than shapes,) while being very careful to not distract from the tunes.

As I said, this isn’t my tradition, at all, in addition to the structural constraints of the form (common meter, for the most part,) I think the stylistic constraints present the larger challenge. I’m trying to figure out how to write text that isn’t cloyingly spiritual, and that fits my own worldview (such as it is,) but that also respects this kind of spiritual aesthetic, for lack of a better term. I’ve been reading up on various religious phenomena, contemplative orders, and the Religious Society of Friends. Because it seems appropriate.

As is, I think always the case, minimalism is always a lot harder than it looks. I’ll keep you all posted!

Software as app Store

Wed, 03 Mar 2010 00:00:00 +0000

This post represents two major ideas, first of “app stores,” and second of “Sass” or “software as a service,” which seems to be the prevailing business model for contemporary technology companies that aren’t stuck in the 80s. With reflection on free software, open source, and the technology industry as a whole. Because that’s sort of my thing.

On the one hand the emergence of these tightly controlled software distribution methods represent a fairly serious threat to free software, as does SaaS particular insofar as SaaS exploits a GPL loophole. On the other hand these models, potentially, represent something fundamentally awesome for the technology and software world, because it represents a commonly accepted paradigm where users of software recognize the value of software, and the creators of software can get compensated for their work. It’s not without its flaws, but I think it opens interesting possibilities.

Free and Freedom

Obviously app stores present a quandary for those of us involved in the free software world. On the one hand app stores are not free, which is a trivial complaint. It’s not the cost, around which “free software” is truly centered, the true failing here is that creators of software cannot choose to participate in an app store system and distribute source code: the interaction and relationship between developers and users is very scripted and detached. These issues all grow out of the reality that app stores--by design--are they’re controlled by a single institution or organization.

Which isn’t itself a bad thing--there are contexts where centralized organization means things get done more effectively, but centralized authority is not without risk. So while this question isn’t resolved, it’s also the kind of question that requires ongoing attention and reflection.

Paying for Software

At the same time, I think it’s very true that the “app store model” and indeed the more successful “Web 2.0” business models (e.g. new businesses on the web, post-2003/2004) have posited that:

Software is a thing of value that users should expect to pay for.

And that’s not, at least to my mind, a bad thing for the software world. Free or otherwise. Or not always a bad thing, particularly for end-user software. For larger pieces of software (in the “Enterprise”) money is largely exchanged for support contracts and for services related to the software: custom features, IT infrastructure, etc. For end user software, support contracts and custom features don’t tend to make a lot of sense in context: so perhaps moving back to the exchange of money for software isn’t a bad thing.

The connection between “value” (which software almost certainly creates), and currency in the context of software is fraught. Software isn’t scarce, and will never be (by nature.) At the same time it does have value and I think it’s worth considering how to arrange economies that involve exchanging money for software. There are a lot of factors that can effect the way that app stores might work, and I think given the possibility for causing interesting things to happen we shouldn’t dismiss them out of hand.

Despite my (potential) lack of clarity at the moment, I think the following posts reflect some of my earlier thoughts on this subject.

Coming of Age In The Science Fiction Community

Tue, 02 Mar 2010 00:00:00 +0000

I said to a new writing friend “I’m young, particularly given that anyone under the age of 40 in the science fiction community is considered ‘a young writer.'” Which is, more or less true (on both counts,) and brought on a couple of trains of thought that I’d like to explore in a bit more depth:

1. The “youth” of a writer is long, indeed much longer than one would expect.

2. I’ve found a community of science fiction writers. Admittedly I’m new and very much on the outside, but I find it delightful that all of the “things I do,” are part of communities one sort or another: Sacred Harp singing, Morris Dancing (in the Midwest, particularly,) Contra Dancing (on the East Cost, particularly,) Free/Open Source software, blogging, and apparently Science Fiction writing.¹

The Portrait of the Author During Youth

I’ve written here before about the challenges and inherent problems of “being a writer:” the work we do is potentially hard to understand, good writing is more than the sum of its parts, and because writing is a sign of education for most people, sometimes it’s difficult to figure out (even those of us who “are writers”) to figure out what’s “writing,” and what’s just throwing words together.

Now to be fair, I’m not complaining that the period of “youth” as a writer is so long. This standard seems wrapped up in the idea that a large component of being a “real writer,” is having lived long enough to have enjoyed a great deal of unique experiences (which can inform your work,) and also to have had enough time writing “crap” to be able to have the (learned) skill of being able to construct quality texts.

It’s really hard to tell people, epically the young, that they need to “wait until they’re older.” But I think once we (I) get done with the pouting, there’s a pretty strong silver lining: the extended adolescence of the writer provides a longer window to read, to experiment, to apprentice to other writers, and to grow as a writer. Additionally, if the “youth” of a science fiction writer is longer than it is for writers in other fields (and I suspect it is, slightly,) the science fiction community has created a way to compensate for the exclusion of science fiction from most academic writing programs. These are all largely good things, to my mind.

Community Discourses

I think communities are fascinating, and I’m delighted to touch so many different and interesting communities. It seems to me that the formation of communities is very much not a project for youth. As young people, our communities are local, and based on where we go to school, where we live, even where we work. The communities I’m thinking about there are, in turn based on what we’re interested in and what we love to do. Although there’s a potential for insularity and self-selecting qualities, there’s also a great potential for diversity. There are a lot of different kinds of science fiction writers, sacred harp singers, folk dancers, open source hackers, and so forth.

There’s another interesting set of common factors for these communities: they’re all built around shared experiences and activities in the “real world” (as it is,) but the members of these communities tend to be scattered across a given geographic area. Though I don’t have much to compare this to, personally, but I think the ways that these communities are supported and connected through the Internet. As much as Facebook irritates me on a technological level, its done it’s job.

The principals under which communities function and adhere are not something I have a terribly firm grasp of, I must confess, but I know what I find myself in one, it’s a good thing indeed.

I have, it seems too many hobbies and avocations. ↩︎

The Things I'm Going To Do Today

Mon, 01 Mar 2010 00:00:00 +0000

Ok, so not really.

This post is mostly about playing a head game with yourself, in an effort to get more organized. But not “head games” in a bad way. On my to do list for the past few weeks I’ve had something like “write a blog post about todo list item titles,” because in light of this post about org-mode it seems like a topic in need of further definition. Basically my goal is to explore, the best way to think about what we have to do, to allow us to accomplish what we want to. The GTD system which so many people are enamored of present a few ideas on the topic, and while the GTD way is a good place to start thinking, it’s not a good place to stop thinking.

We’ve all done it. Made a todo list that we didn’t end up using for one reason or another. Todo lists, need to be useful: they should help us organize our day, and help us keep track of all the things wee need to accomplish. In a lot of ways, maintaining focus over our day and all of the tasks that nag at us are contradictory, so todo lists are failed by design.

The first, and frequent issue in my own organization are lists and plans that go too far and list too many “actionable items.” This divides your time and actions into too many little pieces, leading to a number of outcomes. The first risk is that you might start to ignore the list entirely because it’s too long and complicated, even if it’s an illusion caused by the size of the items on your list. Ergo, the total length of the list you “work off of,” needs to be manageable and comprehensible.

The second risk, is that, an overactive todo list is one where you over plan for yourself, such that your list--while accurate and comprehensible--isn’t useful. Beyond simply providing “outboard memory,” the best todo lists allow us to structure and make plans for our working time. When working (e.g. writing, at the computer, etc.), I like to have my projects chopped up into pieces that can conceivably get done in the time I have to work on them, but that give me the time and freedom to bury myself on a side project, or follow inspiration or a train of thought to its completion when needed. While effective todo lists help you structure your time, flexibility is still valuable.

There are issues on the other end of the spectrum as well: when lists are too short, and the “actionable items” on a list are too conceptually large, the effectiveness of lists is degraded as well. A reminder to “write a novel,” even a specific novel, is less than helpful for helping you accomplish something in the moment. Even a dozen items, on a list where you end up checking something off once every day or two doesn’t help you figure out “Ok, what do I need to work on now.” Besides the chances are, if the items are too large and the list is too small, that you probably have it memorized anyway.

Right? Other strategies?

Kindle and Paradise Regained

Thu, 25 Feb 2010 00:00:00 +0000

As you all might have heard that Amazon (finally) released a Kindle Application for the Blackberry. When I heard this I thought that this would be a good thing, as I have (and quite enjoy) both my Blackberry and Kindle. Here’s the rundown:

The Kindle App for the blackberry is probably the most well designed blackberry application I’ve seen thus far. Having said that, the bar isn’t terribly high.

In a lot of ways, the way (before the Kindle app) to make a “successful” blackberry application is one that figures out how to make its data “fit” into an email or messaging context and then blend that data into the messaging/event feed in a useful sort of way.

This doesn’t do that, and I think learns a great deal from advancements made in iPhone app development. The resolution on the Blackberry Bold is amazing (same number of pixels as the iPhone, much greater density.) and the buttons/interface is really intuitive and well designed. The app itself gets as many thumbs up as I can manage.
I’ve been having phone angst recently. I don’t use it very much, I need to have better filtering of my email and reorganize how I do my voice mail, and while this is easy enough to say here it’s a much more substantial project than I’ve got time for now.
The Kindle App isn’t a replacement for the kindle, but it’s a great compliment, and it makes it much more possible to lighten the load in my back-bag, and it makes it easier for me to entertain myself with my phone. This might not seem like a bit deal, but I think it is.

There are also situations where the Kindle isn’t usable (in bed when the lights are off and various other low light situations) and that’s alright, but the Kindle app is. So that’s a good thing indeed.
I had hoped that the kindle would make it easier to read in the in between moments throughout the day when I might read but didn’t. That isn’t exactly true, as it turns out. Reading on the Kindle still requires a fair piece of directed attention, and it’s not the kind of thing you can idly whip out while you’re waiting in the grocery check out line.

I’m not sure at this point, of course, but I do think that having access to the Kindle on the phone will improve this usability feature.
I’m sort of annoyed by the lack of subscriptions. While you can have multiple devices attached to your Kindle account, when you subscribe to a periodical, that content is only accessible to you on one of your devices. I don’t really like this, and it represents a huge loss of value for the Kindle store.

While I got the Blackberry shortly after the first iPhone 3g came out, the “app explosion” hadn’t really happened yet. I must confess some “app jealousy.” The Blackberry is awesome, and really it does the messaging quite better than anything else around (I’m convinced.) And I love the hardware keyboard. But when I think “I’d like to do something with my phone,” the chance of getting a Blackberry app to do that is… unlikely. I don’t know if I want a lot of apps on my phone, in the end, but I know the hardware is capable, and it’s nice to take advantage of that from time to time. In any case…

Onward and Upward!

There's a Widget for That

Wed, 24 Feb 2010 00:00:00 +0000

Perhaps part of the reason the iPad doesn’t make sense for me is that I’ve never really used an iPhone, and at the core iPad is basically iPhone++. I opted to get a Blackberry because I wanted a device that did email right and messaging, I had/have an iPod Touch that I don’t really use and the little isolated applications that don’t interact well with my existing work flow and lock data in didn’t really appeal to me.

People are always talking about “the apps,” as if there’s something revolutionary happening here, and I’ve always been a bit non plussed. At least for my personal use, more “applications,” or places where “things can be done,” doesn’t always equate into more productivity.

Running in parallel to this, of course is the whole Web 2.0/web-based application development thing. (Is Web2.0 coasting to a halt yet? In any case, it’s probably still a current phenomena worth noting.) Although iPhone applications and Web applications are constructed in very ways that are quite technically distinct, there’s an underlying commonality between the way that both function. These applications are very much not general purpose. The best iPhone (or web) applications take a specific kind of data or information (or a collection of such), and provide a way for you to interact with that information in a way that’s (hopefully) intuitive.

In contrast the old familiar applications typically are designed to be either: platforms that enable you to accomplish a whole range of tasks within a certain sphere (“Office tools,” email applications, emacs), or some sort of filter or pipe that allows you to access data in one format in an interface that’s easy to use (RSS readers, Content management systems, pdf readers, web brwosers, etc.) There are certainly web applications that are basically pipes, and I’m sure Google would love it if we all used their application stack.

So, then, what describes this new “widget,” class of applications for mobile devices and the web? On the one hand it’s a case of developers “making do with what there is:” the iPhone imposes a number of serious constraints: small screen, limited on board computing resources, and a potentially inconsistent data connection. Small structured apps that are all about allowing you to interact with a small set of data work best in this environment. Additional I suspect that technology has advanced to a point where making these kinds of applications is finally possible and worth the it takes to build them: MVC frameworks have finally matured perhaps.

The enduring questions in my mind:

Is the data that we’re throwing into databases by way of iPhone apps and web applications going to continue to be useful in the future? This gets at the “open data” question as much if not more than the “open source” aspect of applications. What’s the realistic utility of our data in the long form?
Does this shift mark a move away from a certain kind of “paper-based logic,” to a more digital way of organizing or data and digital experiences?
Is this just the process of bringing the kinds of “business applications,” that enterprises have developed and used for decades into the consumer sphere?
How do the concepts of software freedom and data ownership transfer to this mode of using technology. While it’s easy to say “data ought to be free,” and “source code still matter” (both of which are true,) I think practically it’s a bit more difficult.
What do these applications look like on the Desktop? Is the future JavaFX, Flex, Silverlight, and Apple Desktop Widgets? Is the desktop going to be stuck in the browser? Are there other options?

And while I think we’re definitely using and creating this kind of software with abandon, I don’t know if we’ve reached good answers to these (and other related) questions. I look forward to exploring these ideas, and hearing your thoughts!

Tumblr Killed the Tumblelog Star

Tue, 23 Feb 2010 00:00:00 +0000

A few years ago, highly citational, link/youtube video blogs came back into style again. This time rather than calling them blogs, we called them “tumblelogs.” I never really got into it, though I tried, and even my original inspiration for starting tychoish.com was to do a more “tumblelog-esque” blog.It never quite worked out. Then I read this post by Michael Coté which inspired a few things:

First, it got the following title to stick in my head and refuse to get out. Second, it left me with the idea that, although successful, sites like tumblr and to a different extent posterous basically ruined the tumblelogging revival.

Here’s the thing about tumblelogs: they worked and worked so well because they were efficient, because the people creating tumblogs were doing something unique and had unique voices, because you could keep your pulse on most of a single discourse by watching only a few sites/rss feeds. And then it became, very suddenly, trivial to make a tumble log. And so everyone had a tumblelog, and it was like blogging was new again, except things “meme’d out” at an epic pace and it became difficult to track what anyone was saying. It was like a distributed denial of service attack on our attention spans.

And as the dust settled, tumblelogs, at least as far as I could see, became less about a sort of delightful amalgamation of interesting content and more about templates, about piping in a fire hose of content from delicious/twitter/etc. So not only were there too many tumblelogs, but the style had devolved somewhat into this weird unedited, awkwardly template-ed mass of “crap” that is (in my opinion) quite hard to read or derive value from.

What Made Tumblelogs Work Originally

The systems that powered them were kludgy but they made it very possible to post content easily. That’s a good thing.
They used a unique sort of web design where design elements (tables/grids/CSS magic,) reflected and accented the content type.
They were largely editorial functions. People followed tumblelogs because their authors were able to filter though content with exceptional speed and grace, and in the process of filtering provide real value.
They were multimedia, and incorporated many different kinds of content. Not just links, not just embedded youtube videos, but snippets of IM and IRC conversations, song lyrics, pictures from flickr, and so forth.
projectionist one of the first and best, was a group effort: when group blogs work, they really work. The tumblelog, seems like an ideal platform for group blogging.

How We Can Make Tumblelogs Work Again

We use publishing systems and tools that are unique and that stretch and bend the form. A tumblelog theme for Wordpress, will probably always reek like wordpress. Same with other popular content management systems. Tumblelogs work because they’re not just blogs, they need to distinguish themselves both visually, and in terms of how their authors write the content.
We undertake tumblogs as a collaborative effort. Group projects complicate things, of course, but they also create great possibilities.
Vary content, intentionally, post quotes, chat excerpts, links, videos, lyrics, etc. Make sure that there’s a great deal of diversity of content. This is perhaps a problem to be solved in software, at least in port.
Emphasize and cultivate editorial voice, and create an interface that forces authors and editors to touch the data.

Thoughts? Suggestions?

ETA: I’ve started to work on this wiki page outlining a “tumble manager” tool. I also did a bit of textual refactoring on February 27, 2010

A Flurry of Contra Dance

Mon, 22 Feb 2010 00:00:00 +0000

I went to a Dance festival a few weeks ago, and wanted to collect, as I am wont to do, some thoughts on the subject. No particular order or organization.

There was a sacred harp singing event at the festival, and then a group of us had a little ad hoc sacred harp session in a hallway (better acoustics) afterwords and the next day. The end result: I feel like I’m starting to really have a clue about Sacred Harp singing. I mean I’m not great or anything, but I don’t feel epically lost at a singing. Because I don’t come from a particularly background, and don’t have any real training with the singing thing, singing from the Sacred Harp has been an adventure. But now I sort of feel like I have the basics: I’m able to get the pitch most of the time. I feel like I have a good sense of myself as a leader and the kinds of songs I like. I’m beginning to become more familiar with a number of different songs in the book, and so forth. I don’t have the shapes memorized (or the middle verses to most songs!) but these things are coming.

It was also a very rewarding experience to be part of a group of singers who did that--for lack of a better term--jamming in the hallway. I don’t play an instrument, I don’t really lead songs in informal settings, so the whole “jamming in the hallway thing” is something I’ve never really been able to participate in. And getting the chance to do that was pretty cool.

But it was a dance festival. So how was the dancing? Pretty good. the space sucked (bad floor,) but the quality of the dance was wonderful. Contra-dancing in the Midwest, where I hail from, isn’t the hip musically experimental thing that it is out here. Part of that is due to the incredibly strong Old-time music community in St. Louis, which isn’t a good thing (even the less amazing contra bands in St. Louis are pretty damn good.) And there are a lot of factors that make contra dancing awesome out here: open/participatory bands as musical training grounds, more bands that travel (because things are closer together,) and more dancers that travel as well. That’s always nice.

Having said that, I’ve never really felt like a contra dancer. In high school, when I got into folk dance, I did a lot of international folk dance, and I think my defining folk-dance participation these days is Morris dancing, even though I probably do more contra than anything these days. When you’re a kid in the dance community, particularly one of the few kids in your local dance community you can pretty much glide through everything on enthusiasm and good intentions. While I think I’ve learned a few things about folk dance, I’m aware that I’m not the flashiest, or the most polished dancer around. This is particularly apparent at big dance festivals. But I got to dance with people who I think of as great dancers, and I had a lot of fun, so maybe it all works out. And then I went to a local dance and felt like a great dancer, particularly in terms of my ability to gracefully recover from flubs, and my sense of a dance’s relative complexity. So there you have it.

Things that are awesome about contra dance:

Gender swapping partners in a dance. Particularly when everyone can go with the flow.
Dips.
Long rooms where you don’t get to the other end of the set in a given dance.
Bands with awesome elements like foot percussion and woodwinds.
Changing lines when you get to the other end of the dance.
Rewriting a dance from the dance floor. Partner do si dos that become balance and swings. Shadow alamandes that become swings.
Spins with your neighbor’s partner in the middle of heys for four.
Conspiring with your partner to mess with neighbors.
Graceful recoveries from dance flubs.

Onward and Upward!

Simple Gifts

Wed, 17 Feb 2010 00:00:00 +0000

I suppose I should apologize for the awful relationship between the title and what I’m about to write about. Titles, particularly for blog posts should be functional and descriptive: google won’t enjoy or take pleasure in your puns. Nevertheless…

I’ve been working on something of binge of blog posts to prevent this from happening again any time soon, and I’ve noticed something: my posts aren’t nearly as epic as they used to be. Nothing that I have in my drafts folder is longer than 750 words, and most of the posts are under 650.

This probably calls for some sort of celebration.

I’m notoriously long-winded, and although I’ve been a really bad practitioner of “keep your blog posts short, concise, and clear,” I really do think that there’s a sweet spot for website-based content around around 600 words that’s really easy to read and comprehend on a computer screen. Even if most of my posts are a bit over this mark.

So what gives? why have I finally been able to figure out how to say what I want to say in fewer words? Here are the current theories:

The writing I do for the day job is teaching me (slowly) to be a bit more concise.
My self imposed schedule is forcing me to be a bit more granular in the topics I choose to attack in a single blog post.
I’m getting to the point quicker. I don’t feel like I’m spending as much time running around my arguments attempting to explain the premise.
I’ve become more of a textual stylist than I ever used to be before. While I don’t think I’m a stunning prose stylist, I’m much more aware of how my paragraphs come together these days, and I think that leads to more clear prose.
I’m getting better at using unordered lists to organize information rather than as a rhetorical crutch. (Most reflexive bullet point ever.)

I’ve written about this before. I had a class in college, where the professor assigned these short (250 words,) “journal” entries that were due on a weekly basis. They didn’t have a topic, and most people reflected on the readings. I reflected on my other classes and how they related to the topic of the class I was taking. Half way through my roomate (who was also in the class,) commented that I hadn’t actually written about the readings for the class.

“The journal entries don’t have topics. And I’m writing about the core material of the class,” I said.

“You have to admit that it’s a bit absurd,” she said.¹ She was right.

I wrote the professor who was apparently fine with my eclectic interpretation of the assignment. The pieces were mostly technical exercises in being clear and concise, and she thought my entries were fine, if a bit esoteric. (I think the exact words were something along the lines of “delightful and widely synthetic.") And so I kept writing those kinds of pieces.

A year later I started really getting into blogging. The rest is history. When I first started at this, I enjoyed the freedom being able to write about whatever I wanted. Now I cherish the structure more than anything.

Its funny how things change, sometimes.

Apologies to H.S. for the liberties I’ve taken with her words. ↩︎

Managing Management Costs

Tue, 16 Feb 2010 00:00:00 +0000

Every system that requires your attention and responsibility comes with some sort of “management cost,” this includes servers that run websites and email, as well as the notes you take and--in my case--the novels you avoid writing.

This post, and really the last one as well, grows out of my interest and desire to stay organized, to work effectively without spending too much time and energy thinking about organization. Except of course that I write a bunch about this sort of thing on the blog, so maybe I’m a bad example of success. At the end of the day we’re all just folk', I guess.

The argument at the present moment revolves around consolidation rather than an approach to design or organization. And the basic premise is: “no matter how complex your organizational problem is, you can probably accomplish what you need to by doing less.”

Feel like you spend too much time reading email, or have too many email inboxes to check (personal email, work email, special project email, listserv email, facebook email, etc.)? Forward your email into one box and filter the hell out of it so that you only read what you really have to and it’s manageable.
Feel like you have too many todo lists? Compile them into a single list and use some sort of tag system to organize it.
Feel like your notes and documents are scared in too many places? Combine them and use some sort of search tool to find things when you need them.

And so forth. In the analog information world (i.e. with papers, notebooks, and books) we often take the approach of sorting things into distinct piles of similar sorts of things, and arranging things physically in our worlds to reflect this basic sorting. For instance, “the science fiction books will be on the first three shelves, the 20Th century philosophy on the next three, college textbooks on the next, and […]” These habits, combined with unfortunate conventions like referring to hierarchical organizational units of a file system (e.g. directories) “folders,” encourages us to translate these real-world conventions to our digital existences. This is undoubtedly a bad idea.

The more data you pile together in one place, even dissimilar data, the more powerful it becomes. Say you have a PDF collection of articles on the anthropology of death and dying, post-colonial literature, and linguistics hanging out in different directories of your file system, and you begin to do research for a story you want to write set in the 1930s in India, where do you look? What if there are relevant articles in all three folders. What if you have a dozen or two dozen folders? What if you have a number of hierarchical organizational trees, and you store your notes, the actual text of what you’re working on, and your reference materials separately with parallel hierarchies?¹ Quite suddenly you’re over-organized and disorganized all at the same time,

The more “system” you have the more difficult it is to manage. The key to success, or part of it at any rate, is being minimalist about your organization. Recognize that adding responsibilities, projects, directories, lists, email accounts, and so forth all come with a cost. And sometimes, being a little less organized means that you’re able to get more done, if that makes sense.

If your experiences reflect this (or run contrary to this logic,) I’d be very interested in hearing about how you have solved, and have continued to solve the issue.

This kind of system actually makes a lot of sense in the paper world, but is borderline absurd in the digital systems. ↩︎

System Design and Organization

Mon, 15 Feb 2010 00:00:00 +0000

By day I write documentation for systems administrators, and as a result I spend a lot (perhaps too much?) time thinking about how we organize computer systems so that they can be both useful and easy to manage in the long run. “Right, so…” you say? Well indeed. Recently it’s become clear to me that there are some generalizable lessons to be learned from sys-admining that might be helpful to those of us who are less organized than they’d like to be.

Which is pretty much everyone, right?

Right. In brief:

Automate everything that can be automated.
Closely followed by don’t automate something that doesn’t need automation.
Prefer simplicity over complexity, and prefer systems that require you to remember fewer things.
Design systems to make it possible for others to easily understand what you’ve done.

To elaborate:

Automation

Computers are really good at doing what you tell them to do, and although we often like to finddle with them to make them work better, ideally the more we let systems take care of themselves. Also tasks that are automated, if the automation is designed and tested properly don’t make silly mistakes. If you’ve written systems to automate your tasks, you can understand and predict how your system is going to handle the kind of data that you throw at it.

The admonishment to “not automate” until you need something, is basically a variant on age old recommendation to “avoid premature optimization.” While automation is a good thing indeed, and if the thing you’re automating is really something that can be delegated to the machine without intervention on your part, then that may be worth your while to automate that task. By the same token, it’s easy to think “we’re going to need to do this thing a lot, I might as well automate it before hand.” Which is a reasonable thought to hand, but this puts the cart before the horse, and leads to two undesirable and possible outcomes: first the task doesn’t need to be automated because it isn’t needed very often; you misunderstand what needs to be done and automate the wrong part of the task, or your automation doesn’t cover the edge cases and will need to be rewritten later.

Conventionally, automation tends to cover “coding” or scripting of some sort of task. Outside of programming and systems development, “automating” a task could be as simple as creating some sort of editor macro, or developing some new structure in your data store (database, files, etc.) to hold or manage a particular kind of data.

Simplicity and Complexity

The basic reasoning here is that while complex solutions are often elegant and attractive, and make a lot of sense when you’re setting something up, they always make you scratch your head six months or a year later when you need to go back and find something that you did back then or make a change to the system. Be wary of solutions to any problem that require too much consistency on the part of the user. If a system only works if you must remember to follow more than a few steps in a precise order, chances are things are too complex, and you’ll end up screwing yourself over later.

Ergo: Err on the side of simplicity, you’ll thank yourself later.

The more components and connections there are in a website application or deployment server the more potential for breakage is. The more complexity there is the better chance that FurtureYou or someone working in your footsteps will be totally confused by what you have set up. The same thing holds for whatever your trying to organize and manage.

Generalizable Organizational Methods

Chances are you’re the only one who will be taking notes/organizing your work/storing information in your system. Nevertheless, I think it always helps to assume that other people are going to need to be able to make sense of your system. Be it your notes, and research or in your web-servers. Other people are sometimes our future selves.

I tend to use the word system, in a way that most people would use the word “method.” I hope that’s not too confusing or distracting. I think I’ll probably elaborate on these topics a bit more before in a later post. In a lot of ways this is part of the core of Cyborg Institute, and if you feel interested or inspired by this kind of stuff, I’d love to hear more from you. Be in touch!

Some Future in your Science Fiction

Fri, 12 Feb 2010 00:00:00 +0000

I finished reading Kim Stanley Robinson’s The Martians, on my Kindle the other day (the short review: It was great, I don’t know how I felt about the poetry at the end, but I liked the collection.) and promptly began reading this month’s Asimov’s. The first story is an alternate history/fantastic history/I-think-there’s-science-fiction-coming-but-it’s-not-here-yet, piece and I can’t bring myself to really read get into it. It’s well written, and I even find myself delighting at the text (in a technical sense.) I think the issue that I’m running into is that I don’t really get the alternate history thing.

Which is, you know, weird. I should break out and say that my fiction tends to be very historically concerned. I’m fascinated by history and there are a lot of historiographical themes and ideas in the stories I write. But they’re all set in the future, and try as I might, I don’t really have much interest in writing stories set in the past of our world. Alternate or otherwise.

Maybe it has something to do with my view of history. I tend to take a big picture approach to history and I tend to think that single events and single individuals rarely really affect history. If you called me a determinist I’d probably gnash my teeth for a few moments and then agree. Which makes constructing alternate histories sort of difficult. Add to that the fact that quasi-deterministic big pictures, though probably accurate and helpful, don’t lend themselves to good stories. When you don’t feel like your characters--any of them--have agency, it doesn’t make for terribly interesting story telling.

At least for me. I think other people can pull it off.

This whole “I want my science fiction to be set in the future,” thing isn’t something I can rationalize or support very well. Clearly I don’t find the past to be a very good “escape.” The future is fun, vast, and full of possibilities and enables the sorts of things that I enjoy most in science fiction: the ability to engage in a critique of the present, high energy stories with adventure, and for lack of a better term, stories that impart a “sense of wonder.” There’s more out there, I just can’t seem to muster the interest.

This isn’t to say that I don’t sometimes find myself enchanted by non-futuristic stories, it’s just not a terribly frequent or predictable sort of experience. I should also be clear, I’m not of the opinion that when science fiction stories talk about the future and are set in a future, that they are about anything except the present at all.

And I’m not terribly proud of this. I suppose we all have our things.

I worry that my tastes aren’t sophisticated enough, that I enjoy stories for the wrong reasons, or that I get too caught up in the scenery and forget to pay attention to what really matters. Despite this whole “writer thing,” that I have going on these days I don’t have very much formal training in literature. It’s sort of awkward to say “I feel like I’m not a very good reader,” that’s definitely something that I battle with.

For those of you who are part of the larger community of science fiction/fantasy/genre fiction readers (which I think necessarily includes writers,) I’d be very interested to learn your thoughts on this subject: how do you relate to the future in the stories that you write and read? The past? Alternate histories? Is there some connection that I’ve mostly failed to see? Am I not alone in this?

Thank you (preemptively) for your feedback.

Git Tips for Writers

Wed, 10 Feb 2010 00:00:00 +0000

The Context

git is this version control system that’s designed to be used in a distributed manner, and supports a very divers and non-linear workflow. While it’s designed to support the work of software developers--particularly in large projects like the linux kernel--at the core, git is just a file system layer that has an awareness of time and iteration. It also does its magic on any kind of text files… code or writing. I use git to manage a lot of my writing--indeed, most of my digital life, which is a bit weird admittedly; and as a result people on the Internet, not to mention my coworkers,come to me with git questions from time. This post is a response to a more recent change.

How I Work

I have two kinds of repositories: general repositories which store a bunch of different kinds of files that I need to work: the general repositories that store files that I always need to get things done, and specific project-only repositories that only have the text (and possibly notes) for a very specific project. I also have a “writing” repository where I do drafting for the blog, and start writing projects that I’d like to version, but are too small yet for their own project repositories. The brief overview:

garen is like my home directory within my home directory, and it has config files, scripts. and other daily essentials.
org stores my org-mode files.
fiction projects: I have five repositories in ~/ that store fiction projects, that I’m theoretically working on in some capacity, though I haven’t touched most of them regularly.
writing holds blog drafting, and a couple of not-exactly-fiction, projects that I’m not quite ready to admit exist.
website content: wikish, tychoish.com, cyborginstitute.com, the cyborg institute wiki and a few other website projects that I’m involved with have repositories to store their content.

The lesson here, about repository organization, is that git wants you to have distinct repositories for different projects. Its possible to merge repositories together (really!) and also to separate the histories of specific directories into their own repositories if you’re so inclined.

I write in emacs almost exclusively, I sometimes use magit, which is a delightful interface to git that works within emacs in a very emacs-centric way. If you use dired, magit will be familiar. Having said that, I mostly just add files, make commits and push repositories. Although I’ve been very interested in flashbake for some time, I’ve never really used it: it seems designed for people who aren’t used to version control or git, and the fact that I am means that it feels cumbersome to me. I suppose I should take this as a challenge, and attempt to hack it into something more usable from my perspective, but I’ve not felt the urge yet.

I use gitosis (but it’s in the debian repositories) on foucault (my server) to manage the publication of my git repositories. I push regularly, both to make sure that all of my machines are up to date, and also as a way of keeping my systems backed up. While I don’t take snapshots of my systems, I’ve been able to set up systems and been up and running inside of ninety minutes after reimaging a laptop without loosing a single bit. Although unorthodox, git is my backup strategy, and the restores work fine. I strongly recommend having your own git hosting set up. It’s not difficult, and while I think git hub is awesome on it’s own terms, independence and self sufficiency is really important here.

I don’t really take advantage of any branching and merging in git, though I’ve played with it enough to know how it works. I do have a branch in the repository for the novel I’m writing for an editor to be able to edit the novel as I write on it without needing to see their changes and comments until I get to that point.

And that’s sort of it. I use jekyll (or an old personal fork) and soon to be cyblog) as well as ikiwiki to publish content, but other than that, I just write stuff.

In any case, if you have thoughts on the subject I’d love to see your input on the wikish git writing page.

Write on!

iPad Reactions

Tue, 09 Feb 2010 00:00:00 +0000

Ok, as a self-respecting geek, I think I have to say something about this whole “iPad” thing.

I’m not as much of an Apple geek as I used to be, that’s for sure. It’s not that I don’t think that Apple’s doing something quasi-innovative and useful in the world of technology and the business around consumer technology. I think bringing UNIX to the hands of “mass market” desktop users was a great move. Although the iTunes Store is not without it’s issues and concerns, the fact that Apple was able to create a viable environment and market that allowed people to exchange money for software and content is probably a good thing. And Apple has brought to a general public a number of hardware configuration (in recent years) that I think make a lot of sense: the mini-tablet (iPhone/iPod Touch), laptops with usable battery lives, the pocket jukebox (iPod Classic,) and so forth.

Deep breath. Having said that… I’m not terribly impressed with the iPad, or moved by it’s potential at the moment. I know Apple often takes a few generations to make something really work, and so I think it’s important to not say “this implementation sucks, and so the whole notion sucks, and is with out merit.” Of all the things that I’ve heard (or said,) in the past few weeks of the iPad the following two threads have stood out:

I’m really quite interested to see what other makers are going to be doing in this space. What’s the Lenovo tablet going to look like? HP? One of the leading complaints about the iPad (and iPhone) is Apple’s total-lock-down over the platform, and I think an Android tablet, or a super Nokia N900 is likely to be much more open and killer awesome.
I’m interested to see what the iPad platform looks like in a revision or two. Add an SD slot? Multi-tasking? Additional input methods? It could look really awesome, and while I have misgivings (see below) I don’t want to write it off entirely.

At the same time, I don’t really feel like there’s an in-between device that I don’t currently have that I’d like to have. In a lot of ways, even I feel like I have too many devices, too many inboxes (of one sort or another), and too much technology to manage. I’m not complaining. The truth of the matter is that my laptop gets great battery life, isn’t very big at all, and does everything I need of a computer, and almost everything I need of technology in general. iPods are better for playing music if I’m moving around or in the car, and the Kindle is great for what it is, and I do sort of have a Blackberry habit.. but…

My technological challenge at the moment is that I don’t have enough time to get done that which I would like to get done, not that I have a situation where I could be more productive if I only I had a device that would do something more." That’s not something that seems to cross my mind very much. It might be nice to have a slightly more accessible emacs instance that I could use to enter snippets of text and work on things in the kinds of moment. I’m thinking a Nokia N900 might fit that bill pretty well, but I’m not sure.

If you’re thinking about getting an iPad, what’s the niche that you see it filing? Do you have a niche that seems like it might be iPad sized?

Status Update

Sat, 06 Feb 2010 00:00:00 +0000

Ok, a confession: I totally ran out of my usual backlog of blog posts, that keeps these pages humming with the signs of life despite the fact that I don’t have time to write a post every day, and often don’t have reliable weekend time to write blog posts. This hasn’t happened in a few months, so it’s sort of disconcerting. One of the reasons why I ran out, is that I’ve been busy doing non-writing things for a few weekends, but a third (or forth?) weekend away from home was too much for my backlog to bear, so here we are.

Here’s a brief overview of what I’ve been up to and the kinds of projects I’m working on, and what the near future holds:

I’m going to Dance Flury in a couple of weeks. I’ve not done the usual dance weekend thing before, aside from the Morris Dance gathering I’ve been going to for years. I’m really looking forward to it.
I’m trying to get a better handle on the blog. I do this binge cycle thing with my blog posts, that leads to situations like the one that I’m currently experiencing, but also doesn’t actually mean that I have spare time to write fiction in a way that I might like.
I’m working a bunch, and while every day is some kind of learning experience, I think I’ve figured out (largely) how to do what I do in a way that doesn’t have me leaving every night feeling like a fraud. There will be more learning curves, for sure, but it’s nice to not feel like I know what I’m doing.
I’ve been throwing around the idea of writing poetry (hynms) vaguely in the tradition of the sacred harp. I’m not much of a poet, and the project I have in mind is a little bit peculiar (but then, this is me we’re talking about, so lets not be too surprised.) At the same time I’m irrationally interested in this project, and we’ll see how it goes.
I switched to using Chromium as my primary browser. There are still a few plug-ins that I really liked on Firefox that aren’t quite as awesome or comfortable in their Chrome-equivalents, but it’s close and I’m sure it’ll get there. I’ve been looking for a viable non-Mac WebKit browser for a while, and am pretty happy with the Chrome.
Check out git for writers, wiki page that I’ve been working on over at wikish to try and explore how writers use and might use the git version control system to manage their writing. Contribute if you like.
This weekend (yay! snowpocolypse) I hooked up my laptop to my desktop rig, and have basically abandoned my desktop as a machine that I will interact with directly. (I’m a server guy, so it’ll get used.) This means I’m using xrandr with StumpWM (pretty swank,) and I must say that I love it. While I adore my laptop, and I really like only having to manage one computer, having this “desktop” setup makes it really perfect: I’m often most comfortable in desk chairs and I like the bigger screen and a real keyboard. So it works well. At some point I’ll write up a more thorough account of how I hacked this together.

Ok, this document has been open on my desktop for a few day and I don’t have anything new to add to it, so lets call it done.

Onward and Upward!

Putting the Wires in the Cloud

Thu, 04 Feb 2010 00:00:00 +0000

I’m thinking of canceling my home data connectivity and going with a 3G/4G wireless data connection from Sprint.

Here’s the argument for it:

I’m not home very much. I work a lot (and there is plenty of internet there), and I spend about two thirds of my weekends away from home. This is something that I expect will become more--rather than less--intense as time goes on. It doesn’t make sense to pay for a full Internet connection here that I barely use.
My bandwidth utilization is, I think, relatively low. I’ve turned on some monitoring tools, so I’ll know a bit more later, but in general, most of my actual use of the data connection is in keeping an SSH connection with my server alive. I download email, refresh a few websites more obsessively than I’d like (but I’m getting better with that), and that’s sort of it. I’ve also started running a reverse proxy because that makes some measure of sense.
I find it difficult to use the data package on my cellphone. The fact that I get notified of all important emails on my phone, has disincentivized me from actually attending to my email in a useful way, and other than the occasional use of googlemaps (and I really should get an actual GPS to replace that…) If I get the right Wireless modem, however, it would be quasi-feasible to pipe my phone through the wireless Internet connection, so this might be a useful clarification.

The arguments against it are typical:

The technology isn’t terribly mature, or particularly well deployed.
Metered bandwidth is undesirable.
Sprint sucks, or has in my experience, and the other providers are worse.

The questions that remain in my mind are:

How well do these services work in moving vehicles? Cars? Trains?
How much bandwidth do I actually use?
Is this practical?

Feedback is, as always, very much welcomed here. I’m not in a huge rush to act, but I think it makes sense to feel things out. It also, I think posses an interesting question about how I (and we) use the Internet. Is the minimalist thing I do more idealistic than actual? I know that we have a pretty hard time conceptualizing how big a gigabyte of data actually is in practical usage. Further research is, clearly, indicated.

Edit: This plan would have to rely on the fact that I might be spending a large amount of time in a city with unmetered 4G access with sprint. I’ve used a gig and a half of transfer to my laptop’s wireless interface in 5 days. I think that would coincide with when I would be doing the heaviest traffic anyway. I wonder how unlimited the unlimited is…

Decreasing Emacs Start Time

Tue, 02 Feb 2010 00:00:00 +0000

One oft made complaint about emacs is that it takes forever to start up, particularly if you’ve got a lot of packages to load it can take a few seconds for everything to start up. In a lot of respects this is an old problem, that isn’t as relevant given contemporary hardware. Between improvements to emacs, and the fact that computers these days are incredibly powerful, it’s just not a major issue.

Having said that, until recently an emacs instance took as much as 7 seconds to start up. I’ve beaten it down to under two seconds, and using emacsclient and starting emacs with “emacs --daemon” makes the start up time much more manageable.

Step One: Manage your Display Yourself

I’ve written about this before, but really even a 2 second start time feels absurd, if I had to start a new emacs session each time I needed to look into a file. “emacs --daemon” and emacsclient mean that each time you “run” emacs rather than start a new emacs instance, it just opens a new frame on the existing instance. Quicker start up times. It means you can open a bunch of buffers in one frame, settle into work on one file, and then open a second buffer and edit one of the previous files you opened. Good stuff. The quirk is that if you’ve set up your emacs files to load the configuration for your window displays late in game, the windows won’t look right. I have a file in my emacs files called gui-init.el, and it looks sort of like this:

(provide 'gui-init)

(defun tychoish-font-small () (interactive) (setq default-frame-alist ‘((font-backend . “xft”)(font . “Inconsolata-08”) (vertical-scroll-bars . 0) (menu-bar-lines . 0) (tool-bar-lines . 0) (left-fringe . 1) (right-fringe . 1) (alpha 86 84))) (tool-bar-mode -1) (scroll-bar-mode -1) )

(if (string-match “laptop” system-name) (tychoish-font-big))

Modifying, of course, the system name, and the settings to match your tastes and circumstances. The (if) statement allows you to have a couple of these -font- functions defined and then toggle between them based on which machine you load emacs on. Then in your init file (e.g. .emacs), make sure the first two lines are:

(setq load-path (cons "~/confs/emacs" load-path))
(require 'gui-init)

Establish the load path first so that emacs knows where to look for your required files, and then use the (require) sexep to load in the file. Bingo.

Package Things Yourself

We saw this above, but as much as possible avoid using the load function. When you use load emacs has to (I’m pretty sure) do a fairly expensive file system operation and then load the file and then compile and load the file. This takes time. Using the require function is not without it’s own cost, but it does save some time compared to load because it lets you take advantage of the work emacs does with the library loading. At least in my experience.

In your various .el files, insert the following statement:

(provide 'package)

And then in your .emacs, use the following statement

(require 'package)

To load it in. You’re probably already familiar with using these to configure packages that you download. Better yet, don’t require at all, but use the auto-load function. This just creates a little arrow inside of emacs that says “when this function is called, load this file, and hopefully the ‘real’ function by this name will be in there.” This lets you avoid loading packages that you don’t use frequently until you actually need them. The following example provides an auto-load for the identica-mode:

(autoload 'identica-mode "identica-mode.el" "Mode for Updating Identi.ca Microblog" t)

Byte Compile files as much as you can.

Contrary to whatever you’ve been told, emacs isn’t a text editor, as much as it is a virtual machine with a good deal of low level functions established for interacting with text and textual environments and some editing-based interfaces. But really at the core, it’s just virtual machine that interprets a quirky Lisp dialect.

The execution model is pretty simple and straightforward, particularly to people who are used to Java and Python: you load source files, emacs imports them and compiles them half way, they’re not the kind of thing that you could read on your own or would want to write, but it’s not quite machine code either. Byte-compiled files are easier for the machine to read, and quicker to process, but they’re not human intelligible. Then when you need to do something with the function that it’s byte-compiled, emacs compiles it the rest of the way into machine code and executes it. Usually this all happens too fast that we don’t really notice it.

One tried and true means of speeding up emacs load times is to byte-compile files manually so that emacs doesn’t have to do it itself when it loads. The emacs-lisp libraries are byte compiled when emacs installs itself, but your files probably aren’t. Now generally, only byte-compile files that you’re not going to be editing yourself regularly. Byte compiled files have an .elc extension, and as soon as there’s a .el file and a .elc of the same name in a directory, emacs will ignore the .el file even if there have been changes made. To byte compile an emacs-lisp file, simply type M-x to get the execute-extended-command prompt, and then run the function byte-compile (i.e. “M-x byte-compile"). Viola!

I hope these all help you all and lead to a slightly more efficient emacs experience.

Winter Break in Reality

Fri, 29 Jan 2010 00:00:00 +0000

I meant to write a more thorough overview of what I was doing with the “extra time” over the holidays. But I don’t think I had as much extra time at the end of the year as I expected to have. What follows is a brief overview what I did do, how the new year has begun and what I’ve been thinking about.

In years past the time at the end of the year was a time to catch up on lost sleep and connections that had fallen by the wayside in the recent months. I also used the time, in some years, to get a lot done: one year I knit about 10 hats. Another, I wrote about a quarter of a novel on a binge. Some years I just vegged.

This year, is different. I haven’t been in school full time for years, and I haven’t received any college credit in a year. I didn’t have significant time off of work. There’s a way in which the holidays were incredibly relaxing. I still have a bunch of friends who are in the later stages of being students, and there’s something awesome about not being a student that’s incredibly relaxing. I mean, working a regular job is not all sunshine and rainbows, but it’s pretty swell, and there’s something about the structure of regular and the mostly even routine that makes it--to my mind--have a greater potential for productivity than “the academic routine.”

In a lot of ways, while I looked forward to holiday time off, and saved up countless projects for that time off, not only did I not make “epic headway” on my projects but I came into the new year feeling sort of behind and tired. Wierd. I blame this on the holidays themselves. It’s as if the entire world slows down: everything gets more difficult for a month or as if the planet is slowly careening to toward this thing that we don’t really enjoy (if we’re being honest,) but that we pretend we really love.

And there’s no getting away from it. You can’t really opt out of the holidays: even if you’re not particularly festive, you can’t control the celebration of other people. You can’t control the fact that the same four songs play on endless repeat in public spaces, you can’t control that everyone wishes you a good holiday, you can’t control all of the federal holidays, you can’t escape tacky decorations, you just can’t escape. And after like 3 days of this, you get tired.

In previous years, the break, the chance to take time off from the big projects I’d been working on (school, applying to graduate school, etc.) was a great opportunity to get “other things done.” Now, there’s no real break from the daily grind, just modulations and finding good balance. That’s an ongoing project, and one that’s better serviced by a good routine and not a few extra days off during a stressful time of year. In any case, I’m glad to have gotten back into things, and I look forward to getting things done.

Onward an Upward!

Wikish and the Personal Public Wiki

Tue, 26 Jan 2010 00:00:00 +0000

First, an announcement. I’ve started a tychoish.com wiki. I’m calling it, appropriately enough “wikish.” You can see a brief introduction and note about my intentions there.

I’ve written a bunch here about the peculiarities of building communities and practices around “the wiki,” as I think it represents a new paradigm for thinking about collaboration and “the text.” I’m, slowly, working on building a community around the cyborg institute wiki, and that’s an ongoing (and fairly specific) project. I’ve also, in much smaller ways, done things with wikis in a couple of other situations: for some group projects I’ve been involved with, a few things for work, and so forth. Perhaps more relevantly, I also used a wiki--much like this one and the others I am responsible for--as the system I used for storing everything in my brain. From these experiences I’ve come to the following conclusion:

In any given wiki, most of the “work,” particular at the beginning, is accomplished by a very small number of contributors. Potentially only one contributor.

Critical mass is a difficult thing to manage or predict, and if you start a wiki and you want it to succeed, you have to be ready to do all of the work of getting it to critical mass, which could take a long time. Fair warning.
Wikis are incredibly unstructured. It’s easy to impose structure on a new wiki, in cases where structure will actually hinder growth and development rather than promote development. Particularly if the kind of content you hope to develop is wiki like. For personal organization tasks, wikis are often not the right answer, even if they appear to work for a long time.
Creating a page in a wiki is often better and more effective than writing an email of some length (say, more than 250 words), particularly when more than two people are involved in the correspondence.
I need another wiki like I need a hole in the head. But, I like that wikish is both public--you all can watch and contribute to what I’m working on--and focused on what I’m working on. The personal wiki, the one that was just for internal use suffered from lack of audience even an imagined audience.
I think putting the novella that I wrote in late 2007 into a wiki and working on revisions and tweaks in that context makes a great deal of sense, and I think wikish feels like the “right place” to put that work.

So that’s the plan. I’ll probably post from time to time about new things that I’m posting there, and I’m perfectly happy to have you all make pages in wikish as you want. I’ve also decided, that wikish will require OpenIDs as the only means of authentication. Just cause. See you there!

Independent Web Services

Mon, 25 Jan 2010 00:00:00 +0000

So much of the time, when we talk about network services, technological/software freedom, and this idea of “Cloud” computing, there’s a bunch of debate: “is it a good idea?” “are we giving up too much freedom?” “how does this work out economically?” “what about privacy in the cloud?” While these are important questions, without doubt, I fear that they’re too ethereal, and we end up tussling with a bunch of questions about the future and present of computing that might not be entirely worth debating (at least for the moment.)

Lets take 2 assertions, to start:

1. There are some applications--things we do with technology--that work best when these applications are running on high performance servers that have consistent connections to the Internet, that we can access regardless of where we are in the world.

2. The only way to have control over your data and computing experience is to be responsible for the administration and maintenance of these services yourself.

Huh?

I mean to say, that if we care about our autonomy, and our freedom as we use computers in the contemporary age (i.e. in the era of cloud computing), the only thing to be done is to run our own services. If the fact that Google has all of your data scares you: run your own mail server. If the fact that all of your microblogging output is on twitter, run your own status.net instance. And so forth.

If we really care about having power over our technological experiences, we must take responsibility for services on the Internet. We can say “wouldn’t it be nice if service providers weren’t such dicks with our data,” or “wouldn’t it be nice if software developers wrote networked software that respected our freedom.” And while it would be nice, these convinces don’t in and of themselves

Control over technology and autonomy in the networked context ultimately means that we as users have to:

Administer networked servers that provide us with the services that we want and need to do whatever it is that we do.
Participate in some exchange for networked services (i.e. pay for service, either in cash or by way of access to data.)

That’s hard! Computers should get easier to use not harder, right?

Leading question there, but…

Yes. One of the leading arguments for consumer-“Cloud Computing” is that by accessing computer services (software) in the browser, developers can provide a more structured and “safe” user experience. At least that’s how I understand it.

While this is a great thing in terms of making computers more accessible, no argument from me, I think we must be careful to avoid confusing of use" with technologically limiting. I fervently believe that its possible to design powerful software that is also easy to use, and I think that as often as not, a confusing technology is an opportunity to provide a teaching experience as much as it presents an opportunity to improve a given technology.

And if it comes down to it, there are situations where it doesn’t matter so much if you’re the one entering the commands into the server. It doesn’t much matter if you are the one managing the server or if you’ve hired someone to configure it for you. As I think about it, there’s probably something of a niche here for people to offer management services in a very boutique sort of style.

If we have to contract to people to do our administration for us, is that really a step in the right direction?

I think, it is. At the moment we pay for our networked computing services (i.e. gmail) by looking at google’s ads next to our mail and giving Google access to the aggregate of our mail spools so that they can mine it for whatever data they need. The other price that we pay for these services is “lock in:” once we commit to using a service it’s quite difficult to change to an alternate provider. Since these are “real costs,” it seems reasonable to expect and want to pay (money) for services that don’t have these costs. Which is where cooperative and boutique-style services make a lot of sense.

I’m not a systems administrator, I just want to do [the thing that I do] and not have to tinker with my computer. This is a lousy idea.

And that’s a lousy question.

To dig in a bit further. I don’t think that “doing the [whatever you do],” would necessarily require a lot of tinkering. It might, of course, and the chances are that we’ve all had to tinker with our technology at one point or another. In most cases tinkering is an upfront rather than ongoing cost. Ideally, the other thing that having full control over your network services you’ll be able to use have services which are more tailored to [the thing you do] than the one size fits all application provided by a third party.

Ok, so what’s the stack look like.

I’m not sure. There’s clearly a common set tasks that we currently use in the networked context. I’m not sure what the application is, exactly, but here’s a beginning of what this “application stack” looks like.

An XMPP Server like Prosody.im, with PyAIMt and other convectional IM network transports.
Some sort of Email Service: Citadel springs instantly to mind as an “all in one solution,” but some postfix+procmail+fetchmail+horde/squirrelmail seems to make some sense
A web server, either for hosting personal websites, or with some sort of authentication scheme (digest?) for sharing files with yourself. The truth is that web servers, are pretty darn lightweight and it doesn’t make sense to not install one. Having said that, people see “web hosting,” and probably often think “Well, I don’t really need web hosting,” when that’s almost beside the point.
SSH and some system for FUSE (or FUSE-like) mount points, so that they can use and store remote files.
There’s probably a host of web-based applications that would need to be installed as a matter of course: some sort of web-based RSS reader, wiki-like note taking. Bookmarking. Some sort of notification service, Etc.
[your suggestion here.]

Bash Loops

Thu, 21 Jan 2010 00:00:00 +0000

I was talking with bear probably two years ago, about programing and how I’m not really a programmer, but I understand what’s going on when programmers talk, and that any time I got close to code, I sort of kludged things together until it worked. This was probably long enough ago, that I was just on the cusp of getting into using Linux full time and being a command line guru.

Of shell scripting, he said something that left something of an impression on me. Something like, “The great thing about the shell, is once you figure out how to do something you never have to figure out how to do it again because you just make it into a script and run it again when you need to.”

Which, now seems incredibly straightforward, but it blew my mind at the time. The best thing, I think, about using computers in the way that I now tend to, is that any time I run across a task that is in anyway repetitive I can save it as a macro (in a non technical sense of the word macro) and then call it back in the future. Less typing, less reading over help text, more doing things that matter.

One thing that got me for a while, was the “loop” in bash. I had a hell of a time making them work. And then a few weekends a go I had a task that required a loop, and I wrote one on the command line, and it worked on the first time through. Maybe I’ve learned something after all. For those of you who want to learn how to build a loop in shell scripting, lets take the following form:

for [variable] in [command]; do

   [command using] $[variable];

done

Typically these are all mashed up onto one line, which can be confusing. Conventionally [variable] is just the letter i, for “item.” Note that the semi colons are crucial, and I think the bacticks are as well (I’d not leave them out,) but they might not be required.

So the loop I wrote. I noticed that there were a number of attempted SSH logins against my server, and while these sorts of SSH probes aren’t a huge risk… better to not risk it. So I wanted to add rules to the firewall to block these IP addresses. Here’s what I came up with:

for i in
  `sudo egrep 'Invalid user.*([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}' /var/log/auth.log -o | \
  egrep '([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}' -o | sort | uniq`;

do

   `sudo iptables -I INPUT -s $i -j DROP`;

done

Basically, search the /var/log/auth.log for invalid login attempts, and return only the string captured by the regex. Send this to another egrep command which strips this down to the IP address. Then put the IP addresses in order, and throw out duplicates. Every item in the resulting list is then added to an iptables rule that blocks access. Done. QED.

It’s inefficient, sure, but not that inefficient. And it works. Mostly this just cleans up logs, and I suppose using something like fail2ban would work just as well, but I’m not sure what kind of added security benefit that would offer, and besides it wouldn’t make me feel quite so smart.

I hope this is helpful for you all.

git magic

Wed, 20 Jan 2010 00:00:00 +0000

The following, mostly accurate conversation (apologies for any liberties) should be a parable for the use of the git version control system: As I was about to leave work the other day…

tycho: I pushed today’s work to our repository, have at, I’m headed out.

Coworker A: Awesome. I did too. (pause) wait. It’s screwed up. I deleted a file I didn’t mean to. (pastes link to diff into chatroom)

tycho: Oh, that’s easy to fix. You can reset back to before the file, add all the changes that are in you’re repository, except the deletion of the file, commit, and then “git reset --hard” and then publish that.

Coworker A: But your changes…

(as an aside, the original solution should still work, I think)

tycho: Oh. Hrm. Right. Well… Rebase to remove the bad commit and then add the file in question back on top of my changes.

Coworker A: Wait, what?

tycho: (looks at clook). Shit, I’ll do it. (turns to Coworker P), have you pulled recently?

Coworker P: Nope I’ll do that no--

tycho: Dont’t!

Coworker P: Alright then!

tycho: (mumbles and works)

At this juncture, I pull out crazy git commands and rebase the repository, back a few commits to pull out a single changeset. And then recommit the file with the changes worth saving (which I had copied into ~/ before beginning this operation.)

One thing I’ve learned about using git rebase is that you always have to go back a commit or two before I think I need to, pick out the hash for the last good commit. Also when using “git rebase -i” I find that the commits are listed in the reverse order that I want them to be listed in.

Another great hint: Issue the following command if you’re an emacs user and you don’t want git to open rebase editing sessions in vim.

git config --global core.editor "emacsclient -t -a emacs -NOW"

The one issue here is that I had to rewrite the history of an already published series of changes. This is why I didn’t want P to pull. When I was done, and the state of my repository was as it should have been, my next push (predictably failed), as it needed to be a “git push -f”, which is something of a scary operation. It worked out, and when everyone pulled the next time everything was fine: I knew it would be for P because their local repository never knew about the first iteration of the history. I was less sure if A’s would adjust so seamlessly, but it did.

tycho: Ok, done. Pull A.

Coworker A: All better! I have no clue what happened.

tycho: It’s cool, don’t sweat it. There’s very little that isn’t fixable. As long as you don’t hard reset changes, and don’t do crazy rebasing stuff, you should be ok.

Coworker A: Like what you just did?

tycho: Pretty much.

Here are the lessons:

“git push” and “git pull” would seem like parallel operations but they’re not. Pull with abandon, it never hurts to pull. But if lots of people are pulling from the same repository, and you push a change that you don’t mean to push, it’s really hard to take that change back in a logical and productive way. So push with caution.
Rebasing is a tool that has great power shouldn’t be feared even though theoretically you can screw stuff up with it. The git way says “commit your changes early and often,” is great, but it can be sort of anti-social, as individual commits become sort of meaningless, and change logs can get hard to manage. Rebasing, though scary, can make it possible to both commit as often as you need to, and then rebase to be presentable.
Fear forced pushes.
Everything in git can be changed, so play with things, and then only publish changes when the repository is in a good working state.

Onward!

Beyond Lists in Org Mode

Tue, 19 Jan 2010 00:00:00 +0000

I’ve written about this problem in org-mode, the emacs outlining and organization tool that I us, before, but I’m readdressing it for my benefit as well as yours.

Org mode is an outlining tool, fundamentally. It provides a nice interface for editing and manipulating information arranged in an outline format. Additionally, and this is the part that everyone is drawn to, it makes it very easy to mark and treat arbitrary items in the outline as “actionable,” or todo items in need of done. The brilliance of org-mode, I think, is the fact that you spend all your time working on building useful outlines and then it has a tool which takes all this information and compiles it into a useful todo list. How awesome is that. For more information on org-mode, including good demonstrations, check out this video.

The problem is a common and recurring one for me. I basically live in the agenda mode--that compiled list of todo items--and I don’t so much use org-mode for making outlines. Truth is, I have a “Tasks” heading in most org files, and I use the automatic capture option (e.g. org-remember) to stuff little notes into the files, and beyond that, I mostly don’t interact with the outlines themselves.

This isn’t a bad thing, I suppose, but it means that org-mode can’t really help you, and you’ve short-circuted the ability of org-mode to improve the organization. Under ideal circumstances, org allows you to embed and extract todo lists from the recorded record of your thought process. If you’re not actively maintaining your thoughts in your org-mode files, it’s just another todo list. That isn’t without merit, but it doesn’t allow the creation of tasks and the flow of a project to spring organically from your thoughts about the project, which is the strength of org mode.

Intermission: I took a break from writing this post to go and reorganize my org files. What follows are a list of “things I’ve been doing wrong” and “things I hope to improve.”

I don’t think I had enough org-files. There are lots of approaches to organizing information in org: one giant file, lots of small files for individual projects, a few mid to large files for each “sphere” of your life.

Initially I took the “medium sized files for major ongoing projects.” I had a writing file, and a work file, and a writing file, and files for the fiction projects that I’m working on, and a notes file, and a clippings file, and so forth. Say about 8-10 files. It works, but I think the thing it did was it caused me to use the org-remember functions to just dump things in a “tasks” heading, and then work from the agenda buffer, and not ever really have to touch the files themselves. Org files need to be specific enough that you would want to keep them open in another window while you’re working on a project. I think the point where you know you’ve gone too far is when the first level headings start to replicate organization that might better be handled by the file-system.
Use the scheduling and deadline functions to filter the todo list into something that is workable. It’s easy to just look at the task list and say “oh no, I don’t want to work on this task right now because it depends on too many things that aren’t done, and there are other things that I could work on.” Scheduling an item, if not setting a deadline, forces me (at least) to think practically about the scope of a given project, what kind of time I’ll have to work on it, and what other tasks depend upon it.
When you’re using org to manage huge blocks of text--or any system, really--it can be difficult if you have multiple hierarchies and depths of greater than two or three. It just gets hard to manage and keep track of things and figure out where things are, particularly given how useful and prevalent search tools are.

Having said that, When you’re organizing tasks in org, that limitation, one that I find myself imposing upon myself doesn’t really work terribly well, and leads to files that might actually be more difficult to read and to work out of.
I started using the “org-archive-subree” function for archiving content when I was through with parts of the outline, This sends the archive to a separate file and while it works, I find it… less than useful. I’ve since discovered “org-archive-to-archive-sibling” which is a great deal of awesome, and I recommend using it exclusively.
Write content in org mode when possible. Though some people (hi Matt!) are keen on using org as a publication system, I’m not sure if this is the right answer, but I do think that its good during very creative phases of projects to do the work in org, mostly as I think it facilitates focusing on the current problem (through collapsing of the tree to show you just what you’re working on,) and also for working non-linearly as you can leave yourself TODO items for later action.

At the same time, if you tend to maintain org files that contain planning for more than one project, I find it cumbersome to also draft in these files. So I think “keep smaller very focused org files, and maybe do drafting in them if appropriate.”

That’s a start at least. I’ve made these changes--which are really quite subtle--and I like the way it feels, but we’ll have to see how things shake down in a few weeks. As much as I want to avoid tinkering with things--because tinkering isn’t the same as getting things done--I really do find it helpful to review processes from time to time and make sure that I’m really working as effectively as I can.

The Two Year Sweater

Mon, 18 Jan 2010 00:00:00 +0000

I finished knitting a sweater. I posted pictures of this to twitter, so I guess in a way, I’ve scooped myself.

But I did it. This sweater has a special story…

I think it’s worth mentioning that--if there are any knitters left reading this that I’m sort of haphazardly working on a collection of knitting patterns and stories/essays. Patterns in the sense that you could get a bunch of yarn and some needles and read and end up with a sweater that probably looks like the one I have. But not patterns in the sense that I’m not writing instructions for knitting, but rather stories about my life and the creative process that embed the instructions for knitting sweater. This post isn’t exactly one of those, thought I do hope to get to the sweater in question at some point soon.

But then don’t they all.

I initially called this sweater “Latvian Dreams” and the idea was that I’d blog about the sweater as I knitted it as a sort of adventuresome knit along.

It turned into a nightmare.

And I never did really blog about it in the way that I might have liked to….

I was working in a yarn store at the time, and it seemed like a good idea at the time. I was knitting a lot. I was pretty serious into blogging at this point, and it seemed like a good idea.

It wasn’t.

In an effort to create a pattern that would be easy for other people to pickup, particularly people who might not have been particularly adept at the kind of stranded two color knitting I find so entrancing, the patterns I chose were almost too simple, and I never really got into them.

It’s not, I suppose terribly fair to say that the patterns were too simple…

The patterns were all symmetrical, both top to bottom and side to side. I chose three different patterns, arranged things to be reminiscent of an Aran sweater, and they even synced up with each-other so that there was a regular repeat that I thought would help people memorize the stitches.

And the whole thing was sort of like pulling teeth.

I mean it all worked out in the end, so I suppose I can’t complain about anything but the time that it took to make the blasted thing. It’s a good sweater. Even though I haven’t blocked it yet, I’m struck by how well it works. The yarn is fine--hence part of the scope of the project--and it fits really well. I must know a thing or two about how to knit sweaters.

And somehow it’s a bit bittersweet.

In a lot of ways this is the kind of sweater that I don’t really have the attention or focus to be able to even ponder making now. Too much attention even in the planning, not to mention the scope of the carry through. It’s not that I’ve lost the technical ability to knit a sweater like this, it’s as if my life has moved on, and it took those kinds of sweaters with it, and that’s sort of hard.

No lies, I’m glad to be done. For sure.

Beyond SQL and Database Technology

Fri, 15 Jan 2010 00:00:00 +0000

People have been thinking about databases recently. Even I’ve been thinking about databases, and I’m not particularly prone to thinking about databases. It’s fair given the ongoing drama of the Oracle/Sun, and even mainstream press of the NoSQL Movement. I’d like to take a step back and think a bit more honestly and holistically about the database application, aboth this “NoSQL” phenomena, and about the evolving role of relational database management systems in our technology “ecosystems.”

(Seriously folks this is what I think about for fun in my free time.)

I’ve been milling over the notion that databases, like MySQL and PostgreSQL and Oracle’s RDBM products, are not particularly “Unix-like.” Sure they run on Unix systems, and look and feel like Unix applications, but the niche fulfill--providing quick access to structured data with a specialized query language, doesn’t jive with the Unix philosophies: small specialized tools for precise tasks. “Plain text” as lingua franca of system tools, and so forth.

Databases solve a problem. Indeed they solve a problem in a very functional and workable manner. I don’t want to suggest that the relational database model is somehow broken; however, I would like to suggest that industrial strength database systems are over utilized, and have become the go-to solution for storing and interacting with data of any kind, even in cases where they’re not a good fit for the job at hand.

I’m not the first person to suggest this, not by a long shot. The NoSQL “movement,” addresses this issue from a couple different direction. It’s true that NoSQL refers to a collection of practices and approaches related to providing systems for storing data that goes above and beyond the type and model of a database system. In the end NoSQL is about addressing the scaling problem: what happens when we have so much data that it can’t easily fit in one database system, or in situations where centralized model is untmaintable for any number of reasons. I think NoSQL is also relevant as we think about storing data that doesn’t easily fit into RDBMs’es: I’ve seen a lot of very poorly architected database systems, that suffer from a “square peg in round hole” problem.

Indeed, as we try and put all of our data in these RDBMs systems, particularly data that doesn’t fit very well, these databases loose their ability to scale. The complex logic required to pull more complex data back out of a database and reassemble it for use and analysis is computationally expensive and doesn’t scale particularly well.

But let’s focus for a moment on the scaling question, apart from the data modeling and storage question. The real problem at the core of the scaling question is: we need a way, a thing, that allows multiple systems to access a shared data store in a reliable and consistent manner.

The ongoing work around clustered file systems seems to address this issue from a much different direction, and perhaps a more interesting perspective. Beyond a certain point--and its a fuzzy point--database systems basically become file system replacements. So rather than work on making databases more like file systems, the thought is (I assume) lets make file systems a bit more “database like.” Like I said, I don’t know a lot about the ins-and-outs of clustered file systems, but I think, in addition to worrying and thinking the future of current database systems, we need to also think about the future of these very scalable and clustered manner.

I’m not sure what the next-generation data storage technology really looks like, the NoSQL stuff is a step in the right direction, but I’m not sure if it’s a large enough step in a lot of ways, as its focus is a bit narrow. To be honest, I’m not incredibly familiar with the work that’s going on in the clustered file system space. Nonetheless, I think it’s important to not just think about the future of the relational database platforms as such, but the model and the underlying problems that these kinds of data storage methods address, and to think about other possible ways of addressing the original issues.

Time Management

Wed, 13 Jan 2010 00:00:00 +0000

I’ve not written here about time management and productively very much recently. I’m a pretty busy guy, I work a lot, I live an hour away (minimum) from my social life, and I have a lot of things simmering in various stages. While I might accept the challenge that I’m spreading myself a bit too thin (I’m working on it!) I feel like the largest challenging isn’t that my attention is too divided, or even that I don’t have enough time to do the things I want to do.

Rather, I think my biggest challenge at the moment is that I’m not particularly good at using very short blocks of time to get things done. The twenty minutes of free time I have in the morning before work, the time after work when I’m too tired of looking at words to write, but not tired enough to go to bed. One of the great things about having lots of projects is that they’re all in different stages and require different kinds attention.

As a result, I’m taking the following strategies in an effort to use time a bit more effectively:

1. Stub out projects during binges, fill in the gaps in the interstitial time.

One of the problems with writing in short little bits and pieces throughout the week, is that writing is often a game of momentum, and it’s hard to really build up speed and absorb yourself in a project, however big or small in a few moments. In most cases, the hardest thing to do in writing is figure out “Okay, so what do I need to write here.” Given this, it’s incredibly easy (at least for me) to become enchanted with the successful binge and the ability to bury myself in a long writing session for hours on end; because that tends to work well, I’m prone to just not trying to write in the interstitial moments.

My approach, recently, and one that I need to pay a bit more attention to maintaining, has been to use binges to start projects, to do a lot of free writing and note taking, and then with things mostly sketched out and “stubbed out” (to borrow a term from the wiki world), it’s easier to write things during the week.

Work on keeping the “to do” list more populated.

I suppose this is really an extension of the above, but I sometimes find myself avoiding adding items to my todo list if I’m close to “clearing the decks” (OCD much?) and I sometimes fall out of the habit of really using my todo list as a method for planning my day and week out. Todo lists have three main purposes that I can see: first they help with remembering things that you might otherwise forget. That’s not something I struggle with in any major way. The second is to do some organizational work up front so that big and hugely imposing tasks seem much more manageable when you sit down to work. Finally, they should all but moot the question “what should I work on now?”

In any case, a todo list is completely useless when left unmaintained, and underpopulated. There’s always something that needs doing, and there’s often time to do something. Todo take the thought out of figuring what to do in those spaces, and they can’t really do that when they’re not kept up to date.

3. Get away from the computer when I’ve lost the ability to concentrate.

When I’m tired or bored (or both) and don’t think I could muster the ability concentrate on a sentence, I rarely muster the courage to get up and do something else. Instead, I usually tab into a web browser and doddle away the evening doing something like refreshing Facebook endlessly, or some-such. While there’s nothing wrong with a little bit of harmless perusal of the Internet, it’s too easy to get sucked in and then not get other stuff done. I think of this as the “cut your looses strategy.” I’m not terribly good at it most of the time, but when it works I’m pleased.

Read more, particularly when I’ve “run out of words."

At the end of a day, I sit down on the couch with my laptop, and I find whatever emacs buffer open that I’d been hacking or writing away on in the morning (or the previous evening), and I think “Dear god, I couldn’t possibly write anything more,” which is a fair feeling: I stare at emacs buffers and hack away on words all day as it is, coming home to do more of this, even if the topics are a bit different, is sometimes difficult. I enjoy writing a bunch and find it to be a very rewarding experience so this isn’t always a problem, but when it is, it is.

My goal is to avoid waisting time because I’m bored or tired and using this time to read instead. I tend to find reading to be refreshing and I feel like I don’t have enough time to read as it is, so this solves a few problems. I think if we look honestly at our days and our goals, most of us might be able to find a way to get the more things done that we want to get done, with such a strategy.

We’ll see how that works.

Starting a Collaboration

Tue, 12 Jan 2010 00:00:00 +0000

Alternate Title: “How to start a collaborative writing project or die trying,”

Step 1: Lock yourself in your office, fire up Emacs, and write an initial draft from beginning to end yourself.

Step 2: Post it on the Internet.

Step 3: Encourage contributions and hands on feedback from your collaborators. Have a piece of cake.

You may thing that I’m kidding, but it’s true. I think there’s a misconception that the way to write something with other people follows a path that might look like: having a meeting to establish the common goals and an outline of what needs to get said, and then another meeting to divide up who is going to write want, and then people go back and write their little parts, and then you mash them all up and everyone rewrites their part till it meshes with the other parts, and then you pray it says what you need it to say, and doesn’t need further revising--except it sort of dose, so you repeat the whole process over again, to revise the text, except with editing instead of writing. And because each stage requires endless conversation, when you don’t have the benefit of face-to-face meetings, things can take a long time: so long, in fact, that most people will have probably lost interest long before something has been written. The short of it is that this method, though very democratic and open seeming, isn’t.

I think there’s a fear, that when a single person puts a lot of individual energy into a text (or any kind of project, really,) and doesn’t consult with collaborators at every turn that it somehow becomes not a collaboration. This is emphatically not true. There is significant difference between endless group process and the collection of meaningful feedback; a real distinction between a text created with a process that involves many people, and a text that many people can agree represents their interest, purposes, and needs.

I think, though I’m not certain, that one could replace the words “writing” and “text” in the above, with “programing” and “code” but I don’t know for sure.

At work, I have a moto: “you can’t edit it if it doesn’t exist yet.” The more interesting thing, I think, in every context is when you go off into your own office, fire up the emacs, write something, and then say “so how does this look?” People sometimes say, “nice, but you used ‘setup’ as a verb in the third paragraph,” or “ok, but you left out a section about flux capacitors in section two, and I think that’s crucial for understanding most of section three,” but these are problems that are fairly easily addressed.

Now it could be the case that I’m just that awesome (unlikely), but I think it boils down to the fact that most people don’t understand how to make texts. I also think that a lot of “group process,” can be obsoleted by an individual who can produce something, and has a good sense of the group’s desires, and who knows how to check in with various group members at the right moments. While these skills can be listed quite effectively, and it’s true that there is no rocket science involved: some things are easier said than done.

Not every collaboration works, and there are a lot of variables at play in any situation where a group of people must come together to make something, but in nearly every situation beginning with “hey, I want make something with you, look at this draft,” is better than “I was thinking about making something with you but I wanted to get your feedback first.”

Just sayin'…

End User RSS

Mon, 11 Jan 2010 00:00:00 +0000

I’m very close to declaring feed reader bankruptcy. And not just simple “I don’t think I’ll ever catch up with my backlog,” but rather that I’ll pull out of the whole RSS reading game all together. Needless to say, because of the ultimate subject matter--information collection and utilization and cultural participation on the Internet--and my own personal interests and tendencies this has provided some thinking… Here goes nothing:

Problems With RSS

Web 2.0 in a lot of ways introduced the world to ubiquitous RSS. There were now feeds for everything. Awesome right?

I suppose.

My leading problem with RSS is probably a lack of good applications to read RSS with. It’s not that there aren’t some good applications for RSS, its that RSS is too general of a format, and there are too many different kinds of feeds, and so we get these generic applications that simply take the chronology of RSS items from a number of different feeds and present them as if they were emails or one giant feed, with some basic interface niceties. RSS readers, at the moment, make it easier to consume media in a straightforward manner without unnecessary mode switching, and although RSS is accessed by way of a technological “pull,” the user experience is essentially “push.” The problem then, is that feed reading applications don’t offer a real benefit to their users beyond a little bit of added efficiency.

Coming up a close second, is the fact that the publishers of RSS sometimes have silly ideas about user behaviors with regards to RSS. For instance there’s some delusion that if you truncate the content of posts in RSS feeds, people will click on links and visit your site, and generate add revenue. Which is comical. I’m much more likely to stop reading a feed if full text isn’t available than I am to click through to the site. This is probably the biggest single problem with that I see with RSS publication. In general, I think publishers should care as much about the presentation of their content in their feed as they do about the presentation of content on their website. While it’s true that it’s “easier” to get a good looking feed than it is to get a good looking website, attending to the feed is important.

The Solution

Web 2.0 has allowed (and expected) us to have RSS feeds for nearly everything on our sites. Certainly there are so many more rss feeds than anyone really cares to read. More than anything this has emphasized the way that RSS has become the “stealth data format of the web,” and I think it’s pretty clear, that for all its warts, RSS is not a format that normal people are really meant to interact with.

Indeed, in a lot of ways the success of Facebook and Twitter have been as a result of the failure of RSS-ecosystem software to present content to us in a coherent and usable way.

Personally, I still have a Google Reader account, but I’m trying to cull my collection of feeds and wean myself from consuming all feeds in one massive stew. I’ve been using notifixlite for any feed where I’m interested in getting the results in very-near-real time. Google alerts, microblogging feeds, etc.

I’m using the planet function in ikiwiki, particularly in the cyborg institute wiki as a means of reading collection of feeds. This isn’t a lot better than the conventional feed reader, but it might be a start. I’m looking at plagger for the next step.

I hope the next “thing” in this space are some feed readers that add intelligence to the process of presenting the news. “Intelligent” features might include:

Noticing the order you read feeds/items and attempting to present items to you in that order.
Removing duplicate, or nearly duplicate items from presentation.
Integrate--as appropriate--with the other ways that you typically consume information: reading email and instant messaging (in my case.)
Provide notifications for new content in an intelligent sort of way. I don’t need an instant message every time a flickr tag that I’m interested in watching updates, but it might be nice if I could set these notifications up on a per-folder or per-feed manner. Better yet, the feed reader might be able to figure this out.
Integrate with feedback mechanisms in a clear and coherent way. Both via commenting systems (so integration with something like Disqus might be nice, or the ability auto-fill a comment form), and via email.

It’d be a start at any rate. I look forward to thinking about this more with you in any case. How do you read RSS? What do you wish your feed reader would do that it doesn’t?

The Blog, Next in Lisp

Thu, 07 Jan 2010 00:00:00 +0000

Here’s a crazy idea: in addition to posting an RSS feed, say I start posting the content of the blog as Common Lisp code. Not, to replace any format that I currently publish in, but as an additional output option. Entries might look something like this:

(tychoish:blog-post
   (tychoish:meta-data
      :title "The Blog, Next in Lisp"
      :author "tycho garen"
      :pubtime #'(format () time-t)
      (tychoish:blog-tags '(lisp cyborg crazy))
      (tychoish:archive-collection '(programing)))
  (tychoish:blog-content (markdown)
    "Here's a crazy idea: in addition to posting an RSS feed, say I start
    posting the content of the blog as Common Lisp code. Not, to replace
    any format that I currently publish in, but as an additional output
    option. Entries might look something like this: [...]"))

That’s pretty. In a lispy story of way. I’m not sure that it’s actually correct, and it makes calls to functions that don’t exist, of course. But I hope you can get the gist enough to see where I’m going with this, and maybe enough to correct my newbish mistakes.

By my count there needs to be functions for: blog-post, meta-data, blog-tags, blog-content, and markdown. And of course it’s missing some notion of what these functions might do. I’m not terribly sure what they could do, build a better indexing system for the site (lord knows I need it), or more easily create a Lisp-based content reader/browser thatls like a feed reader but more in some way that I haven’t envisioned.

In a lot of ways, this isn’t any different from RSS. And it is RSS, basically, except you don’t have to parse it into some format that your programing language can understand, (assuming you’re programing with Lisp, of course, but you are, aren’t you?) because it is your programing language. At least in my mind this has a lot in common with the Sygn Project in that both projects focus on providing some sort of loose standard that allow us to share and use data openly and freely, using formats that are easy (enough) to construct by hand, are human readable, and easy to process and use programatically.

In any case, it shouldn’t be terribly hard to generate this format, the question is: does seeing the data like this present possiblities to anyone? And, while we’re at it, if anyone wants to help define some of the more basic functions, that might be awesome. I look forward to hearing from you all?

Corporate Government

Wed, 06 Jan 2010 00:00:00 +0000

I was talking to someone, probably a coworker, and I was saying something about a “government,” except I slipped and said “corporation.” An easy mistake, if not a common one, perhaps, and certainly somewhat telling. In this post I want to discuss a number of ideas that have been lingering about in my thoughts regarding the role of corporations on culture and technology, and the role of corporate structures on the conveyance of cultural values.

In a lot of ways this post is the successor to my posts on: Martian Economics and Transformational Economics

Maybe that’s a bit much for one blog post. In brief:

Corporations and Governments

We can think of both corporations and governments fall into the larger category of “formal social institutions,” or “formal collective institutions.” They are both, at least in theory, productive beyond the ability of a single individual, and both dominate the shape and course of our lives to significant degrees.

Corporate Structure and Cultural Transmission

How corporations, and governments more obviously, are structured and behave is--I would argue--a means of creating and transmitting cultural values.

Praxis and IBM: Autonomy and Bottom Up Organization

I use these two examples--one fictional, one actual--as possible illustrations of a different sorts of ways of thinking about corporate organization. Both of these examples represent large institutions engaged in diverse operations, that are organized (I think) with a great number of quasi-autonomous operations and divisions, which contribute to common project but have the freedom to operate independently and encapsulated ways. This strikes me as a unique modality.

This leads to a lot of questions, and not very many good answers. I suppose that’s not intrinsically a bad thing.

One of the biggest problems with corporations as far as I’m concerned is that by virtue of their fiduciary responsibility they have no obligation to operate in a sustainable manner or in the common interests of either their employees or the public.

Can the potentially harmful potentials inherent in corporate person-hood be offset by certain types of organization?
Is there a better way to manage and organize our political society that balances the power of governments, corporations, that is sustainable and efficient?
We talk, and think, a lot about how the Internet affects how people use technology, and how the Internet creates new possibilities for business. How does the Internet change the way we organize our work lives? Has technology made smaller corporate operations more sustainable and able to compete?
Are the alternatives loose and autonomous-cells in corporate organizations that might be able to address the concerns regarding efficiency and sustainability?

And so forth…

Common Lisp, Practically

Tue, 05 Jan 2010 00:00:00 +0000

So in the emacs session running on my laptop (13 days plus) I have a number of buffers open, a great many of which include the entirety of Practical Common Lisp thanks to emacs-w3m, which I’ve been working through slowly. I’ve written here about how I find Lisp to be intriguing and grok-able in a way that other programing languages aren’t really.

My exposure to lisp isn’t great. I hack about with my emacs code, and I do a little bit of tweaking with the window manager that I use (written in common lisp), StumpWM, but other than that I don’t actually have much experience. What follows are a series of reflections that I have with regards to lisp:

Although there’s a lot of really amazing capabilities in Common Lisp, and a lot of open source energies behind Lisp… Lisp isn’t flourishing.

This shouldn’t be a great surprise to anyone, lisp is sort of the epitome of “Programing Languages that don’t get enough respect.” Having said that there are a lot of lisp projects that aren’t really well maintained at all. Even things that would just be standard and maintained for other languages (various common libraries and the like) haven’t been touched in a few years. While it’s not a huge worry, it does make it a bit worrying. Having said that, I don’t think lisp is ever really going to go anywhere, and Common Lisp seems like a pretty darn good spec. But I don’t have any real exposure to Scheme, and Arc isn’t really real yet, I guess.

Lisp works funny, particularly for people who only have a passing familiarity with programming.

We’re used to programming languages that either pass the source code through an interpreter (e.g. Python, Ruby, PHP, Perl, and I suppose Java and C#) compile into some sort of intermediate bytecode and then run that code on a virtual machine, then output stuff; conversely there are languages which compile down to some sort of native binary and then execute directly on the hardware. Examples of this second class of languages include: C, C++, and Haskell. Sorry if my examples or descriptions of the execution model aren’t particularly precise.

When you run lisp code, you define stuff and load it into the memory of a lisp process, and then stuff happens as the program runs. It’s compiled to native code (I’m pretty sure at least,) but there aren’t binaries, in the conventional sense. To get a “binary,” you have to dump the memory of the program, and pretty much the entire lisp process into a blob. So the base size for executables is way bigger than one might expect. I’ve also had some success at running scripts with sbcl shebangs from the terminal. That’s pretty nifty, not that I’ve really done very much of that, but its nice to know that it’s possible.

Web programing in Lisp. I’m not so sure about that.

So you might see lisp code, and think: “So. Many. God. Damn. Parentheses.” and you’d be right. But even well formatted HTML is considerably less “human readable” than Lisp, and I don’t think there’s a lot of room for debate there. But when you think about it, Lisp actually makes a fair amount of sense for the web.

I’ve actually done a little bit of poking around and from what I can see, the actual architecture and deployments of lisp aren’t terribly bad. There are Apache modules that will pass requests back to a single lisp process (mod_lisp similar to how fastcgi works,) and there’s always the option of running performance CL specific web-application servers and just proxying requests to those servers from Apache. Lisp is, or can be, pretty damn fast by contemporary standards, and although there’s a lot of under-maintained lisp infrastructure, the basics are covered, including database connectors and java script facilities which might not be incredibly enticing, but all the parts are there.

I mean, having said that, I’m not a web developer, or really much of a developer in general, but it’s fun to think about, and even if I only use Lisp to hack on various things here and there, I’m still learning a bunch from the book and that seems more than worthwhile.

Another One Already

Mon, 04 Jan 2010 00:00:00 +0000

Alternate titles for this post include, “Depending on When You Start Counting,” and “Happy New Year.”

I must confess that I don’t do holiday’s very well. It’s not that I’m a huge curmudgeon (though I probably am) or that I don’t like celebrations (though I don’t much.) More, I think it’s that I’m mostly a homebody, and given the option, will in most cases, choose to spend any given evening at home writing and hanging out with the cats. Furthermore, I’m generally of the opinion that formal excuses are not needed for spending time with your friends and family. Nevertheless there is a certain sort of cultural momentum around holidays like New Years, and its hard to avoid them entirely.

For many years, my annual cycle has largely been on the academic calendar. Indeed for a few years after I graduated formally, I still took a few classes, and was wrapped up in applying for graduate school and enough of my friends were still in school that I seemed to stay on the academic year. This year with, I noted the beginning of the academic year, mostly because for the first time, really the first time in a long time I wasn’t in school, I wasn’t trying to be in school, and that wasn’t a bad thing. I still have a lot of academic habits: the impulse to review and summarize my work about every four months, the way I stricture and organize my work is very reminiscent of a sort of academic way of looking at things. Shrug.

But it’s a new year, depending on when you start counting, and given what a ride 2009 was, it seems like a bit of reflection is in order. The most significant thing was the fact that I took a job half way across the country and moved in late June. This has lead to a number of interesting developments: I met a number of people in “real life” who had previously been on-line friends, I’ve learned a lot about my skills and abilities and myself as a writer, I’ve developed a circle of friends that delight me.

This isn’t to say that it was a stellar year. I spent six months not working (really,) and a lot of time unsure about what I was going to do for work, let alone “my career,” all my friends were in graduate school and most of them weren’t anywhere near where I was, and so forth.

But I kept, hacking away at various projects, kept thinking and writing about new things, and did my best to seize opportunities when they arose. And somehow it all worked out. In retrospect it’s all very weird, to think how monumental this year has been, and the ways that I’ve really pushed myself to do things that don’t seem very “me” like. In the end I’m pleased with where I am and where I’ve come.

But perhaps, more significantly, I’m excited to what the next year holds.

As it should be, I suppose. I hope you all are doing well in this regard and I look forward to talking with you throughout the year to come.

The Care and Feeding of a Writer

Thu, 31 Dec 2009 00:00:00 +0000

“Being a writer,” is a strange thing. I’ve written about this before, but there’s more here. I had a conversation with Caroline about what it means to be a writer and I thought some of the things we came up with were pretty good. I hope I’ve done a sufficient job of capturing what we talked about.

One of the things that we emphatically agreed upon was the fact that writing, despite being something that we’re all taught to do in school, as part of being “an educated person” is not something that everyone is particularly capable of.¹ Not that that’s a bad thing. Having said that, because everyone reads at least in some capacity, and most people know how to write, when you tell someone “I’m a writer,” or you say something like “I’ve been working on this novel for about 16 months,” you’ll get funny looks. Guaranteed.

There are, I think, a few major issues at play here:

1. Because reading is an automatic facility for most people, I’m convinced that laypeople are very prone to misunderstanding the amount of work that any given text represents.

I might even go so far as to say, writing is not something that people automatically think of as “work.” I’m going to sit down and spend some time talking to imaginary people in my head who live on a spaceship a thousand years hence, in an attempt and hope that I can productively explore the post-colonial condition in a new and different way, doesn’t sound like it should be hard. But it is, and I think much more difficult than the kind of writing that I do professionally.

2. Because most writing education focuses on grammar and extensive reading (which are great things,) the gulf between “people who write” and “people who don’t,” is often not about writing technically solid prose or not (despite the fact that in primary and secondary education, this is the major p).² Rather, the ability to understand something (a process, event, or story) in enough detail to describe with an awareness of an audience is the real challenge.

Indeed when successful, all of the “work” of writing is entirely opaque to the reader.

3. Writing a text (an article, essay, or story, as opposed to a memo or message,) is not something that an individual can--or should be expected--to accomplish independently. You can’t tell a writer “go write this story, and when you get back we’ll print it.” Editors, from the person who says “I need you to write this,” to the people who give you feedback as you progress on a project, to the people who prepare a text for publication. You can’t do it alone: there are too many conflicting interests at play in the writing of any text that make it challenging for a single person to write alone.

I don’t bring these up to complain. Nor do I think the solution to these problems is to “just give writers a bit more respect.” Knowing these things about writing, I think we could, quite productively, change the way we educate students to write. It might also, I think lead to some productive reorganizations of how writers and editors organize themselves to get “content” produced.

After a while of talking through these ideas, the conversation produced the following gem, which I simply must share with you:

caroline: you know how everyone has a memoir now?

tycho: yeah…

caroline: you know what I don’t care about?

caroline: most people’s memoirs

Onward and Upward!

I’m aware this sentence ended in a preposition. Piss off. ↩︎
I had the longest run of “highest-possible C” grades on my papers in high school English, and while I think I might have managed to average out to a B, mostly, that always felt like an accomplishment. In a lot of ways, my failure to achieve in high school English is probably mostly responsible for the fact that I avoided writing almost entirely in college. ↩︎

Thinking Like a Web Developer

Wed, 30 Dec 2009 00:00:00 +0000

I’ve been reading a lot about web development in the last few weeks. I’m not exactly sure why. There are some interesting things going on in terms of the technology. Frameworks that provide for some interesting possibilities abound, and while I don’t know if the web is the only future for programing, it’s certainly a big part of the future of the way we interact with computers.

So what are you working on developing tycho?

A whole lot of noting. I know how the technology works. I know--more or less--how to build things for the web, and yet I’ve not said “you know what I need to build? A web app that does [this awesome thing]”

Maybe it’s because I’m unimaginative in this regard, or that I tend to think of web applications being a nearly universally wrong solution to any given problem.

I think it’s quite possible that both of these things are true. It’s also likely that when approached with a problem with technology or with data, I don’t instinctively think about how to solve it pragmatically, much less with some sort of web-based system. As I think about it might be the fact that my mind is intensely qualitative. In my psych major days I always had problems coming up with ides for non-hokey quantitative studies (Insofar as such things exist.)

In a lot of ways the questions I ask of the technology aren’t (particularly) “how can I manage this data better,” but rather how can I interact with this technology more efficiently. While I don’t think data interaction is a solved problem, I feel like I’m pretty far ahead of the game, and that the things I do to improve how I work has more to do with tweaking my system to shape the content and way that I’m working. While there’s often some little bits of code involved, it’s not the kind of thing that’s generalizable in the way that an application or web site might be.

The Imperative Tense

Most of the time, you put me in a room with programmers and tell us to talk about our work, the conversation will be really lively. Aside from the fact that I use programmers tools to write, and use a very iterative approach

One thing I notice many of my coworkers doing is saying “I’m going to write a program that’s going to do these four things, and it’s going to be written in such a way as to make these other things possible,” (insert words of awesomeness in this sentence.) And I think “Cool! I can’t wait!”

For a long time this way of talking confused me and almost put me on edge. When I have an idea for a new project, I get these images, and an interesting concept to toy with and I have little conversations in my head with the characters, and I see their world from their eyes, and it’s sort of an absurd experience and I don’t tell people about this. I mean, I might say “I got an awesome idea for a new book,” but usually not more than that. And the truth is that I get ideas for stories all the time and I know that I’ll never really be able to write most of them.

I’m okay with the way programmers plan projects, and I’m pretty happy with my own methodology. Having said that, I think the difference in the way that I think about and plan projects has a lot to do with the way I think about these things.

Onward and Upward!

Reading Habits

Tue, 29 Dec 2009 00:00:00 +0000

I bought a Kindle. I am weak.

(Note: I drafted this post early last week, and it arrived last Wednesday, and I started using it in earnest over the weekend. Nevertheless, this post is written from the perspective of my past self.)

In any case, there are a number of questions that you may be asking yourself at this juncture.

ZOMG That’s a lot of DRM that you’ve signed up for. How does that make you feel?

I’m not wild about it. I mostly view the DRMed kindle stuff as: not as a collection, but rather as a convince for reading specific texts on demand and as needed. And on those terms, I can live with it. There’s all sorts of things wrong with what I’m about to say but: DRM is most onerous if you think that the files you download are “your possessions.” Because they aren’t. When it’s just a dinky file that you have the ability to read in a highly convenient way, that’s easier to swallow. Having said that, if you’re not buying something that you get to keep the books as they are now are too expensive.

I’m about 100 pages into the book I’m currently reading (it rocks, more on this later) and I picked it up the other day to discover that the cats had helpfully chewed the back corner. This isn’t the first time this has happened. While I don’t really care, I can still read it, part of reason I don’t seem to care is that the quality of books as objects these days doesn’t particularly impress me. So I don’t feel like I’m loosing anything. And if I want a real book-object, such a thing can be had.

I have a suspicion that you have more than a few paper books that you haven’t read. What are you going to do with them now?

Read them. I don’t think that I’ll stop reading paper books, though I think a great deal depends on context. I suspect that I might not take paper-books out of the house very much. I don’t have a lot of books, but I certainly have a few, and I know that I mostly have them with me for nostalgia, and not because I actually intend to read them any time soon.

Paper books, on my shelf, represent possibilities, in a way that the object of the Kindle is a possibility. I think even considering the limitations of the Kindle, these two truisms balance each-other out.

How do you think you’ll use the kindle?

I think once the initial buzz of the Kindle wears off, I’ll probably settle into a rhythm whereby I’ll read periodicals, fiction, and documents that I generate (along the lines of slush) on the Kindle along with anything I read out of the house, and then read reference material off of paper. I’m mostly worried about how the kindle might screw with my--often quite good--spatial memory for texts. We’ll have to see how this develops.

I’m strongly considering joining a gym in the next few weeks and I hope/expect to read whilst doing the aerobic thing. The kindle seems ideal for this.

I hear your a slow reader, is this really worthwhile?

Perhaps, and I think that my main issue is that I’m really bad at setting aside time to read when I’m awake enough to actually read. This is separate issue from the Kindle, and one I suspect I’ll address in future posts. Having said that, I’m attempting to carve out a bit more time for reading in my day--as reading more is a personal goal--so I’d say that yes: Despite my apparent slow pace, a Kindle is worthwhile.

Do you have any Kindle related questions for me?

Three Predictions

Mon, 28 Dec 2009 00:00:00 +0000

Ok folks, here we go. I’ve put the word futurism is in the title of the blog and its near the end of the calendar year, so I think it’s fair that I do a little bit of wild prediction about the coming year (and some change.) Here goes:

Open technology will increasingly be embedded into “closed technology”

The astute among you will say, “but that is already the case:” The Motorola Razor cell phone that my grandmother has runs the Linux Kernel (I think.) And there’s the TiVo, lets not forget the TiVo. Or, for that matter the fact that Google has--for internal use, of course--a highly modified branch of the Linux Kernel that will never see the light of day.

That’s old news, and in a lot of ways reflects the some of the intended and unintended business models that naturally exist around Open Source.

I’m not so much, in this case talking about “openness” as a property of code, but rather openness as a property of technology, and referring to long running efforts like XMPP and OpenID. These technologies exist outside of the continuum of free and proprietary code but promote the cyborg functioning of networks in an transparent and networked way.

XMPP says if you want to do real time communication, here’s the infrastructure in which to do it, and we’ve worked through all the interactions so that if you want to interact with a loose federation (like a “web”) of other users and servers, here’s how.

OpenID solves the “single sign on” problem by creating an infrastructure for developers to be able to say “If you’re authenticated to a third party site, and you tell me that authenticating to that third party site is good enough to verify your identity, then it’s good enough for us.” Which makes it possible to preserve consistent identity between sites, it means you only have to pass credentials to one site, and I find the user experience to be better as well.

In any case, we’ve seen both of these technologies become swallowed up into closed technologies more and more. Google Wave and Google Talk use a lot of XMPP, and most people don’t know this unless their huge geeks (compared to the norm.) Similarly, even though it’s incredibly easy to run and delegate OpenIDs through third parties, the main way that people sign into OpenID sites is with their Flickr or Google accounts.

I’m not saying that either of these things are bad, but I think we’re going to see a whole lot more of this.

A major player on the content industry will release a digital subscription plan

I think, perhaps the most viable method for “big content” to survive in the next year or so, will be to make content accessible as part of a subscription model. Pay 10 to 20 dollars a month and have access to some set quantity of stuff. Turn it back in, and they give your more bits. Someone’s going to do this: Amazon, Apple, Comcast, etc.

It’s definitely a hold over from the paper days when content was more scarce. But it gets us away from this crazy idea that we own the stuff we downaload with DRM, it makes content accessible, and it probably allows the of devices to shoot down (to nominal amounts). While it probably isn’t perfect, its probably sustainable, and it is a step in the right direction.

Virtualization technology will filter down to the desktop.

We have seen tools like VirtualBox and various commercial products become increasingly prevalent in the past couple of years, to decrease the impact of operating system bound compatibility issues. This is a good thing, but I think that it’s going to go way further, and we’ll start to see this technology show up on desktops in really significant ways. I don’t think desktop computing is in need of the same kinds of massive parallelism that we need on servers, but I think we’ll see a couple of other tertiary applications of this technology.

First, I think hypervisors will abstract hardware interfaces away from operating systems. No matter what kind of hardware you have or what it’s native method of communication is, the operating system will be able to interact with it in a uniform manner.

Second, there are a number of running image manipulation functions that think operating system developers might be able to take advantage of: first the ability to pause, restart, and snapshot the execution stat of a running virtual machine has a lot of benefit. A rolling snapshot of execution state makes suspending laptops much easier, it makes consistent desktop power is less crucial. And so forth.

Finally, system maintenance is much easier. We loose installation processes: rather than getting an executable that explodes over our file system and installs an operating system, we just get a bootable image. We can more easily roll back to known good states.

Not to mention the fact that it creates a lot of economic benefits. You don’t need IT departments maintaining desktops, you just have a guy making desktop images and deploying them. Creating niche operating system images and builds is a valuable service. Hardware vendors and operating system vendors get more control over their services.

There are disadvantages: very slight performance hits, hyepervisor technology that isn’t quite there yet, increased disk requirements. But soon.

Soon indeed.

Web Frameworks

Wed, 23 Dec 2009 00:00:00 +0000

I’m not a web developer. I write the content for (a couple) of websites, and I’m a fairly competent systems administrator. Every once and a while someone will need a website, or I’ll need my site to do something new that I haven’t needed to do before and I’ll hack something together, but for the most part, I try and keep my head out of web development. Indeed, I often think that designing applications to run in the web browser is the wrong solution to most technological problems. Nevertheless, my work (and play) involves a lot of tinkering and work with web-applications, and I do begrudgingly concede the relevance of web applications.

In any case I’ve been reading through the following recently, and I (unsurprisingly have a few thoughts:)

The Trouble With Frameworks

I really enjoyed how this post located “web frameworks” in terms of the larger context: what they’re good for, what they’re not good for, and why they’re so popular. I often feel like I see a lot of writing about why FrameworkA is better or worse than FrameworkB, which doesn’t really answer a useful question. While I wouldn’t blame my gripe with web-based applications entirely on the shoulders of frameworks, it’s interesting to think of “the framework problem” as being a problem with the framework (and the limitations therein) rather than a problem with the web itself.

This isn’t to say that frameworks are inherently bad. Indeed, there is a great deal of work that websites require in-order to function: HTML is a pain to write “by hand,” consistent URLs are desirable, but it’s undesirable to have to mange that by hand. If you need content that’s dynamic, particularly content that is database-backed, there is all sorts of groundwork that needs to be done that’s basic and repetitive even for the most basic functionality. Eliminating this “grunt work” is the strength of the framework, and in this they provide a great utility.

However, from an operations (rather than development) perspective, frameworks suck. By producing tools that are broadly useful to a large audience, the frameworks are by nature not tuned for high performance operations, and they don’t always enforce the most efficient operations (with regards to the databases). Thankfully this is the kind of issue that can be safely delegated to future selves, as premature optimization remains a challenge.

Thoughts on Web.py

Though I’m not much of a Python person, I have a great deal of respect for Python tools. I swear if I were going to learn a language of this type it would almost certainly be Python. Having said that, the tool looks really interesting: it’s minimal and stays out of the way for the most part. It does the really “dumb” things that you don’t want to have to fuss with, but doesn’t do a lot of other stuff. And that’s a great thing.

I’m not sure how accurate this is, but one of the things that initially intrigued me about web.py is that it sort of felt like it allows for a more “UNIX-y” approach to web applications. Most frameworks and systems for publishing content to the web work really well as long as you don’t try and use anything but the application or framework. Drupal, Wordpress, and Rails seem to work best this way. Web.py seems to mostly be a few hooks around common web-programing tasks for Python developers, so that they can build their apps in whatever way they need to. The the monolithic content management approach doesn’t feel very UNIXy by contrast. I think this is something that I need to explore in greater deal.

Having said that, I’m not terribly sure if there are problems that I see in the world that need to be addressed in this manner. So while I can sort of figure out how to make it work, I don’t find myself thinking “wow, the next time I want to do [this], I’ll definitely use web.py.”

But then I’m just a dude who writes stuff.

Deleuze and the Utility of Materialism

Tue, 22 Dec 2009 00:00:00 +0000

(ETA: On second thought, perhaps this essay should have been called “Materialism and the Utility of Deleuze,” but both work.)

Here’s the second part in my (re)contemplation of Deleuzian theory. Here’s part one.

Everything is a machine. Celestial machines, the stars or rainbows in the sky, alpine machines--all of them connected to those of [the] body. […] There is no such thing as either man or nature now, only a process that produces the one within the other and couples the machines together. Producing-machines, desiring-machines everywhere, schizophrenic machines, all species of life: the self and the non-self, outside and inside, no longer have any meaning whatsoever. (p. 2)

-- Giles Deleuze and Felix Guatteri Anti-Oedipus Originally published in 1972, English translation 1977. Translated by Helen R. Lane, Robert Hurley, and Mark Seem.

I think one of the key reasons that I keep returning to Anti-Oedipus is that it provides a way to be a fierce materialist while addressing the kinds of questions that idealists (i.e. psychoanalysts) raise. This in itself isn’t particularly unique (I suppose,) but I’m particularly taken with the way that they approach questions of subjectivity, identity, experience, and development without engaging or furthering the discourse of psychoanalytic thought.

Initially I think I was off put by all the psychoanalytic language in the text, and the way that they seem to argue incredibly fine points against Lacan and Freud. As I look at it more and more, I realize the point of Anti-Opedipus is to say “don’t think about these issues in Freudian terms, and with Freudian assumptions! Think about subjectivity and identity as phenomena with material foundations and mechanistic underpinnings!”

I, perhaps unlike the milieu that Deleuze and Guatteri were writing in, was never particularly enchanted by psychoanalysis, but I have been incredibly interested in the kinds of issues that analytic thought engages, and Anti-Oedipus provides a way to entertain those kinds of discussions without engaging in a troublesome intellectual lineage.

But to tie this post back to the last one, this approach to thinking about ourselves as subjects, to our creativity and desire, to the cultural implications of our identity, is not something that’s particularly useful addition to a theoretical framework. Right? I’ve not done a lot of this kind writing recently, but it strikes me that the call to be a materialist, and to think about the mechanics of social and personal phenomena is, as we say, “non-trivial.” Being Anti-Oedipal isn’t just something that you sprinkle here and there; it’s not a grand-theory-of-everything, but once it seeps in a bit, it makes it possible to think about the world and experiences in--what I’d call--a more productive way. Perhaps it’s true that Anti-Oedipus is a book of ethics after all.

I underlined the paragraph from the last post nearly four years ago. I think I’ve finally gotten it. I think, more than anything, that is a marker of my own development.

Onward and Upward!

Perhaps Someday We'll Call This Deleuzian

Mon, 21 Dec 2009 00:00:00 +0000

I would say that Anti-Oedipus (may its authors forgive me) is a book of ethics, the first book of ethics to be written in France in quite a long time (perhaps that explains why its success was not limited to a particular “readership:” being anti-oedipal has become a life style, a way of thinking and living.) How does one keep from being fascist, even (especially) when one believes oneself to be a revolutionary militant? How do we rid our speech and our acts, our hearts and our pleasures, of fascism? How do we ferret out this fascism that is ingrained in our behavior? The Christan Moralists sought out the traces of the Flesh lodged deep within the soul. Deleuze and Guatteri, for their part, pursue the slightest traces of fascism in the body.

-- Michel Foucault, writing in the preface to Anti-Oedipus, by Giles Deleuze and Felix Guatteri.

I’ve spent a while away from Academia and geeky theoretical academic thoughts for a while. Then I discovered this twitter account and I got drawn back into it. I read the tweets and I thought, “you know,” these are hilarious on their own because they are so off the well, but I think I actually understand what’s going on. I’d have conversations with unsuspecting coworkers about little bits of Deleuzian theory. H.S. came for a visit and we had a rather long conversation about Deleuze and theory. I don’t know that “I’m back,” is exactly the right way to phrase this, but I definitely enjoy the added perspective that I’m able to bring to this stuff now.

I was never a very good theorist or philosopher, though I enjoy watching from a far, I tend want answers to different kinds of questions. I’m not, nor have I ever been “a scholar” of the “Capitalism and Schizophrenia” diptych--I haven’t even read it in its entirety--but it’s been a great influence me. Of the things that I read and interacted with in college, I’d have to say that Anti-Oedipus and A Thousand Plateaus are the texts that I return to with the greatest frequency. And I never even took a class that assigned D&G!

I’ve read a fair number of papers and other pieces that have attempted to use Deleuze’s work as theoretical framework or some such, and I’ve always been disappointed by what happens as a result. For starters, the chance of Deleuze citations being: of the Rhizomatics essay at the beginning of “A Thousand Plateaus,” or from his collections of film criticism are overwhelming. This is unsurprising as this probably represents the most accessible of portions of Deleuze’s work. Also unsurprising is my sense that no matter what the paper is about, the Deleuzian theory overpowers whatever the author is trying to say. Deleuze’s thought is pretty darn heavy, and there’s no way around it.

And from some perspectives this is actually pretty funny: when you read Anti-Oedipus it’s not “fluffy,” but it’s pretty playful. There are lots of metaphors and images that draw out the logic and the point. There’s a lot going on, but it’s not dense (certainly not in the way that Derrida is dense.) This has lead me to ask a two important questions:

If the writing is not very difficult or opaque, why do (Americans) who attempt to use the work fail to capture the playfulness, and seem too fall flat?
Why am I (and clearly others as well) so intrigued by this work, and why do I (we?) keep returning to this text? Particularly since it’s so difficult to use in support of other arguments.

The answers, I think bring us back to Foucault’s assertion in the preface, that Anti-Oedipus is (counter to first impressions) a book of ethics rather than a book of cultural and social theory or even a commentary on Marxist and Freudian theory. When reading the texts, Anti-Oedipus (and A Thousand Plateaus) don’t feel like ethical manifestos, but I think that this explains why it’s so difficult to use and remains so intriguing.

That’s enough for now, but I hope you’ll pardon my impulse to blog about Deleuze for a little longer, as I think there’s another post or two here.

We'll Always Have Debian

Thu, 17 Dec 2009 00:00:00 +0000

I know I just wrote a long piece about Arch Linux and for most things I’ve pretty much switched to Arch Linux as my primary, day to day, distribution. In fact, when an Arch Linux issue comes up at work my coworkers call me first. And I suppose it’s well earned. But if you were to ask me what my favorite Linux distribution project was, I’d probably say Debian as often as not.

I run a lot of Debian, straight up, unmolested (mostly) Debian Stable. There are a lot of practical reasons for this: it’s stable, I have faith that it’s going to work and do what I need it to. Aside from keeping on top of normal security issues, the system is stable and doesn’t require attention to keep up to date. And, in nearly every case the package manager can be trusted to do the right thing. There are also a ton of little niceties in the distribution: debconf, the management tools for Apache, and the shear diversity of the packages. It all adds up.

I mean, I have gripes with some things that Debian does, but they’re always little. I find myself asking “Why didn’t you enable mod_rewrite by default? Really?” or “Would it have killed you to include software that was less than 3 years old in this release?” But never “Why is this broken by default?”

With projects like Ubuntu getting press, attention, and energy (and money!) I can’t help but think that the outsider might think of Debian as being a bit… put upon? Or not good enough in it’s pure form? The Ubuntu folks are pretty good about talking about their Debian roots, and it’s totally clear to anyone who really takes a good look at Ubuntu that most of its awesomeness is due to being Debian-derived. Even if that isn’t terribly clear from the outside.

I also really enjoy the ways in which Debian has managed to grow and sustain itself, and create something that is so magnificent in scope. The Linux Kernel project is huge, the desktop projects are massive in terms of what they carry under their umbrellas. Distribution projects that start from nothing and control and build the entire stack are, I think, particularly intense because of the shear size of the project.

This of course holds true for all distribution projects, and doesn’t make Debian particularly special, I suppose. The thing is that Debian’s coverage is massive compared to other tools. Arch provides a great framework for an operating system, and makes it really easy to do a number of things, but there are nowhere near as many packages nor as many contributors. Ubuntu is, by contrast a great project but is mostly a process of “tuning” Debian into a system that’s more targeted for specific applications. Again these aren’t criticisms, but it does make Debian more of an impressive proposition.

And I guess, because of this, even though most of the time when I interact with a Linux system it isn’t actually Debian, I almost automatically categorize myself as a “Debian person.”

Shrug.

Poems are Made out of Words

Wed, 16 Dec 2009 00:00:00 +0000

I remember having this epic fight conversation with a poet-friend from college about aesthetics and art and literature. I’m not sure exactly what brought it on, or particularly why I thought my side of the argument was in any way defensible, but it came back to me recently. So as I’m wont to do, here’s a post in review of these thoughts.

Act One: Poems are Just Words

I think in the first iteration of the argument, I took the opinion that poems existed (mostly) to transcend the experience of the written word on the page. That the project of poetry was about getting past words and constructing some sort of image or transcendent experience, or something.

Did I mention that I wasn’t a poet? I’m not. Not at all. I’m not even particularly good at reading poetry. I’ve sometimes written poems, and even I am a good enough reader to tell that they’re crap.

In any-case, H.S.’s argument was that poems were just words on paper (or screens)¹ and that’s it. That writing itself is an act of putting words together, and experimenting with how words come together in (quasi) fixed mediums. And nothing more.

I don’t really know what my beef in this argument was. This was certainly before I started writing again. I guess my argument was that writing was simply an imperfect means of conveying an idea, and the real work and creativity of “being a writer” was really in coming up with good ideas and practical logic that illustrates your arguments.

And while that’s true, from one perspective if you squint at things the right way, I don’t think it’s really true about writing as a whole, and certainly not creative projects. It might be true that that’s a pretty good summary of academic writing, particularly entry level academic writing, but I’m not sure.

When I find writing that I’m impressed with, I keep coming back to the idea that it’s just “words on the page,” and somehow that makes. My skill--insofar as I have one--and the asset that makes me employable (I think) is the fact that I can turn ideas and thoughts (which are thick on the ground) into something useful and understandable by normal folk.

Act Two: Rethinking William Gibson

So, ok, lets be honest. I don’t really like William Gibson’s work very much. I thought Neuromancer expressed a social commentary that was totally obvious almost instantly, and it hadn’t stood the test of time particularly well, and I felt it sort of read like the rehab journal of an addict who hadn’t quite cleaned up entirely. This was just my reaction on reading it, and not a particularly well reasoned critique.

I mean I will acknowledge the book’s impact, and I think I read it too late which probably accounts for my reaction. And although I responded so poorly to it, I don’t really have a lot of a problem with literature that is of its time. In any case, I was thinking about Gibson recently, and casually comparing him to some other writers, and I found myself saying (of another writer of the cyberpunk ilk), pretty much without realizing it:

…which is fine, except [they] didn’t have Gibson’s literary chops. I mean Gibson’s work is incredibly frustrating but his writing is superb.

And I sort of realized after I’d said the above, that I had inadvertently conceded the argument from Act One, years later. Sure there’s a lot of idealism in writing, but writers aren’t differentiated on the basis of how awesome their ideas are. It all comes down to how they put the words together.

The side effect of this transposition is that, somehow, I’ve started to be able to read (and enjoy) short stories more than I ever was before. And much to my surprise, I’ve been writing the end of this (damned) novel as a sequence of short stories. At least in how I’ve been thinking of it. I could go on with more and additional examples, but I think I better leave it at that for now. Thoughts? Anyone?

Or in my case emacs buffers. ↩︎

Getting the Most from Arch Linux

Tue, 15 Dec 2009 00:00:00 +0000

Much to my surprise, my essay of a few months ago on Why Arch Linux Rocks is quickly becoming one of my most popular posts. Who would have thunk? In any case, while I’ve written a few additional bits here and there about using Arch, I thought it would be good to write some more concrete and practical reflections on what I’ve learned from Arch and hopefully someone out there will find this useful.

The thing about arch, is that it’s by nature a very minimal operating system, and it doesn’t shy away from the command line. There’s this peculiar logic amongst GNU/Linux Distribution Developers that says: we want people to use our operating system and to use Free Software, and since most people are scared off by command line interfaces and editing config files, lets try and obscure the command lines and the config files as much as possible. That’s a good idea.

Right. Not really.

And for some distributions, this really works. Arch says, rather, that while command line interfaces and config files can be confusing they’re not “bad,” and it’s possible to teach people how to interact with the command line and edit their config files. Furthermore, while command lines and config files might be different and unfamiliar to users who are used to checkboxes and menus, they are extraordinarily simple.

Back to the minimal part.

When you install Arch, you get a prompt, the basic GNU tool chain (install base-devel during the install process, it’s worth it,) the Kernel (of course), the GRUB boot loader, and that’s about it. Everything else you have to install yourself. While you might thing “this is a huge bother” the first time you install Arch, the when you get to the end of the process you have a system that does exactly what you want it to and nothing more. Having a computer system so tailored to your needs and workflows is actually, sort of a unique and wonderful experience. Rarely, I think, in this day and age do we get work with a computer in this way.

Nevertheless, having a system that is this minimalist means that setup can be a bit… intense. Once things are installed the right way it works great, but I found that the first few times I ran an Arch install, it was like I spent the next two weeks installing little things over and over because I couldn’t keep track of what I needed. And then I installed a second system, and it was the same thing over again. The third time I did it, I’d wised up and managed to have a better time with it. A few weeks ago I created and redeployed a virtual machine (in virtual box) that I use as my primary work computer (long story), and it was painless.

What remains of this post are a collection of lessons that I’ve learned and some suggestions I have for having the best arch experience possible.

Save your /home directory and copy it to the new machine. I used the following command over the local network from the home server: :
```
rsync -azrv /home/tychoish tychoish@192.168.256.1:/home
```
Changing the paths, usernames, and IP address to their relevant and valid values. This way all your configuration files and data end up on the new machine, with permissions preserved. This typically takes the longest time of any setup operation.
Run system updates as soon as possible. Arch is a rolling release distribution. The longest you’ll typically go between updates is the time between when the installation media was created and when you install your machine. The longer the divide between current status and when you run an update the greater the chance of breakage happening.
Use ABS, the “Arch Build System,” to compile any software that isn’t in the main arch repositories. Save the PKGBUILD scripts (if not the packages you’ve made with them,) in your `~/abs/`` directory. This makes installing all of the weird and tedious software much easier.
Avoid getting a machine with weird or uncommon wireless or video drivers. At this point, I’m choosy enough about my hardware that I won’t get a computer (much less install Linux on it,) if it’s not “Intel everything:” wireless card, video card, chipset, etc. Sure a fancy NVidia card might be more sexy, and there are a lot of good reasons to use AMD and ATI silicone. But, one can be very sure that Intel hardware is going to work with Linux; and with other gear, it’s much more hit and miss. Or it can be. And my time and serenity is not without value.
Maintain a list of what packages you want to install on your system:
- X11 Dependencies
  
  Lets not talk about how long it’s taken for me to remember to install the X11 keyboard and mouse drivers:
```
xf86-input-keyboard xf86-input-mouse
```
- Additional package management tools
  
  The abs tool provides a BSD-ports like interface for building packages, while the ‘yaourttool makes it easy to make and build packages from the [Arch User Repository][aur] (AUR.) Typically I use [yaourt][] to download a package and then build it in theabsdirectory. Make sure you've installedbase-devel` as well, because you’ll want to eventually.

Music Applications

I use the music player daemon and a number of additional applications to play music. And I install these packages as a matter of course.
mpd mpc gmpc alsa-utils jack-audio-connection-kit
The tychoish Tool Chain

Your specific list of “tools that you must have in order to function properly,” probably varies a bit. Here’s mine (some packages are from AUR):
git emacs emacs-org-mode emacs-w3m-cvs rxvt-unicode
screen wcalc sudo htop urxvtcd zsh swiftfox-i686
To explain quickly those packages which aren’t obvious from their title: wcalc is a command line calculator tool; swiftfox-i686 is an optimized build of Firefox, except it runs more smoothly in my experience and is compatible with the FF plugins that I depend upon. urxvtcd is a shell wrapper for urxvt that opens a client of the urxvt-daemon (urxvtc) if there’s a daemon already running, and starts a daemon and opens a window if there isn’t. htop is a system process monitor.
Email Tool Chain

I use the following packages to manage my email:
procmail msmtp fetchmail gnugpg
From AUR I also use:
lbdb mutt-sidebar
lbdb is an address book database tool, and I use my build to get the best mail-reading client in the world.
Window Management

Arch is great for having pretty good support and inclusion of up-to-date packages for esoteric window managers. The sequence for installing StumpWM is a bit non intuitive. First, install the following packages from the normal repositories:
sbcl nitrogen gtk-chtheme
The last two are tools for changing the look and feel the desktop and GTK applications (respectively). Now from AUR, install (in the following sequence):
clx cl-ppre stumpwm-git
Put the following code in your ~/.sbcl file:
(require 'asdf)
(pushnew #p"/usr/share/common-lisp/systems/" asdf:*central-registry* :test #'equal)
(asdf:operate 'asdf:load-op 'cl-ppcre)
(push #p"/usr/share/common-lisp/systems/" asdf:*central-registry*)
(asdf:operate 'asdf:load-op 'cl-ppcre)
And you should be good. Configure your .xinitrc to your liking, and make sure your have a .stumpwmrc.
Networking Tools

To manage my own connections and connect to the OpenVPN I use netcfg and OpenVPN. This requires the following packages:
netcfg netcfg-auto zsh-netcfg netcfg-openvpn openvpn
There are also a suite of network diagnostic tools that I always install, but seem to forget for various parts of a week.
mtr whois dnsutils
Also, I’m always befuddled by this for a few moments, but ssh isn’t installed by default. I, as a result install the following tools:
openssh sshfs
And add the following line to /etc/hosts.allow to permit inbound SSH connections.
sshd: ALL
Spelling Correction

This always frustrates me, though I understand the logic. Install the following three packages to get spelling support thought the system:
aspell ispell aspell-en
If you need spelling correction for a non-English language, replace aspel-en with that language, or add that language to the end of this list.

Links on the Art of Techology

Mon, 14 Dec 2009 00:00:00 +0000

I have a collection of links that I’d like to share with you. I hope you enjoy and find them as enlightening as I have. Some of these are dated but, I’ve been milling through them for a while and I feel like they’re worth sharing. In three parts:

Computer Programming and Hacking

There’s a long tradition of computer scientists and eminent developers thinking about their software development as an Art. Donald Knuth’s major work is called “The Art of Computer Programming,” and literate programing is fundamentally an artistic rather than a technical idea. The idea of the “lazy programmer” from the Perl world has some obvious artistic implications (hell, Perl’s license is called the “Artistic License”), and the Extreme Programing (XP)/Agile Programming world is very much about addressing programming as creative challenge rather than a purely technical challenge.

I particularly like the contrast of the first two articles with the third. While I’m not sure it’s particularly conclusive in the brief overview and with such a small sample, the insights about the ways that programmers approach problems is pretty useful. Makes me want to get the book.

I may not be a programmer in the conventional sense (or even any of the unconventional senses that I’m aware of) but there are two things that I know for sure. First: functional programming makes sense to me in a way that nothing else ever really has, and secondly that I am fascinated by the different ways that people use and manage projects with Git.

I think functional program makes sense to me because little blocks that “do something” matches the way my brain is shaped in some useful way. Every introduction to object oriented programming I’ve ever experienced starts with like 50 pages (or equivalent) of crud about data structures and types . Which I’m sure make sense if you know what’s coming next, but if you’re a programing n00b: less than helpful.

Also, regarding the use of git: it’s fascinating how many different ways and different work-flows people manage to squeeze out of software like this! What he’s doing makes sense, a lot of sense on paper, but most people don’t publish long-running branches in such an organized manner. Sure there are “vendor branches” for release maintenance, but branches in git tend to be much more ad hoc from what I’ve seen. Anyway, good one for the file.

The Trouble With Frameworks

I’ve been looking over weppy for the past few weeks, along with the bindings for a couple of weird databases (of the NoSQL variety, loosely), as part of a brief attempt to learn how to think like a web-developer. (It’s hard more on this later.) I find myself both incredibly enchanted with the prospect of some of these frameworks (particularly the python ones), and yet at the same time very unsure of what they represent in terms of the Internet. Frameworks aren’t going anywhere, and I think by some measure they are “getting better,” I do think they make it awfully easy to avoid putting in the time to “get it right,” which might not matter most of the time, but when it does oh boy, does it.

Academia, Anthropology, Literature

I wrote a series of articles on the future of publishing and I found myself returning again and again to these two essays as a source of inspiration, and I haven’t quite gotten them out of my head.

I’m not sure if I agree with this consensus, but it seems pretty clear that multi-media websites, twitter, and “blogs” (by some definition of the term) are acceptable replacements for journalistic publishing (newspapers, magazines). These essays, engage literary publishing, and force (particularly the second) us to think about the economics of booksellers, including branding, brick and mortar shops, which I think is incredibly important.

In the piece on Cyborg Anthropology, Amber Case discusses looks at the Singularity from a much less technical perspective. In a lot of ways this reminds me of my post on the Dark Singularity from a few months back. The singularity is, of course ultimately a cyborg problem.

Aaron Swartz’s piece on the academy… I don’t know that he’s wrong, exactly, but I think I would avoid being as unilateral as he is. On the one had disciplines exist mostly to organize education not research, and if I were going to make a conjecture: we see more disciplanarity and sub-disciplining in fields with substantive funding outside of the academy. Doctors, lawyers, psychologists, biologists, chemists, have teeny-tiny little sub-fields; and by contrast you see a lot more interdisciplinary activity in the academic study of anthropology, literature, mathematics, and, say musicology. Interesting, nonetheless.

The Industry of Technology

Two posts here from James Governor of RedMonk that have guided (at least topically) my thinking in a couple of areas recently. This is particularly noticeable if you look over the recent archives. The first addresses Flash and the business of advancing the platform of the web. My trepidation with regards to flash is mostly the same as my trepidation with regards to all web technology. When you get down to it, my grumbling all goes back to the fact that the web is developing into this thing that is all about media rich software, and less about hypertext, which is where my heart has always been. My argument keeps going back to “take the applications off of the web as we know it, and use a platform that (like GTK+ or QT) that’s designed for applications, and create a hypertext experience that really works.” But it’s nice to have articles like this one to pull my head out of (or into?) the clouds to remind me (and us?) what’s really going on in the industry.

This is an old one from the same source, about the new “patronage economy” which in many ways defines what’s going on at RedMonk. I particularly enjoy how Governor contrasts the New Patronage with other popular web 2.0-era business models. I do worry, looking at this in retrospect about the ways in which such patronages are stable even when the economy is in the crapper (as it were.) I’m not sure if there’s an answer to that one yet, but we’ll see. I guess my questions, at this juncture are: first, does patronage-type-relationships give “start ups” another funding option which is more stable than venture capital. Second, doesn’t the kind of organization and work that’s funded by these patronages subvert the kind of work that they depend upon (i.e. big high-margin businesses?)

That’s all I have at the moment. If you have links or thoughts to share, I’d live to see them!

Email is Still Broken, but at Least People are Talking

Thu, 10 Dec 2009 00:00:00 +0000

Email is broken. I don’t think the assertion really even requires an example for everyone to concur. Spam, ongoing wars over decades old standards and protocols, bacon (automatically generated email that you want), listservs, poorly written clients, and mixed expectations regarding replays, make email frustrating to deal with under the best of situations.

Email is also inescapable. There’s no other technology around at the moment which pushes messages out to users as effectively (every other thing is pull), provides administrators with as many tools and options, or has such ubiquitous adoption rates. I’d love to be able to say “don’t send me emails, I won’t get it,” but the truth is that I get and send a bunch of email, and every other option for “next wave” digital communication fails to impress.

Including Google Wave.

Don’t get me wrong, Wave is nifty. Hampered mostly by an interface which is foolish and gimmicky, I think Wave provides capabilities that we need on the Internet. Wave makes multi-user chat useful for people who have trouble wrapping their brain around IRC. Wave strikes the right balance between “real-time” and persistence (which is mighty difficult), and most importantly…

Wave gets people thinking about email, and what the ideal replacement might look like. And that’s a good thing indeed.

Maybe, more importantly, Wave moves many of the trivial uses of email, into a more suitable format (XMPP) so that email, or whatever replaces it, has a more easily defined task. Maybe there are a couple of XMPP applications: Jabber, Wave, and (----)¹ that work in concert to alleviate the pressure on email and make it much more clear what “the email killer” will look like.

To my mind, one of the biggest problems with email is that we use it for too much, including a number of things that it’s not ideally suited for. Perhaps the way to fix email is to fix all of these other communication problem, and see what’s left.

Here’s a list of the things what I think we use email for today, to which it is unsuited, followed by a couple of examples?

Notifications. There’s been a reply to a comment on your blog, someone has commented on your face-book status update, it’s time to renew your domain name, your payment has been processed. Etc.
File Transfer. Look at my resume. Here’s the paper I owe you. I was looking for your article and wondered if you might have a copy you could share? Here’s this funny picture my uncle sent me. Etc.
Content Review. Could you take a look at this essay or bug report? I mentioned this in an IM but didn’t want to send it in that format.

This leaves us with a relatively short list of things: genuine letters (i.e. first class mail; generally “asynchronous personal communication”) and listservs (which like Usenet are probably the best way to have async-group communication online from the users perspective). And spam, but we don’t care about that.

So what’s the best way to handle what’s left? Hell, what’s the best way to handle the rest of it?

I don’t think there are good answers yet, at all, and I look forward to continuing to thing about these issues in a big way.

I think this might tie back to my posts on notification systems, but more along the lines of what we currently use SMS for (say,) but I’m not sure what this looks like in a personal (much less interpersonal) method. Lots and lots of whitelisting, that’s for sure. ↩︎

How I'd Like to Be Notified

Wed, 09 Dec 2009 00:00:00 +0000

In my last post on this topic I tried to avoid talking too much about the actual details of an implementation of what my ideal notification system would look like. In part because I think that the problem of staying on top of our digital (cyborg?) in a “real time” manner is something that transcends the specific method of implementation. I and many other awesome people have been thinking about this “real time” problem for a couple of years, and mostly we’ve thinking about things like microblogging (think twitter and identi.ca) and XMPP and related topics. These are all important issues that we need to get worked out, but there are issues regarding the use of “real time data” that I think need to be addressed from a more indivudal/user perspective. This notification issue is one of them.

My requirements, in summary:

There should be some common method of creating and distributing notifications that’s easy to setup up and tie into scripts (i.e. with some sort of command line interface.) and that is independent on application. I don’t want notifications from my email to appear in a different place from notifications from IRC, for instance.
Taking the above one step further, notifications should be machine independent. If an event that I’d like to be notified of happens on my server (in the cloud, as it were) or the desktop at home, I’d like to know about it on my laptop when I’m on the road. Preferably, I’d like the system to know if I’ve seen a notification on my work computer before alerting me to it at home, but I’m willing to slide on this one.
I need the ability to filter and control what notifications I see. There are times when I really want to track a particular word or phrase on twitter, and there are times when the last thing I want to know is weather or not someone has said something or other on twitter. Fine grained controls are key. I’d really like to put some sort of filtering pipe intweent “messages received” and “notifications sent,” so I can use it the way I use procmail for email.

Thoughts on Implementation:

XMPP seems like an obvious tool for implementing the backbone of this. From my perspective of a slightly-more-than-casual observer, getting one machine to know if a notification has been seen by machines is a bit thorny at best.
Notifications need to tie into the window manager and desktop experiences. in some sort of unobtrusive way, but also in a way that is consistent with that window manager normally works.
While many notifications will probably be created by sending a simple string to a command, some notifications will probably be generated from RSS or JSON feeds, and the best systems would provide some baked in support for pulling events from these kinds of data.
You can get irssi to send xmpp notifications, which is both incredibly ironic, and kind of awesome, as a building block. Also, this emacs-fu for using lib-notify from emacs might be another good starting point.
Dumping a firehose of notifications in an XMPP window, might be a good start, but I think once a notification has been sent (and logged and dismissed?), notifications need to disappear forever. Also--and this is the point of the filtering--raw events take processing in order to be useful. The firehose, of anything when exposed to the user, isn’t particularly useful.
[insert your thoughts here]

I will, of course let you know how this goes.

Five. More. Things.

Tue, 08 Dec 2009 00:00:00 +0000

The latest in a fine tradition of tychoish posts…

Five People I Enjoy on Twitter

Five Improvements to Web Browsers

Enforce document structure standards on the server. Documents must be structured and organized within the constraints of a couple of conventions or else the server throws a 500 error.
Locate all design and presentation on the client side, and allow designs to be fully independent of the design.
Scripting happens on the client side, in sandboxes, and are integral parts of the browser applications.
Therefore, scripts shouldn’t be provided with the browser or by the viewer, not by the content creator. In other words, die JavaScript, die, and replace it with lightweight greasemonkeyesque+webkit-style browsers. Except might as well use python/perl/ruby/etc. while you’re at it.
Applications, can and should interact with servers and infrastructure and data over the network, but “the web” shouldn’t convert interactive applications. The goal here is to make the web-as-information-distribution work.

Five Things I’m quasi likely to Acquire in the Next Few Months

An Amazon Kindle
Some sort of roasting pan
A New Laptop Bag
A table and chairs (one of which might be good for spinning)
A gym membership

Five Things That are Awesome about Dance Weekends

You can dance with someone more than once, without it being creepy or weird. At an evening of a dance, it’s only really realistic to dance with someone once because there are 15 or 20 dances max. At an all day dance, we’re talking 60 or 70 dances.
Innocent and awesome dance crushes.
The “gender” thing is more flexible at big dance weekends. Which is fun. It’s probably true that part of the fun is in part due to the fact that queering up the gender roles in a contra dance makes contra a little harder, but I also just enjoy it for it’s awesome factor. M.N. also noted that when you come across older men dancing together in the contra lupine, you know it’s all gravy. And that’s more likely to happen at a dance weekend.
Awesome bands. In short: Giant Robot Dance.
Squares that don’t suck.

Five Changes in Habits that I Would Have Never Expected to See In Myself

Sacred Harp/Shape Note Singing
A Decrease in knitting output and interest
Weekend plans and events three to four weekends out of five.
The degree with which I’m a neat freak about my apartment. (Not absurd, but I still have moments where I surprise myself.)
Contra dancing as my main form of dance, as opposed to Morris and/or International folk.

Doing Wikis Right

Mon, 07 Dec 2009 00:00:00 +0000

Wiki started as this weird idea that seemed to work against all odds. And then it seemed to really work. And now wiki is just a way to make and host a website without taking full responsibility for the generation of all the content. To say wiki is to say “collaboration” and “distributed authorship,” in some vague handwaving way.

But clearly, getting a wiki “right,” is more difficult than just throwing up a wiki engine and saying “have at it people.” Wiki’s need a lot of stewardship, and care, that I think people don’t realize off the bat. Even wikis that seem to be organic and loosey-goosey.

I have this wiki project, at the Cyborg Institute Wiki that I’ve put some time into, but not, you know a huge amount of time particularly recently. Edits have been good, when they’ve happened. But all additions have come from people who I have asked specifically for their contributions. I don’t think this is a bad thing but this experience does run counter to the “throw up a wiki and people will edit it” mindset.

I’ve started (or restarted?) [a wiki that I set up for the OuterAlliance][oa-wik]. You can find out more about the OA there (as it gets added) or on the OuterAlliance Blog. Basically, O.A. is a group of Science Fiction writers, editors, and critics (and agents? do we have agents?) who are interested in promoting the presentation and visibility of positive queer characters and themes in science fiction (literature).¹

In any case, the group needed a wiki, and unlike the C2 Wiki, the people who are likely to contribute to this wiki are probably not hackers in the conventional sense. As I’ve sort of taken this wiki project upon myself, I’ve been trying to think of ways to ensure success in wikis.

Ideas, thoroughly untested:

Invite people to contribute at every opportunity, but not simply by saying “please add your thoughts here.” Rather, write in a way that leaves spaces for other people to interject ideas and thoughts.

Create stubs and pages where people can interject their own thoughts, but “red links” (or preceding question marks in my preferred wiki engine) are just as effective as stubs in many cases. The thing is that wikis require a lot of hands on attention. While stubs don’t require a lot of attention and maintenance, they require some. My favored approach recently is to make new pages when the content in the current page grows too unwieldy and to resist the urge to make new pages except when that happens.

Reduce hierarchy in page organization unless totally needed. You don’t want potential collaborators to have to thing very much about “where should I put this thing.” The more hierarchy there is the greater the chance that they’ll have to either think about it and/or that they’ll not find a place to put their contribution and then not contribute. This is undesirable.

Hierarchy is problematic for most organizational systems, but in most wiki systems, it is really easy and attractive to divide things into lots of layers of hierarchy because it makes sense at the time. The truth is, however, that this almost never makes sense a couple of weeks or months down the road. Some hierarchy makes sense, but it’ll take you hundreds of thousands of words to really need 3 layers of hierarchy in most wikis.

Leaders and instigators of wiki projects also, should know that creating and having a successful wiki represents the output of a huge amount of effort. There’s a lot of figuring out what people mean to say and making sure that their words actually convey that. There’s a lot of making sure people’s comments really do belong on the page where they put them. And more often than not leaders put in the effort to write a huge amount of “seed” content as an example to contributors in the future. It’s not a bad gig, but it’s also not the kind of hting you can just sit back and let happen.

Other thoughts? Onward and Upward!

It’s an awesome group, and a useful and powerful mission, and I think the OA has learned a lot from, and is well connected to some of the activity around anti-racism, that’s been lingering in science fiction over the last year to eighteen months as a result of the “RaceFail” hubbub of a year ago. The fact that there’s this kind of activity in and around Science fiction is one of the reasons that I love being a part of this genre. ↩︎

Notifications and Computing Contexts

Thu, 03 Dec 2009 00:00:00 +0000

Maybe it’s just be but I think notifications of events on our computers suck. On OS X there’s Growl, and GNU/Linux desktops have the libnotify stuff, and I’m sure there’s something on windows, but I don’t think this really addresses the problem at hand. Not that the software doesn’t work, becuase it mostly does what it says it’s going to do. The issue, I think is that we need, or will very shortly need, much more from a notification system than anything around can handle.

Lets back up.

I don’t know about you but there are a lot of events and messages that I get or need to get, including: new mail, some instant messages, mentions of cerain words on IRC (perhaps only in certain channels), notifications of when a collaborator has pushed something to a git repository, updates to certain RSS feeds, notifications of the completion of certain long-running commands (file copies, data transferes, etc.) and so forth. I think everyone likely has their own list of “things it would be nice if their computer could tell them about.”

The existing notification systems provide a framework that enables locally running applications to present little messages in a consistent, and unified manner. This is great. The issue is that for most of us the things that we need to be notified of aren’t locally running. At least in my case, instant messaging, IRC, git, and the key RSS feeds that I want to follow aren’t “locally running applications.” And to further complicate matters, no matter how your slice things, I use more than one computer, and in an ideal world it would be nice for one machine to know what notifications I’d seen on another computer when I sat down. In other words, my personal notification system should retain memory of what it’s shown me and what I’ve awknowladged across a number of machines.

That doesn’t happen. At least not today.

I have a few ideas about the implementation that I will probably cobble together into another post, and I’d love to hear some feedback if any of you have addressed this problem and have solutions.

It strikes me that there are two larger themes at work here:

1. Personal computing events occur locally and remotely, and notification systems need to be able to seemlessly provide these kinds of notifications. While I think a lot of the hype around clound computing is--frankly--absurd, it is fair to say that our computing is incredibly networked.

2. People don’t have “a computer,” any more, but rather several: phones, desktops, “cloud services,” virtual private servers, and so forth. While we use these systems differently, and our own personal “setup” are often unique, we need to move between these setups with ease.

These two shifts, networked computing and multiple computing contexts, affect more than just the the manner in which we receive notifications. But really, I think, outline the ways that the way we use computing has changed in the past few years. There’s a lot of buzzwords around this shift, in the web appliacation and cloud computing space particularly, and I don’t think that the “hipster”/“buzzword” experience is widely generalizable. It’s my hope that these conclusions are more widely applicable and useful: both for the development of a notification system that we need, and for thinking about application development in the future.

Like I said above, I’d love to hear your thoughts on this subect, and perhaps we can work on collecting thoughts on the Cyborg Institute Wiki. Take care!

I Didn't Write A Novel Last Month

Wed, 02 Dec 2009 00:00:00 +0000

I didn’t write a novel last month.

I’ve always been a bit of a contrarian about NaNoWriMo, the project where writers and people who don’t think of themselves as writers attempt a novel writing sprint during the month of November. Well, it’s a 50,000 word “novel,” which is in the end a bit short for a novel and a bit too long for a novella, but we’ll call it a novel for the sake of simplicity.

My basic gripe is that NaNoWriMo is a fun gimmick for people who aren’t used to turning out huge volumes of writing every day anyway, it doesn’t do a lot to really ensure or guarantee success. At it’s heart NaNo has a very democratizing idea: anyone can write a novel, I fear that it does more to impede success than encourage it.

My reasoning:

November is the worst month, with the holidays and the time change, potentially. November also tends to be bad for people who are in school or who teach school, as the semester draws to an end. December might be even worse in these respects. Here are some arguments for other months: February or March, (in the northern hemisphere) are cold months where you just want to stay at home, and what better time to write a novel? May is upbeat and fresh. June has no major holidays and rides a wave of Summer euphoria. Defenders of NaNo say “There’s no good month, so we might as well use November.” I reject this logic. Picking a bad month of the year can do a lot for the success rate of the people who attempt the project, I figure.
The novel is too short. While it would probably also decrease success rate to make the NaNo Novel a publishable length (60-80k or so), I think having people end up with a piece that’s sort of unusable in the “real world” can be discouraging as well. If they get something they like from NaNo, and they want to publish it, they have to write at least another ten thousand words and possibly as much as 50 thousand words. Digital distribution helps these things a bit, but the size is an issue.
What “real writers” do, is write every single day. The trick to being able to be a writer isn’t the ability to turn out a quantity of prose on demand. It’s the ability to sustainably work on projects all year round. To turn off the internal editor long enough that you get something on the paper, and then turn it back on so that “something” doesn’t suck. They’re able to take experiences and turn them into texts.
I’m not sure that the fetishization of the novel form is particularly productive. I think there’s a lot of power and future in shorter forms. For learning how to be a writer, writing shorter works is probably a more effective way to learn to tell stories and create characters anyway.

Having said that, congrats to the people who did NaNo. Keep writing. You’ve probably found a few extra hours in your day that you didn’t know you had. Keep writing and doing awesome things with that time. And if you’re a huge fan of NaNo, don’t worry too much about me, I’m just an ornery guy with too many opinions.

It’s true: I’ve been working on a Novel for more than a year, and while I’m closing in on the end of the draft. Its now done yet. Soon, perhaps. Also, I think I should probably do some blogging here about learning to write, and teaching writing given that I got here by way of a strange path and feel so strongly about these things.

Window Sizes in Tiling Window Managers

Tue, 01 Dec 2009 00:00:00 +0000

There’s an issue in tiling window managers, that I think a lot folks who are used to floating window managers never expect. I wrote a post to the Awesome listserv a while back explaining this to someone, and it seems to have struck a chord (I saw the post linked to last week). I thought I’d write a brief post here to explain what’s going on in a more clear and general way.

The Problem

When tiled, windows don’t seem to take up all the space that’s available to them. This creates weird “gaps” between windows. But only some windows: Firefox is immune to this problem, while terminal emulators like xterm, and urxvt, and gVim, and emacs get all funky.

What’s Happening

The application that are affected by this draw their windows based upon a number of fixed width columns. We’ll note that terminal emulators, as well as GUI versions of programmer’s text editors like vim and emacs, all used fixed-width fonts and often allow you to set window sizes based on the number of columns (of characters).

As a result, these applications are only able to use space on the screen in increments of full characters. Most of the time, in floating window managers, we never really notice this limitation.

In tiling window managers you do notice, because the window manager forces the windows to use all available space, except in some windows it leaves these weird gaps at the bottom and right of the window. Sometimes the gaps end up in the window, as unusable buffers, and sometimes they end up between windows. It looks funny, pretty much no matter how you slice it.

What You Can Do About It

The truth? Not much.

The Awesome Window Manager, by default shows the gaps between the windows. I always found this to be the “more ugly” option. You can alter this behavior by searching your configuration file for size_hints_honor and making the line look like this:

c.size_hints_honor = false

This tells Awesome to ignore windows (client’s) when they say “I want to have these dimensions.” It doesn’t fix the problem but it does get rid of the gaps.

The real solution is to tweak text sizes, fonts, and any buffering elements (like a status bar, mode line, or widget box), and window borders so that the windows aren’t left with extra space that they don’t know how to cope with.

By real solution, I really mean “only option:” it’s really impossible to get all of your fixed width applications to have exactly the right number of pixels. You can get close in a lot of situations, and I’ve always found this to be much less annoying than using floating window managers.

The Original Post

Just for giggles, I’ve included a quoted portion of what I posted original to the listserv on the topic.

The one big of information that might be important: The urxvt terminal emulator, when not “honoring size hints,” is unable to really properly draw the “extra space” with the proper background. I suspect this is a bug with the pseudo-transparency system they use. As a result there are often a few pixels with the background in an inverted color scheme. Same problem as above, but it looks funny if you’re not used to it.

What’s happening is that urxvt (like many terminal emulators) can only draw windows of some specific sizes based on the size of the characters (eg. x number of rows, and y number of columns.) So while you may have larger and the equivalent of say 80.4x20.1, urxvt can’t do anything with this extra space.

If you honor size hints, the windows will end wherever they can, and use as much space as they can, but leave gaps between windows if the total space isn’t properly divisible. If you don’t honor size hints, the windows themselves take up the extra room (but they can’t do anything with the extra room, so they just leave it blank, and sometimes the transparency is a bit wonky in those “buffers”).

So there you have it. I hope this helps!

Of Dialectical Futurism

Mon, 30 Nov 2009 00:00:00 +0000

A while back, lets say, in August or so¹, I redid the design of this site and added a new subtitle: “Dialectical Futurism.” The dweeby, philosophy geek in me really enjoys this, just as an idea. As it’s sat at the top of the page, I’ve also thought more and more that the subtitle is actually a pretty good summary of what I’m trying to accomplish here. This post is an attempt to do two things:

1. Concisely summarize my “blogging project” and thus explain what dialectical futurism means.

2. To do a bit of a status update on the blog, as a sort of “self report,” of what I think seems to be working and what doesn’t.

Part One: What “Dialectical Futurism” Meas In Practice.

It’s always risky, I think for a self-claimed “Science Fiction” writer, to declare themselves a “Futurist” of any sort. Because of the genre’s link to the future, I think there the danger that people might think that we’re putting forth our stories as works of prediction.

While I think my interest in futurism comes from a similar place as the interests that drive my fiction, the practice of futurism (in the form of this blog) and the practice of fiction are very different. Ultimately, both are historical endeavors and futurism is tends to be much more tightly focused on the recent history. At least for me.

Dialectical futurism, is about a conversation between me, and the past, and me and the possible future, it’s an attempt to synthesize a pragmatic view of what will happen, with an optimistic view of what I would like to happen. It’s about putting all of the topics I blog about like Open Source, Free Software, Cooperative structure, Economics, and the “New Media” in conversation with each other and seeing what kind of cool innovative things happen.

Part Two: The Status of the Project

The project of being a writer is one of constant self improvement, I think. One thing that I didn’t mention in that post is that no matter how awesome you are as a writer, you’re always trying to get better at writing. There’s always some improvement to make, some short-falling in your ability to communicate that you’re working at improving.

I have the sense that I am getting better, if that’s a meaningful judgment. One improves as a writer, I’m convinced, by writing, and writing a lot, and as I write a bunch for this site and a bunch for work, and a bunch for other projects, I think I’m starting to get better. Also, I get a lot of feedback from coworkers on my writing, which I think has been helpful. Editors are a good thing indeed.

I think I’ve gotten better at figuring out how to write good blog entries--it still takes time, but I get into the grove more quickly. I’m getting better feedback, and I’m reasonably happy with where the traffic is. I mean, there’s always room for improvement, but things are headed in the right direction.

My short term goals are two fold:

To focus my energies on reading and improving my background knowledge in a number of areas. I want to be more contextually grounded in existing conversations regarding economics, anthropology, and cyborg-related materials.
To spend a lot more time on fiction writing. This means developing new habits, adjusting priorities, and spending some serious time making fiction projects work. So there.

We’ll see how this goes, and thanks as always for reading and putting up with me.

I’m just guessing here. I could go back and check, but August sounds right. ↩︎

Comfort Cheese

Fri, 27 Nov 2009 00:00:00 +0000

I, rather innocently, posted something to identi.ca / twitter / Facebook. The canonical text is:

signs you’ve spent too much time in the upper midwest: sometimes when stressed, you impulse buy cheese

I spent three years in Wisconsin--southern Wisconsin--for college. Ever since then I’ve related to cheese differently.

So here’s what happened:

I was shopping, Mostly minding my own business when I passed the cheese display. Not the fancy cheese display, you know, but the one next to the designer coffee creamers, and the pre-made baking doughs.

And, pretty much before I realized what had happened, I scooped up a half pound of store-brand mozzarella. I don’t think I even stopped the cart.

In the parking lot, I realized what had happened. I had cheese at home. A half pound of Munster--and pretty nice Munster I thought, though I’ve not met a Munster I didn’t like--waited for me in the fridge at home. The reason why I got it? Easy:

A few weeks ago, I was in Philadelphia staying with a friend who grew up in the Twin Cities and we had the most amazing mozzarella for dinner. I looked up after my second slice and said. “Wow, we’re so Midwestern.” It was undeniably true. A friend from the east cost, immediately confirmed for us that she thought we were weird. But it was amazing cheese. And we didn’t

As for my little block of cheese: it too was amazing. Not quite as good, I suppose, but it was refreshing and in a weird way it reminded me of home.

The people on Facebook also responded with a rather active little thread. Morris dancers from Minnesota. Contra Dancers from St. Louis. Classmates from College. Perhaps my experience wasn’t uncommon.

Cheese. Who would have thought it.

Building the Argument

Tue, 24 Nov 2009 00:00:00 +0000

I was talking to my grandmother (Hi!) last week, as I do most weeks, and we discussed the blog. She’s been a regular reader of the site for many years, and lately, we’ve enjoyed digging a little deeper into some of the things that I’ve written about here. She said, I think of the Owning Bits, and I agree, that it sort of seemed that I was building something… more.

But of course.

I don’t know that I’ve connected all of the dots, either in my head or on the blog, but I think that the things I’ve been blogging about for the last year or so are all conected, interwoven, and illuminate incredibly interesting features of each other when considered as a whole. There is “something building” here. To recap, so that we’re on the same page, the nexus of subjects that I’ve been milling over are:

Free Software and Open Source Software Development.

I’m interested in how communities form around these projects, how work is accomplished, both technically, and organizationally. I’m interested in how innovation happens or is stifled. How the communities are maintained, started, and lead. From a social and economic perspective there’s something fundamentally unique happening in this domain, and I’d like to learn a lot more about what those things are.

This topic and area of thought have taken a backseat to other questions more recently, but I think it’s fundamentally the core question that I’m trying to address at the moment. I think that I’m going to be making a larger point of addressing open source methodologies in the coming weeks and months as part of an attempt to pull things back together. I think.

The Information Technology Industry.

I started writing about the IT industry because I found itreally difficult to think about Free Software without really knowing about the context of free software. One really needs to understand the entire ecosystem in order to really make sense of what open source means (and doesn’t mean.) Particularly in this day and age. Initially I was particularly interested in the Oracle/Sun Merger, and the flap around the ownership of MySQL; but since then, I think I’ve branched out a little bit more.

I’ve tried very hard to not frame the discussion about the IT industry and open source as a “community” versus “enterprise” discussion, or as being “free” versus “non-free,” or worse “free” versus “commercial.” These are unhelpful lenses, as Free Software and Open Source are incredibly commercial, and incredibly enterprise-centric phenomena, once you get past the initial “what do you mean there’s no cost or company behind this thing.”

The Cyborg “Moment,” or the way people actually use computers and technology, and the ways that people *might* use technology in the future.

In the same way that thinking about the IT Industry provides much needed context for properly understanding why “open source communities” exist and persist: thinking about how we actually use technology, how we relate to techno-social phenomena, and how these relationships, interfaces, and work-flows are changing. Both in changing response to technology, and changing the technology itself. It’s all important, and I think the very small observations are as useful as the very large observations.

In some respects, certainly insofar as I’ve formulated the Cyborg Institute, the “cyborg” moment can really be seen as the framing domain, but that doesn’t strike me a distinction that is particularly worth making.

Corporate Law and Structure and Cooperative Economics.

Interestingly, my discussion of cooperatives and corporate organization began as a “pro-queer rejection of gay marriage,” but I’ve used it as an opportunity to think about the health care issue, as a starting point in my thinking regarding EconomyFail-2008/09. The economics of open source and Free Software are fascinating, and very real and quite important, and I found myself saying about six months ago that I wished I knew more about economics. Economics was one of those overly quantitative things in college that I just totally avoided because I was a hippie (basically.)

While it could be that my roots are showing, more recently I’ve come to believe that it’s really difficult to understand any social or political phenomena without thinking about the underlying economics. While clearly I have opinions, and I’m not a consummate economic social scientist, I do think that thinking about the economics of a situation is incredibly important.

New Media and the Publishing Industry.

I’ve been blogging for a long time. And I’m a writer. And I want to write and publish fiction as a part of my “career,” such as it is. As you might imagine these factors make me incredibly interested in the future of publishing of “content,” and of the entire nexus of issues that relate to the notion of “new media.”

Creative Commons shows us that there has been some crossover between ideas that originated in the “open source” world with “content” (writ large.) The future of publishing and media is a cyborg issue, an ultimately techno-social phenomena, and thinking about the technology. that underpins the new media is really important. And of course, understanding the economic context of the industry that’s built around content is crucial.

So what’s this all building to? Should I write some sort of monograph on the subject? Is there anyone out there who might want to fund a grad student on to do research on these subjects in a few years?

The problem my work here so far--to my mind--is that while I’m pretty interested in the analysis that I’ve been able to construct, I’m not terribly satisfied with my background, and with the way that I’ve been largely unable to cite my intellectual heritage for my ideas and thoughts. I never studied this stuff in school, I have a number of books of criticism, potentially relevant philosophy, and important books in Anthropology (which seems to fit my interests and perspectives pretty well.) I’m pretty good at figuring things out, but I’m acutely aware of a lacking in my work of reference, methodology, and structure. As well as of any sort of empirical practice.

So maybe that’s my project for the next year, or the next few months at any rate: increase rigor, read more, consider new texts, pay more attention to citations, and develop some system for doing more empirical work.

We’ll see how this goes. I’d certainly appreciate feedback here. Thanks!

I Am a Writer

Mon, 23 Nov 2009 00:00:00 +0000

It’s a weird thing, this “being a writer” stuff. I’m sure I’ve written about this here to some extent. As a kid, I think--at least I tell myself now--that I wanted to be a writer. There was something about writing that mystified me and challenged me and had me totally entranced. I had a hell of a time with writing in high school, enough that I really shied away from formal training as a writer in college in almost entirely. I took two English classes in college, sort of (they were cross listed as something else), and I knew for sure that I wasn’t going to be a writer.

Right.

And then I got out of school, and something clicked. Actually, the revival of my blogging efforts that stuck hit during my final semester of college. And one thing lead to another and, here I am. I write this blog, that’s you know… fairly prolific. I have job where I write things day in and day out. I write fiction a fair piece, though not as much as I might like.

One might think--I certainly did--that getting a job as a writer would put to bed all of my insecurities and doubts about being a writer. But it doesn’t. I’m not complaining, mind you, but it’s still weird.

Fundamentally, writers have a peculiar way of being in the world that is always a bit unsettling and alienating. Certainly we’re all different, and the experience of being a poet is different than the experience of being a technical writer is different from being a science fiction, but I’m convinced that there are some common features.

Writing, at least for me, is sort of about turning experiences into words. This isn’t some wishy washy practices of translating the feeling of moments into words; but rather a pretty simple observation about practices. No matter if I’m writing science fiction stories or systems administration documentation, my ability to write is always dependent upon doing things in the world and gathering experience. Without this, I run out of stored experiences, of “mojo” and my writing becomes flat and painful, if I can manage to write anything at all.

Now the writing part, after a while becomes pretty straightforward: Sit down. Figure out where you need to go in a given text and about how long you have to get there plus a few other variables, like voice and audience. And then you just sort of let the “experience,” part flow out onto the page¹ as you sort of mold the thoughts into the path you need to follow.

And this leaves the walking through the world part. It’s reflexive and feels normal, until you realize that you are instinctively collecting images, snippets of speech, moments, situations, little stories, and other bits of miscellany, in some master database in your head. Every conversation becomes an experiment in expressing an idea or a theory. It’s not so overt that it makes “living” difficult, or conversations awkward (though it does sometimes), but I sometimes have to remind myself that what’s going on in my head isn’t what’s going on in everyone’s head. No really.

Another problem with being a writer is that, everyone writes, or knows how to, at least in the abstract. Some folks don’t like it, and some folks aren’t particularly “skilled written communicators,” but we all know how to do it. This isn’t the case for a lot of professions, disciplines, or even hobbies. Not everyone can write a computer program, not everyone can knit a sweater, or cook a meal, or analyze great amounts of data, engineer more advanced agricultural technologies and crops, and so forth.

This creates some tension: since so many people know how to write, and yet most people don’t as a matter of course, there is some mystification around what writing requires. “What are you doing this weekend,” they’ll ask. “I was thinking of staying home and writing, and maybe going for a walk or two,” I say gleefully. “That sounds dull, and don’t you " they say. “Well, yes, but it sounds amazing. I wonder if I have enough food to get me through the weekend,” I say. Welcome to my life.

Despite all this I nearly always feel like a cheat and a fake. My fiction is totally unpublished, and I’m not sure I’m writing in the correct direction, or doing the right things to be able to really have an active fiction writing career in the next 7-10 years. I’m constantly unsure about the success of the blog: it’s self published, sometimes it feels like I don’t have any reviewers outside of friends and readers-who-have-become-friends. And while I’m quite pleased and proud of what I’ve been able to accomplish at work, and I think that We/I’ve been able to be pretty successful, I’m really part of a team and what I write is so terribly niche.

I think that’s the other part of being a writer that’s so strange. No matter how much of it you do, no matter how much of your income is the direct result of the way you commit words to paper (or emacs buffer): you’re still just another hacker.

But maybe this is true for everyone. I can accept that. I hope you all have a good day.

Wow, I used a dead-tree metaphor. Have no fear, when I say “page” I really mean emacs buffer. ↩︎

technology task list

Fri, 20 Nov 2009 00:00:00 +0000

Thought I’ve gotten away from it a bit in recent months, tychoish has a long history of being an outlook of lists of various things. While I’m not sure I want to post all of my lists for everyone to point and laugh at, the following might be worth exploring.

This is a list of things I need to get worked out with my new computer, with technology in general. I post it both because I need an excuse to do a little brainstorming, and because it might be nice to get a little feedback from you all. Without further ado:

Get USB Mounting/Auto-mounting to work more smoothly.

I use USB mass storage devices so rarely that I’m totally oblivious as to how I should go about setting this up with Arch Linux.

Reformat and server-ify my desktop.

Since I’m basically not using my desktop as a desktop anymore… and there are some things that just don’t work… and there are no files left on it that I don’t have backed up elsewhere… I think it’s time to do a system wipe. I want to put Arch on it. I had thought about putting Xen on it and using virtual machines, but I’m now in a place where the increased management burden of that would outweigh the benefits of that. So I think I’m just going to set it up like a server, (but I suppose setting up a lightweight desktop wouldn’t be a big stress). Mostly I think having a server at home will be useful for testing, development, and othersuch projects. In any case, it’s not terribly useful as it is.

Reorganize my music collection (now on laptop).

I copied over my music collection and while I’ve had a bunch of luck with mpd, I need to spend some time reorganizing the music. It’s on the list, and I shall do it.

Straighten out the situation with my external hard drive.

Yeah, no clue here. I hope it’s alright. I’m going to try and use the Mac at work to see if I can’t make it work better. I may crack the enclosure and put it in my desktop once that’s in better working order.

Acquire accessories:

There’s stuff I’ve had on my shopping list for a while. In no particular order:

A more suitable laptop sleeve.

As it turns out, I have this backpack that’s great for lugging stuff around, but it’s bigger than I need most of the time, and the laptop padding could handle my 15.4" PowerBook back in the day. Current laptop is quite small, so it’s sort of overkill. This is lower priority.

Additional power adapters.

The battery on this puppy is amazing. Having said that it’s nice to have a power adapter that can just live in my bag so I don’t have to fuss with repacking the power adapter every time I leave somewhere. I think one at my desk at work, one for home, and one for my bag is my usual complement and Lenovo power adapters are a lot cheaper than mac ones…

Wireless access point for home.

Somehow I don’t have one. Oversight. Must procure soon. The thing is that I have an ancient 100 foot Ethernet cable that seems to do the job pretty well.

Sort out Sleep/Wake Cycle

I think I mostly have this one sorted out. Basically, I had problems with the new laptop freezing when waking up from suspend/sleep when the network state upon return was different than when the laptop went into suspend. A little tweak to the ACPI event script, and everything seems to be in order.

Write Network Management Triggers

I’m using the preferred network manager suite for Arch Linux (e.g. “netcfg”) and it works great, except it’s sometimes a bit bothersome to mange things, when I think it ought to just work. So I think I have a solution: create shortcuts and triggers in the window manager to get network stuff working a bit more smoothly. Now I just have to make it work.

Tinker with StumpWM contrib packages

Once I got Stump WM working and set up, I mostly abandoned it. There’s all sorts of cool lisp things in the contrib/ directory that I haven’t tinkered with. Well, except for mpd.lisp, and even then not terribly much. I think I’d get something out of playing with those and so it’s on the list.

Figure out what to do with the x41.

I’m not sure. The old laptop works, and I feel like I should do something with it… But what?

On Publishing

Thu, 19 Nov 2009 00:00:00 +0000

I’ve been thinking about publishing and the publishing industry of late. I’m sure some of it is related to my wanting of a kindle and my resulting thoughts on consolidation, and maybe some small measure of it has to do with the fact that sometimes it easier to think about publishing and the future of publishing than it is to think about ones own creative projects. So be it.

First, “what is there to think about?” you ask? Well, lots of things: I’ve written about wanting a kindle, and some thoughts about consolidation, and finally some thoughts on digital publishing More recently I’ve been thinking more about the “work” of publishing and content creation, apart from the changing business models and technological context.

Publishers (of any kind, and their editorial departments), by contemporary convention are responsible for reading through the slush and figuring what’s good and what not. Ideally publishers are the stewards of taste, and the people who figure out whats “good” and what people want to read. On some fundamental level, publishers are curators. The second main function of publishers are as the provider and organizer around services. Publishers contract with copy editors, with design and layout people, they get the cover art, they do promotional work, and the million other things it takes to turn a manuscript into a book.

As the traditional publishing model has… deteriorated, I think a lot of people have been interested in figuring out “what happens next?” myself as much as anyone. Having said that, the way in which the traditional publishing model has deteriorated has shaped how we think about what comes next. This makes sense of course, but I want to challenge myself to think about things more broadly (and you, dear friends as well, but I’m sure you’ve already figured this one out.)

I mean, it’s not like the old media died in a day. The blogging phenomena started, and writers/etc. were able to promote their work directly in ways that they hadn’t managed to before. Margins on book sales went down, which has cut into promotional budgets (as much as anything). Also, thanks to developments in technology the size of most first runs is much smaller than it used to be. This is probably a good thing, but it also means that the capital investment on new authors and books is much less than it used to be. …and the end result of this is that we’re prone to seeing publishing companies as “Authors Services” companies.

As a model for “what comes next,” services for authors is a huge part of what we need from the publishing companies. Centralizing and connecting authors with people who can provide big-picture editing, with people who do copy editing and proof reading, with people who do the cover art and layout of the book itself, and with people who can do the promotional work, getting the book plugged into the distribution channels. These are real needs, that aren’t going to evaporate any time soon.

But what about the editorial and curatorial roles of publishers? What about the branding associated with publishing houses? I think there’s probably some future for critical discourse in blogs and in digital forums, which will provide some of these functions, but that’s not the full answer, and I’m not sure what the full answer is.

As I wrote, earlier, I think figuring out some sort of subscription system to support content creation and distribution. I think having the economic superstructure in place, or at least worked out conceptually is really important before we start working on new technologies, like ebook readers, and digital content distribution channels.

It’s an interesting time to be around, that’s for sure.

Owning Bits

Wed, 18 Nov 2009 00:00:00 +0000

Some First Principals:

It is difficult, and likely impossible to technologically restrict the duplication and redistribution of digital resources. In other-words, digitally accessible resources will never be scarce.
Creators of content (music, literature, software) should be reimbursed for their work, and there should be business models that support these kinds of pursuits. In other words people should be compensated for the creation of content in a viable and sustainable manner.
Information probably does want to be free. Creators of information, should want their information to be “free” because the information that has the most power and influence is that which is most accessible.
The conveyance of physical objects (books, etc.) is a source of concrete value. Physicality is not the only basis for economic exchange, of course, but it’s a good place to start.

Questions and Answers:

What is the “content problem?"

I suppose the core of the problem with content these days is that we don’t have a good set of business models that can support the creation of new content in an ongoing sort of way. The industry around content is unstable and in flux: newspapers are hemorrhaging money and it doesn’t seem likely that they’re going to be able to do anything other than (maybe) prolonging the amount of time between now and when they collapse. Some paper companies might surrivive, but the consolidation and “flufification” of a great many newspapers, doesn’t seem to have a great deal of long term potential.

Same thing with the book publishers. There are some that seem to be doing interesting things. Tor.com seems to be a good example of a step in the right direction. And maybe there will be an ebook platform that makes sense, or maybe we’ll see some sort of revival in niche-booksellers that will revive an interest in book collection.

But the bottom line seems to be that we need to find some better way to support the creation of content. Because what we have now doesn’t seem to fit the bill. And whats on the horizon, doesn’t seem to be much better.
Isn’t content a horrible word for this?

I confess that the word “content” makes me a bit sick from time to time. Not only is it awkward to lump the concerns of musicians, academics, writers, journalists, and perhaps even software developers all in one label. I’m not even sure if the concerns of content producers as a whole, if we can address these folks as a group, are particularly aligned.

Some academics use the term “knowledge production,” to refer to the core output of their work, and I’m using content in a similarly broad context. Writing/Literature/Music, “Art,” and even thought this might be a tad bit unconventional I think there’s not much that separates the consumption of software and the consumption of essays (for example.)

There are also a number of different dimensions upon which we can think about content and the future of content: the experience of consumption; the process of generation; and the business models which support the creation and consumption of content.
Owning Bits? What do you mean?

I said, a few weeks ago of the whole DRM issue, that I thought “we needed to get away from the whole ‘owning bits’ metaphor for content distribution.” The whole DRM thing that so many of us find so onerous would be mostly become a non issue if we dropped the pretense that when you download a song or a book or a movie that you’re “buying them.” If you’re just watching the bits for a while, who cares what the digital restrictions are? If prices are reasonable for content, who cares if you can only “have” a half dozen books at once? I think it would all work out. But maybe that’s just me.

The questions that result, however are much more interesting: What does it mean, socially and politically, if we don’t own information? What is reasonable and sustainable pricing? What kinds of distribution schemes make sense?
Besides the general “fear of copying” that has heretofore plagued the content industry, what new technological challenges might the content industry face in the mid-to-short term future?

One of the major issues that I think we’re going to have to deal with is the fact that digital information systems are too mutable.

While this flexibility gives us lots of very powerful information resources, like Wikipedia and the ability to correct flaws in digital versions, it also means that Amazon can remove books from the Kindle at will. Furthermore, it means that creators and publishers can (attempt) to “take back” content if they have second thoughts about it. The mutability issue is obviously a mixed bag, but I think the most useful information and the most free information will have some sort of versioning information.
What are the business models that will support content in the in the future?

The one downfall of Project Xanadu is that it pushed forward an idea of “micropayments,” and the idea if we charge a la carte for content and have the per unit cost for content low enough that somehow it’ll all work out. The problem with this, however, is that the psychological border between “free” and “not-free” is much larger than the border between “a few cents” and “a few dollars.” The end result: micropayments keep failing.

It’s a shame that this idea was the most successful idea to proliferate from Xanadu.

My current bet is that some sort of subscription model is likely to win out. Pay a few dollars a month, and get access to some reasonable quantity of content. Have different levels of subscription to meet different needs and demands, and I think there’s potential there.

The other prevailing model is the “rockstar” model, where the content creator goes on a tour and uses honoraria and merchandise sales to offset the cost of content creation. We see this both for authors who tour to support books as well as the musicians for whom I’ve named the model. It works, it focuses the transaction of physical objects.
You seem to like subscription models. What are the implications of a shift toward subscription models in terms of the way people relate to the information (music, writing, etc.) that they consume?

I’m not sure. I think creating a technological limitation which stores version information in some sort of immutable index. I don’t think publishers will really go for this.

I think subscription models may also revive (in part) the interest and power of the physical-object-market. In the way that Libraries don’t cannibalize booksales (and may encourage and support the sale of real books,) I think digital content subscriptions could have the same effect on content.

As always I look forward to your thoughts and responses to these questions. See you in comments!

The Web Application Layer

Tue, 17 Nov 2009 00:00:00 +0000

This post is an attempt to ask “what next?” in the world of contemporary application development. I’m disturbed by the conveyance of applications in this format. This is not news to popular readers, but, rather than complain extensively about the state of the contemporary technology, I think it would be more productive to muse on possible improvements and some of the underlying structural concerns in this space

In No Particular Order…

Today, we routinely design and implement user interfaces in HTML and JavaScript these days. I’m not convinced that HTML, or any XML based format is really all that good for conveying well formatted structured text, much less pixel perfect graphic design and application interface.

Lightweight text markups like Markdown, reStructuredText (for all it’s warts), and Textile are human readable, provide structure, and convey text well. Furthermore, it’s very possible to efficiently translate them into very high quality output formats, including XML formats and LaTeX.

One of the driving forces behind the convergence on “web-technologies” is that JavaScrpt/HTML/CSS are all thought to be “cross platform” technologies. It doesn’t matter if you’re running on a Mac, or a PC, or UNIX system, if it has a web browser it’ll run there. The web application movement realizes the “write once run everywhere,” notion that Sun attempted with Java in the 90s (except, that Java never really worked for that.) Except that every browser implements JavaScript/HTML/CSS in a different way which means, that it’s really “write once and tweak it to death so that IE/Firefox/Webkit don’t break.” There are some things (like jQuery; HTML5) that make this better, but the browser market is dirty and browser makers will never be incentivized to comply with the standards.¹

RESTful APIs² are, I think, leading to more desktop applications. Or at least making them more possible. It used to be very much the case that if you wanted web-connected data you had to go to a website. Now, if you want data from the Internet, in most cases it can be gotten in an easy to process formant (i.e. YAML/JSON) and then folded into a desktop application. In addition to “rounded corner power” and “social media,” the biggest impact of “Web 2.0” has been the increasing awareness and interest the API[^quality] in the more general public.

Adobe AIR is a wonderful idea. Even smaller lightweight devices like smartphones and netbooks are so powerful that it doesn’t make a lot of sense to have them operate as such “dumb” clients. Conventional web development has developers cobble together server applications that put together content and then chuck it off to the client for rendering. With APIs (see above) it doesn’t make a lot of sense to leave all the heavy lifting on the server. Adobe understood this with AIR, the problem with AIR? It’s wildly proprietary, applications look out of place on every platform, and performance is miserable relative to “real” native applications. Its a great idea, and I’m terribly interested to see what comes next in this space.

I’m grumpy about HTML 5 because I remain unconvinced that web standards are really a viable way of regulating sane design and development practices. It also seems too likely that HTML 5 solves the problems we were having 2 years ago, rather than the problems we’ll have over the next 10 years. Also, I think this world needs a hell of a lot less XML, in any form.

What are your thoughts?

[^qualit]y]:I’m not sure there is any singular aspect of the whole “Web 2.0 thing” that is unequivocally bad or good. I think on the whole web design is better now than it used to be, but “rounded corner power” at this point all looks the same, and it’s really difficult to achieve in a clean technological sort of way. And the web has always been social; so while that’s not new, it’s nice that the web has caught on even if the whole “social networking silo” phenomena is less than desirable. The same thing goes for RESTful APIs: it’s great that data is more accessible, but it sucks that APIs can be so proprietary and I’m not convinced that HTTP is the “right” or “best” protocol for this technology. But these things happen.

You may think that I’m simply being pessimistic, and you might say that IE 7 and 8 are a huge step in the right direction and I think that this might be true, but the only reason to create and maintain a browser (to my mind) are: masochism, to get people to use your search engine, and to be able to implement special proprietary (and non-standard) features. The competitive advantage comes from the unique enhancements that a given browser is able to offer over the other browsers in the market. For a while (e.g. 1999-2007?) the more standards compliant a browser was the better pages looked in it. I’m not sure that will continue to hold true. ↩︎
This is a simplification, but lets think of this as the obligatory API that all web-services provide today: from twitter to flickr, to YouTube, and beyond. These allow programmers to connect to the service using the HTTP protocol ↩︎

Technology Update

Mon, 16 Nov 2009 00:00:00 +0000

I fear I’ve been posting too many posts in the vein of “so here’s what I’m up to folks,” rather than you know writing about things that may be interesting to folks out there in Internet land. Nevertheless, here’s another slightly more technical post.

I’ve mentioned in passing a few times over the past few months that:

I’ve been interested in shifting to Arch Linux. While I’ve been running Arch in a VM on my work desktop, I have been quite slow to move additional machines over to Arch. It’s not for lack of wanting, but I have a hard time disrupting something that works when it’s already working.
Also on the software front, I’ve switched to using the Stump Window Manger, and while I’ve talked a bit about this on the blog, I’ve done virtually no reporting of my ongoing progress with this.
I missed the days when I only had one computer and it went everywhere with me. While I like having all this computing possibility around, I’m moving around enough these days that it doesn’t make sense to be tied down to a desktop. I like sitting on the couch and writing, and I like being able to go off for a weekend and be able to work on the projects that I really need/want to work on. That’s hard when you have a desk and an “office.”

This post provides updates with regards to these subjects.

Moving to Arch Linux and StumpWM

A few weeks ago, I had this massive cascade of software issues. Mostly things were provoked by the switch to Stump. Basically, the issue was that Stump wasn’t embedded in all of the desktop frameworks that are so popular these days, there were a number of system resources that just didn’t work with the new Window Manager.

The thing was that my systems were running a terribly hacked up version of Ubuntu. I was running weird kernels, I’d mostly given up on the display managers, and the systems were just messy. So the problem wasn’t so much with Stump, as it was with the way that Ubuntu packages and manages certain aspects of the system inside of desktop functionality. I’m thinking specifically of the ways that networking and sound are managed by dbus. If that didn’t make sense to you, don’t worry.

Since the chief problem boiled down to “this system is too complex for me to be able to manage,” and it no longer became an effective use of my time to maintain the system as it was… I wiped everything and finally put Arch on the laptop.

And it went on smoothly, and everything worked. Arch is a tinkerers distribution, there’s no doubt about that. Since I did have Arch experience it wasn’t a terribly traumatic experience. It took a little while to figure out how to make Suspend and Resume work (i.e. for the laptop when the lid closes,) and manually managing network connections isn’t incredibly straightforward until you get the hang of it, but it all works now. And I couldn’t be happier

The Experience of StumpWM

This isn’t really a full report, but more a note to say that my brain has really adapted to Stump, and I’m quite happy with the experience. Stump doesn’t in and of itself increase the ways I’m able to be productive, but… I do think that I work more efficiently when using Stump.

There’s still a lot left to be done with regards to the tweaking of stump for me. I need to play some more with the MPD (music player) integration, and there are a number of other contributed Lisp packages that I really want to play with. Also, I finally figured out how the Key Binding Map works when I had gotten my basic keybidning needs taken care of and I haven’t touched it since then. Now I know how I use the system and I’m ready to tweak things again, but I haven’t gotten around to it.

Additional thoughts regarding Stump, from a more “objective” perspective: it is incredibly stable, and while it’s not blindingly lightweight, it lives in 20 megs of ram and that’s about it. I never need to restart the window manager or X anymore, and that’s kind of nice.

So in short, the Stump WM is a great thing and I need to write a bit more about the actual using of it. But first I need to do a little more tinkering. Because I’m like that.

The Consolidation of the Gear and Laptops

What a strange heading. In any case. I gave in and bought a new laptop, last week. I found a great deal on a used Lenovo x200 with great specs, and I thought that it would solve the majority of my issues with my existing technology.

First, it was considerably newer than the laptop I have been using for most of the past year: more RAM, dual core system, bigger and faster hard drive. Second, it had all of the qualities of the old laptop that I adored: it’s a 12" laptop which means very portable without making sacrifices, and it forgoes a trackpad for a “ThinkPad Nipple” for a mouse. Finally (and perhaps most importantly,) the screen resolution is 1200x800 (up from 1024×768) which makes it possible to comfortably tile two windows next to each-other at once. This is the same resolution as my 15" PowerBook G4 (and I think all of the 13 inch MacBooks). It’s a good size, and I was really aching for the increased screen space.

It turns out that all of these concerns were addressed fully with this new system. The screen is perfect, it’s peppy. it’s also nice to return to the modern computing world. I continue to be mightily impressed with the build quality, design, and functionality of IBM/Lenovo hardware.

My computer consolidation isn’t yet complete: my desktop hasn’t yet been backed up and converted to Arch, but it’s getting there. I’m also not quite sure what happens with the old laptop. I’m thinking of keeping it around as a spare, but if anyone has a need for a really awesome ThinkPad x41 they should be in touch.

Onward and Upward!

The Dangers of Consolidation

Fri, 13 Nov 2009 00:00:00 +0000

I mentioned in an earlier post that I thought Barnes and Noble was largely responsible for the ongoing and impending collapse of the publishing industry, and that’s just the sort of thing that I couldn’t leave a lone without a little bit of further pondering.

The assertion is that Barnes and Nobel, and Borders particularly as they competed for near total domination of the local-book retail market, forced a consolidation of the publishing industry at the very moment when the worst possible thing for publishing was consolidation.

Consolidation allows an operation to make a bunch of money quickly. The mechanics of this are pretty simple, after all. When yo consolidate you can cut all sorts of mundane expenses, from the physical costs of maintaining parallel operations, to hard costs like printing and shipping costs that can benefit from collective organization.

Amazon had a role in the consolidation of sale of books, certainly, but Amazon has always been a distribution and data company, primarily. Their strategy is to find a way to turn a profit on the sale of goods, any goods: they do this by having a complete inventory of everything and levering a lot of data concerning buying habits and browsing habits to make sure people who are shopping find something to buy.

Where Amazon’s limiting factor is connecting people who want to buy things (books) with books they might like to buy, the “traditional” book sellers, are limited by the amount of shelf space they can use to display and promote books. So they edged all of the little booksellers out of business by having huge stores and coffee shops and so forth, and then faced with too many books and not enough shelf space, they used their muscle to push the publishing industry toward increased consolidation and a “blockbuster” business model.

Blockbusters are how the movie industry works. Production companies make a bunch of movies, on the premise that if one or two turn a huge profit, they can afford to make a number of movies that flop or that just break-even. Hence the great power of reliable successes: another John Grisham novel, Return of the Mummy King VI, etc., the “copy-cat” phenomena and the erosion of the independent movie production business.

Book sellers were culpable as well--consolidation is attractive in what are essentially commodity businesses--and selling easily produced paper-based volumes is a commodity business. Maybe it would have happened anyway, but the undeniable market success of Barnes and Nobel is not, given what I can tell from where I’m sitting, a marker of success of the publishing industry as a whole.

And you know, when you’re a book seller, throwing the publishing industry onto the tracks before an oncoming train, to achieve some mildly impressive profits for a decade seems… not incredibly bright. And not the kind of thing that I’m interested in supporting or putting my faith in.

It strikes me that this “consolidation meme” is a common feature of unsustainable and inauthentic economy, and it extends beyond book retail into other failed and failing sectors of the ecomony.

Banks. Obvious here. The big banks lost track of the micro economics that make the macro economics go, and we got things like sub-prime morgages, because while they make sense from the consolidated-bank perspective they don’t make sense to people. Like, the John Grisham-esque Legal/Drama/Thriller book makes a lot of sense to the booksellers and the publishers, but most people can only really read so many of them before loosing interest.
Software. Microsoft’s production of windows makes a lot of sense if you’re a big company, but if you use computers in a specialized way, Windows is like an illfitting suit from Target. It works, but it’s uncomfortable and rough around the edges. There is general consensus that “The Microsoft Way” isn’t the best technological solution to the various problem, even among people who use it regularly (developer tools might require a slightly more complex investigation).
Your Example Here. Leave a note in the comments.

Thoughts?

On Wanting a Kindle

Wed, 11 Nov 2009 00:00:00 +0000

I have a confession to make. I really want a Kindle. Bad.

No really. I do. The DRM scares me, and I think the books are just the other side of “too expensive,” and because I come from a long line of “book collecting people” I think there are a lot of books that I would want to own in the paper. Furthermore, I have a great laptop for reading books (a small tablet), and I have a very long history of using small form computing devices (think palm pilots and pocket pcs) to read books. And yet, I returned to paper a few years ago, and don’t feel really bad about that.

I’m not going to get a Kindle, at least not yet. I want to see what the Barnes and Nobel “Nook” looks like. I need to upgrade the laptop more, and I think something like the Nokia n900 might end up being a better device in this space and even if it isn’t, I think we’re going to see a lot of development in the “tablet” space in the next year, and it seems premature to buy now. For me.

Given all these caveats, I think its interesting to think about why I want the Kindle so bad. Here are some questions and answers:

So given all these caveats, why do you want a Kindle so bad?

I’ve held one on a number of occasions, and I’ve always been struck with how nice they feel. They’re solid and they’re thin. The text is clear and readable, the page advance buttons fall wonderfully under your thumbs. The experience, at least on these second generation devices, is really quite good.

I’ve rather famously, taken an entire bag of books along with me for a long weekend trip. A weekend where, I ended up reading about two and a half pages. So, the fact that you can take a whole pile of books or more properly the potential for getting the one right book you want, is appealing in a practical way.

Is this just about the hardware, or is there more?

I think the Kindle is the ideal distribution mechanism for periodical literature. The codex is likely to be of enduring importance for quite while, but I’m almost certain that the magazine and newspaper isn’t. While blogs are great, don’t get me wrong, I think there’s a need for publications that are in-between the “book” and the “blog,” and I think the Kindle is a great space for those kinds of texts. Practically, I’d like to read more content of that sort, and if I had a kindle, I suspect that I’d get a lot of use out of it.

The instant distribution model is a huge plus, and I really like to read. Cory Doctorow says something to the effect of “Ebook readers will fail, because a ‘good’ ebook reader would need to remove distractions and malfunction possibilities as effectively as paper, and devices that ‘only’ read books, won’t sell very well next to devices which also check your email and play games.” And I think that’s probably a true observation, but it looks like the Kindle does single-function pretty well. I think the next year, or so, will be really interesting as we see more tablets in the market.

You’re obviously not going to get one today, so what would make you change your mind?

The DRM and the price of the books. The DRM really needs no additional condemnation. I think 10 dollars is a bit steep for books, particularly because it’s so flat rate, and while it’s cheaper than the hardcover (and that’s good,) it’s also more than a paperback in most cases. And at least in a paperback you have something on your shelf. And the DRM really adds insult to injury. If they distributed the files in plain text/html and some weirdass XML format that would be one thing, but they give you a blob that is certain to be next to useless in a year or two. If books were 3 bucks, or 5 bucks, or even 6 or 7 bucks--even if the device was 300--or there was some sort of subscription service, I wouldn’t mind the DRM, but as it is… the DRM makes the economics difficult for me to compute.

If the DRM is such an issue why have you gotten this far?

A lot of times in the paper-book world you buy a book. Read a hundred pages (or maybe twenty?), and then are so disgusted by the book that you can’t bare to read any more, and you set it aside. And often times a trip to the bookstore (particularly in advance of a trip) means buying a number of books, when only some of these books will be of worth (to you) to justify their expense.

These situations are less likely to happen with a Kindle. There are significant samples, and you carry the bookstore around with you. I suspect the chances are that you only really need to “buy” the books that you read, which might end up being significantly cheaper in the long run.

The Kindle is a physical manifestation of a shift away from the physicality of information, but it’s only really a symptom and not a leading cause this shift, right? If you accept this, if you accept that most information and knowledge only exists as bits and photons, then all of the rituals that we build around books (collections, libraries, shelves) are less important.

What about the *Nook*?

The nook is a more impressive platform. For sure, it fails the Doctorow test of (potentially) being too interesting for tasks that aren’t reading books.

I think I probably have some more writing to do on this subject, but, in general I think Amazon is a better and smarter company than Barnes and Nobel, and if the name of the game in ebook readers is “vendor lock-in” then I trust Amazon a bit more. In a lot of ways, I hold B&N responsible for the ongoing/impending collapse of the publishing industry.¹

In any case, mostly, at the moment I just want to wait and see before I make any sort of decision on the subject.

Thoughts?

The consolidation that B&N and Borders organized for the sale of books collapsed a lot of the niche markets that were maintained by niche booksellers, and the much lamented disappearance of the midlist and backlist. The current “blockbuster supported” book industry isn’t sustainable beyond the next 5 to 10 years or so. ↩︎

Interview with Michael Pobega

Tue, 10 Nov 2009 00:00:00 +0000

Who are you? What do you?

I’m a 19 year old Perl hacker slash college student. I’m a Computer Science major at SUNYIT in Utica. To be truthful, I’m not a very good student; I spent my study hours teaching myself Perl and UNIX.

As for my personal life, I live one boat ride away from Manhattan, so when I do have time home it’s pretty hectic. Being upstate for college is so … quaint, compared to living in the big city.

I’m not really working on any large projects, but I am in the midst of working on a very lightweight an elegant blog software; the goal is to have one page (index.pl), one RSS feed (feed.rss) and one template file (template.html), and combine them all into a fully-functional blog. Then hopefully after that project is done I’ll have the time to run my Sysadmin blog -- I plan on publicising a lot of the random hacks I make at work. For example, I just spent the last two days working on a daemon that uses Net::DBus to automatically log a user out when a preset idle time is reached.
Smaller Computers, More Powerful Computers, or Cheaper Computers?

Smaller, generally. I like saving money, and I love netbooks. As I always say, the speed of the computer is up to the knowledge of the user installing the software on it. My EeePC 901 running Debian GNU/Linux runs as fast as most “modern” computers running Windows Vista.
Lets talk about technology: What kind of technology do you use, and what’s the coolest thing that technology enables for you? What about your technology do you find frustrating?

I love the freedom of expression technology brings. Being universal, it’s hard to apply country-wide laws to internet use which opens up a whole new realm of free speech for those who previously haven’t had it. But PERSONALLY, I just enjoy tinkering, learning and achieving goals. Nothing feels as good as finally finishing that program you’ve been working on for the last week.

What I find frustrating is the amount of outdated documentation you run into. The reason I spent so long on my Net::DBus script is because the docs I was using were outdated. Thankfully buu on #perl pointed me to Net::DBus::Dumper, and I figured everything out myself.
Favorite Linux/UNIX Command (whatever, as long as it fits on one line.)?

vi, of course :]

Or if you mean a singular shell command, it’s probably between perl, sed and awk. Those three programs have saved me countless hours of file editing.
The single scariest thing about the future?

Google SkyNet(tm)
Favorite Website?

Hmm … Considering I don’t browse the web too much, I’d have to go with http://latfh.com/. The laughs never stop coming.
What do you think is going to be the most important event of the next 10 years?

This one’s hard to say. I think it can be one of three things;
1. The development of high-functioning AI
2. Using computers to replace non-functioning human senses (eyesight, smell, etc)
3. Apple publicly announcing that their computers are overpriced
One thing that you wish you could learn?

French. For some reason I can’t seem to wrap my brain around spoken language.
Emacs vs. Vi

vi, of course :]

I mean, don’t get me wrong; Emacs is a great operating system, but it’s lacking a good text editor.
Where can we find more about you/your projects?

Currently I don’t have a site setup, but when I do it will be at any of these locations:
Or you can just follow me on Identica, @pobega

Industry, Community, Open Source

Mon, 09 Nov 2009 00:00:00 +0000

In “Radicalism in Free Software, Open Source” I contemplated the discourse of and around radicalism in and about Free Software and Open Source software. I think this post is a loose sequel to that post, and I want to use it to think about the role.

I suppose the ongoing merger of Sun Microsystems and Oracle, particularly with regards to the MySQL database engine weights heavy on many of our minds.

There are a number of companies, fairly large companies, who have taken a fairly significant leadership role in open source and free software. Red Hat. Sun Microsystems. IBM. Nov ell. And so forth. While I’m certainly not arguing against the adoption of open source methodologies in the enterprise/corporate space, I don’t think that we can totally ignore the impact that these companies have on the open source community.

A lot of people--mistakenly, I think--fear that Free Software works against commercialism¹ in the software industry. People wonder: “How can we make money off of software if we give it away for free?"² Now it is true that free software (and its adherents) prefer business that look different from proprietary software businesses. They’re smaller, more sustainable, and tend to focus on much more custom deployments for specific users and groups. This is in stark contrast to the “general releases” for large audiences, that a lot of proprietary audiences strive for.

In any case, there is a whole nexus of issues related to free software projects and their communities that are affected by the commercial interests and “powers” that sponsor, support, and have instigated some of the largest free software projects around. The key issues and questions include:

How do new software projects of consequence begin in an era when most projects of any notable size have significant corporate backing?
What happens to communities when the corporations that sponsor free software are sold or change directions?
Do people contribute to free software outside of their jobs? Particularly for big “enterprise” applications like Nagios or Jboss?
Is the “hobbyist hacker” a relevant and/or useful arch-type? Can we intuit which projects attract hobbyists and which projects survive because businesses sponsor their development, rather than because hobbyists contribute energy to them. For example: desktop stuff, niche window managers, games, etc. are more likely to be the province of hobbyists and we might expect stuff like hardware drivers, application frameworks, and database engines might be the kind of thing where development is mostly sponsored by corporations.
Is free software (or, Open Source may be the more apropos terminology at the moment) just the contemporary form of industry group cooperation? Is open source how we standardize our nuts and bolts in the 21st century?
How does “not invented here syndrome” play out in light of the genesis of open source?
In a similar vein, how do free software projects get started in today’s world. Can someone say “I want to do this thing” and people will follow? Do you need a business and some initial capital to get started? Must the niche be clear and undeveloped?
I’m sort of surprised that there haven’t been any Lucid-style forks of free software projects since, well, Lucid Emacs. While I’m not exactly arguing that the Lucid Emacs Fork was a good thing, it’s surprising that similar sorts of splits don’t happen any more.

That’s the train of thought. I’d be more than happy to start to hash out any of these ideas with you. Onward and Upward!

People actually say things like “free software is too communist for me” which is sort of comically absurd, and displays a fundamental misunderstanding of both communism/capitalism and the radical elements of the Free Software movement. So lets avoid this, shall we? ↩︎
To be totally honest I don’t have a lot of sympathy for capitalists who say “you’re doing something that makes it hard for me to make money in the way that I’ve grown used to making money.” Capitalist' lack of creativity is not a flaw in the Free Software movement. ↩︎

Current Projects

Fri, 06 Nov 2009 00:00:00 +0000

It’s been a while since I’ve written about what I’m working on, so I wanted to write up a little post on the subject. Just to keep myself honest.

Last time I did this, I tried to promise myself that I’d get a draft the novel I’m working on done by the beginning of November in time for me to not do the NaNoWriMo project--as is my custom. That isn’t going to happen.

I have, however, begun to stub out three files which will form the core of the remainder of the book. I have the very end of the biggest section of chapter eight, and then four more chapters. The plan is to write what feels more like four short stories with four or five adjoining little scenes. I’m not sure that this will seem all that different from the outside when I’m done, but I think this change in plan will make things easier to write.

This project is one that I both adore, and am pretty pleased with (at least at the moment,) but I’m also keenly aware that I need to be done with it, and I need to move on, as it’s been in progress for more than a year, and none of my reasons for not finishing it yet are very good.
I’ve been slowly working on a knitting project. A sweater knit at a fairly fine gauge, and incredibly plain. I’m happy with the project but I’ve pretty much given up entirely on Television watching, and as a result haven’t found a lot of time to do knitting on a regular basis. I knit during a meeting, and for a few moments here and there during a couple of social interludes, but haven’t really gotten into it. It’s going well, and I’ve got about 9 inches done of the body. 7 more till the armhole shaping begins.
I’m continuing to do the contra dance and shape note things. I think the shape note experience has been helpful for the way that I understand and participate in music, and that’s a good thing indeed. I’ve picked up a few new contra dance things, though if a given week is busy, contra dancing tends to be the first thing to disappear. I’m okay with that. I’ve also taken to going for walks in the morning before work, rather than in the evening, which is, I think better for my mind during the day at work, and also for getting work done on projects in the evening.
It seems like there’s always something else in the project of “getting your technology to work the way it ought to,” and as a result it seems like I always have something to hack upon. With my laptop running the right operating system, and doing so pretty well, the list of things to hack on have cleared up significantly. I have a desktop that I’m not using as well as I could. There’s always something else to work on with regards to my writing setup, though that’s mostly abated for the moment. I really need to find some better way to read RSS feeds. I have some hacking to do with regards to websites. There’s always something to work on, I suppose.

Oh, and I’m working a lot, but then that’s how it goes. The work projects are actually pretty fun, and they’re going well, so that’s good. If only there were more hours in the day.

Cheers!

Interview with Angie Marshall

Thu, 05 Nov 2009 00:00:00 +0000

Today’s interview is with Angie Marshall. I think introductions here are pretty much uncalled for, so here we go:

Who are you? What do you?

Angie Marshall, Legal Assistant is my paying job, but I am a Farm wife, mother of 3, a knitter, a former quilter and a half-assed gardener. I have multiple knitting projects OTN currently, but most consistently, I knit socks.
Merino or Blue Faced Leicester?

Either is awesome, but I will admit a fondness for Merino.
Lets talk about technology: What kind of technology do you use, and what’s the coolest thing that technology enables for you? What about your technology do you find frustrating?

I use a palm centro for a phone, but alas, no interwebs connection from that. I like that it keeps my names/addresses/calendar at hand, but I am frustrated that i can’t create I have an iPod touch that 98% of the time I use to listen to inspirational speeches or podcasts or audio books. I have music on it, but… meh. I have Scrabble and Soduko applications on it and I love them. This iPod frustrates me because I cheaped out on it and didn’t get a big enough one so I am constantly juggling what gets put on it. I love my laptop, but wished the battery held more charge.
Favorite book you’ve read in the last year? Runners up?

Hanging my head in shame… the Twilight series. I listen to audiobooks mostly, but I did read those in real book form. I read for entertainment.
Favorite Websites?

Ravelry and Facebook.
What do you think was the most important event of the last 15 years?

The election of our current President.
What do you think will be of the next 10?

I look forward to our scientific minds developing a fuel source that doesn’t depend on foreign oil. I hope that we take the lessons learned from this last war and we return to a self-sufficiency mindset so that we never again make the mistake of sending our troups to war over oil. >
One thing that you’re most looking forward to in the next year?

Getting better control of my life, cutting back to a 4 day work week (affirming, affirming), vacation, knitting camp, fresh asparagas morel mushrooms
One thing that you wish you could learn?

Spanish, there is such a need for spanish speakers in all areas of employment. My French from long ago High School, just isn’t much help.
Cats vs. Dogs?

Cats in the house, Dogs outside. Nothing like a warm cat “spot” (or 2 or
1. on a cold morning or a sick day.
Where can we find more about you/your projects?

Knit4Angine; The Grumpy Farmer but I don’t post much; perhaps this interview will goad me into posting a little more often.

Radicalism in Free Software, Open Source

Wed, 04 Nov 2009 00:00:00 +0000

The background:

zonker on Selling vs. Shaming
bkuhn on Denouncing vs. Advocating: In Defense of the Occasional Denouncement
Various other conversations on identi.ca over the past few weeks.

In light of this debate I’ve been thinking about the role and manifestations of radicalism in the free software and open source world. I think a lot of people (unfairly, I think in many cases) equate dedication to the “Cause of Free Software,” as the refusal to use anything but free software, and the admonishment of those who do use “unpure” software. To my mind this is both unfair to Free Software as well as the radicals who work on free software projects and advocate for Free Software.

First, lets back up and talk about RMS¹. RMS is often held up as the straw man for “free software radicals.” RMS apparently (and I’d believe it) refuses to use software that isn’t free software. This is seen as being somewhat “monkish,” because doesn’t just involve using GNU/Linux on the desktop, but it also involves things like refusing to use the non-free software written for GNU/Linux, including Adobe’s Flash player, and various drivers. In short using the “free-only” stack of software is a somewhat archaic experience. The moderates say “who wants to use a computer which has been willfully broken because the software’s license is ideologically incompatible,” and the moderates come out looking rational and pragmatic.

Except that, as near as I can tell, while the refusal to use non-free software might be a bit traumatic for a new convert from the proprietary operating system world, for someone like RMS, it’s not a huge personal sacrifice. I don’t think I’m particularly “monkish” about my free software habits, and the only non-free software I use is the adobe flash player, and the non-open-source extensions to Sun’s Virtual Box. I’m pretty sure I don’t even need the binary blob stuff in the kernel. For me--and likely for RMS, and those of our ilk--sticking to the pure “free software” stuff works better and suits the way I enjoy working.²

In short, our ability to use free software exclusively, depends upon our habits and on the ways in which we use and interact with technology.

To my mind, the process by which the pragmatic approach to free software and open source radicalizes people like RMS, is terribly unproductive. While we can see that the moderates come away from this encounter looking more reasonable to the more conventional types in the software world, this is not a productive or useful discussion to entertain.

In any case I think there are a number of dimensions to the free software (and open source world,) that focusing on “how free your software” is distracts us from. Might it not be useful to think of a few other issues. They are, as follows:

1. Free software is about education, and ensuring that the users of technology can and do understand the implications of the technology that they use.

At least theoretically, one of the leading reasons why having “complete and corresponding source code” is so crucial to free software is that with the source code, users will be able to understand how their technology works.

Contemporary software is considerably more complex than the 70s vintage software that spurred the Free Software movement. Where one might have imagined being able to see, use, and helpfully modified an early version of a program like Emacs, today the source code for Emacs is eighty megabytes, to say nothing of the entire Linux Kernel. I think it’s absurd to suggest that “just looking at the source code” for a program will be educational in and of itself.

Having said that, I think free software can (and does) teach people a great deal about technology and software. People who use free software know more about technology. And it’s not just because people who are given to use free software are more computer literate, but rather using free software teaches people about technology. Arch Linux is a great example of this at a fairly high level, but I think there’s a way that Open Office Firefox plays a similar role for a more general audience.

2. There are a number of cases around free software where freedom--despite licensing choices--can be ambiguous. In these cases, particularly, it is important to think about the economics of software, not simply the state of the “ownership” of software.

I’m thinking about situations like the “re-licensing” such as that employed by MySQL AB/Sun/Oracle over the MySQL database. In these cases contributors assign copyright to the commercial owner of the software project on the condition that the code also be licensed under the terms of a license like the GPL. This way the owning copy has the ability to sell licenses to the project under terms that would be incompatible with the GPL. This includes adding proprietary features to the open source code that don’t get reincorporated into the mainline.

This “hybrid model” gives the company who owns the copyright a lot of power over the code base, that normal contributors simply don’t have. While this isn’t a tragedy, I think the current lack of certainty over the MySQL project should give people at least some pause before adopting this sort of business model.

While it might have once been possible to “judge a project by the license,” I think the issue of “Software Freedom” is in today’s world so much more complex, and I’m pretty sure that having some sort of economic understanding of the industry is crucial to figuring this out.

3. The success of free software may not be directly connected to the size of the userbase of free software

One thing that I think Zonker’s argument falls apart around is the idea that free software will only be successful if the entire world is using it. Wrong.

Lets take a project like Awesome. It’s a highly niche window manager for X11 that isn’t part of a Desktop Environment (e.g. GNOME/KDE/XFCE), and you have to know a thing or two about scripting and programing in order to get it to be usable. If there were much more than a thousand users in the world I’d be surprised. This accounts for a minuscule amount of the desktop window management market. Despite this, I think the Awesome project is wildly successful.

So what marks a successful free software project? A product that creates value in the world, by making people’s jobs easier and more efficient. A community that supports the developers and users of the software equally. Size helps for sure, particularly in that it disperses responsibility for the development of a project among a number of capable folks. However, the size of a projects userbase (or developer base) should not be the sole or even the most important quality by which we can judge success.

There are other issues which are important to think about and debate in the free software world. There are also some other instances where the “hard line” is over radicalized by a more moderate element, nevertheless I think this is a good place to stop for today, and I’m interested in getting some feedback from you all before I continue with this idea.

Onward and Upward!

Richard Stallman, founder of the Free Software Foundation and the GNU Project, original author of the GNU GPL (perhaps the most prevalent free software license), as well as the ever popular gcc and emacs. ↩︎
Arguably, it’s easier for software developers and hacker types like myself to use “just free software” because hackers tend to make free software to satisfy their needs (the “scratch your own itch” phenomena), and so there’s a lot of free software that supports “working like a hacker,” but less for more mainstream audiences. Indeed one could argue that “mainstream computer using audiences” as a class is largely the product of the “proprietary software and technology industry.” ↩︎

Package Mangement and Why Your Platform Needs an App Store

Tue, 03 Nov 2009 00:00:00 +0000

When I want to install an application on a computer that I use, I open a terminal and type something to the effect of:

apt-get install rxvt-unicode

Which is a great little terminal emulator. I recommend it. Assuming I have a live interned connection, and the application I’m installing isn’t too large, a minute later or less I have whatever it is I asked for installed and ready to use (in most cases.)

Indeed this is the major feature of most Linux Distributions: their core technology and enterprise is to take all of the awesome software that’s out there (and there’s a lot of it,) and make it possible to install easily, to figure out what it depends on, and get it to compile safely and run on a whole host of machines. Although this isn’t the kind of thing that most people think when they’re choosing a distribution of Linux, one of the biggest differentiating features between distributions. But I digress.

I’ve written about package management here before, but to summarize:

We use package managers because many programs share dependencies, that we wouldn’t want to install twice or three times, but that we might not want to install by default with every installation of an operating system. Making sure that everything gets installed is important. This, is I think, a fairly unique-to-open-source problem, because in the proprietary system the dependencies are installed by default (as in there are more monolithic development environments, like .NET, Cocoa, and Java¹, and other older non-managed options).
One of the defining characteristics of open source software is the fact that it’s meant to be redistributed. Package management makes it easy to redistribute software, and provides real value for both the users of the operating system and for the upstream developers. Or so I’m lead to believe.

In the end, we’re back at the beginning where, you can install just about anything in the world if you know what the package is named, and the operating system will blithely keep everything up to date and maintained.²

While GNU/Linux systems get flack for not being as useable as proprietary operating systems, I see package management as this huge “killer feature” that open source systems have on top of proprietary system. We’ll never see something like apt-get for Windows not because it’s not good technology, but because it’s impossible to mange every component of the system and all of the software with a single tool.³

And then all these “App Store” things started popping up.

As I’ve thought about it, “App stores,” do the same thing for application delivery on non-GNU/* systems that package management does for open source systems. We’re seeing this sort of thing for various platforms from cell phones like the iPhone/Blackberry/Andriod to Inuit’s QuickBooks and even for more conventional platforms like Java.

Technically it’s a bit less interesting. App stores generally just receive and distribute compiled code,⁴ but from a social and user-centric perspective, the app store experience is really quite similar to the package management experience.

I’m surely not the only one to make this connection, but I’d be interesting to move past this and think about the kinds of technological progress that stem from this. App stores clearly provide value to users by making applications easier to find, and to developers who can spend less time distributing their software. Are there greater advancements to be made here? Is this always going to be platform specific, or might there be some other sort of curatorial mechanism that might add even more value in this space? And of course, how does Free Software persist and survive in this kind of environment?

I look forward to hearing from you.

Let’s not bicker about this, because the argument breaks down here a bit, admittedly, given that Java is now, mostly open. But it wasn’t designed as an open system, and all of these solve the dependency problem by--I think--providing a big “standard runtime,” and statically compiling anything extra into the program’s executable. Or something. ↩︎
Package management sometimes breaks things, it’s true, but I’ve never really had a problem that I haven’t been able to recover from in reasonably short order. I mean, things have broken, and I will be the first to admit that my systems are far from pristine, but everything works, and that’s good enough for me. ↩︎
While its possible to use more than one package manager at once, and there are cases even on linux where this is common (i.e. CPAN shell, system packages (apt/deb, yum/rpm) and ruby gems, haskell cabal and so forth) it’s not preferable: Sometimes a package will be installed by a language-specific program manager and then the system package manager will install (over it) a newer or older version of the package, which you might not notice, or it might just cause something to act a bit funny on some systems. If you’re lucky, Usually stuff breaks. ↩︎
Which means, I think indirectly that we’re seeing a move away from static linking and bundling of frameworks and into a single binary or bundle. This is one of the advancements of OS X, that all applications are delivered in these “bundles” which are just directories that contain everything that an application needs to run. Apple addressed the dependency problem by removing all dependencies. And this works in the contemporary world because if an App had to be a few extra megs to include its dependencies? No big deal. Same with Ram usage. ↩︎

Links on Technology, Blogging, and Emacs

Mon, 02 Nov 2009 00:00:00 +0000

A mostly technology-centric collection of links:

Emacs starter configuration scripts. I can’t, for the life of me, recall why I went looking for this, but last week I ended up with a whole host of basic configuration files that people have published. I’ve thought about doing this for my own files, but I’ve not had it properly cleaned up and working in a non-embarrassing way in a while. Most of these are on github, which is a phenomena that could tolerate some investigation, but no matter. Here they are, linked to by screen name: ki, elq, jonshea, larrywright, defmacro (har, just got it), jmhodges, technomancy, markhepburn, and al3x. I’d love to collect more of these, so maybe comments or the cyborg wiki.
Adjunct to that, a few more cool emacs and related links and points: First, paraedit which is a little tool which makes editing lisp easier, as well as an org-mode tip from Nathan Yergler about using org-rembmember with firefox and ubiquity. which might be of interest to some of you. I also have in the file [this link about yet another lisp dialect (yald?) called Lysp, but I don’t have much more than that. I, on the other hand will have more to say about this in the coming few weeks.
My **friend Chris Fletcher discusses his experience with contemporary blogging services** in this post. I’m not sure. Right? I mean blogging is so different today than it was when I got into it. I remember when you handed FTP credentials to blogger so they could publish your blog with their system to your site. Surely people don’t do that anymore. One of the things that I noticed at Podcamp (more on that on another post) that, frankly horrified me a bit, was that there was a whole class of bloggers who wanted to do “this thing,” but they had no interest in running their own website or making that investment of time and energy.

And maybe that’s what blogging has become. In a lot of ways doing a blog is something anyone can do pretty easily, and having a website is no longer a big part of participating in this discourse. While I’m a big fan of independence, and I don’t think the technological burden is that high. “Doing websites,” very much made me the geek I am today, so I’m not sure. Having said that, LiveJournal has never easily fit into a niche: It was blogging before there was blogging. It was social networking before we said that. It was subculture/niche before that became the thing. If I had more time in my life I’d figure out some way to study and capture that history.
For all of you OS X Desktop User Interaction Geeks, there’s this thing that lets you hide unused windows baked into the window manager. I think. I have access to OS X, but I don’t really use it enough to give this a try. GNU Screen and lots (and lots) of Emacs buffers make it possible to keep a lot of irons on the fire without getting distracted.
A **good example of a zshrc** file if that’s your thing. I think it’s my thing. Alas. I’ll write more about this once I get more used to it and figure some things out. Mostly, I’m finding that one can use it as a pure superset of bash without ill effect.

Updates and Recent Events

Fri, 30 Oct 2009 00:00:00 +0000

It’s been a while since I’ve written a status update, but I think I’ve been up to a few things which haven’t managed to work their way onto the site, so I think I’ll collect them all here, in a sort of “state of the tycho” report.

I’ve gotten more into the shape note singing. I went to the New York State Sacred Harp Convention in Cambridge New York and had a blast. I’ve written here before about this, and I don’t know how much I can really add to that. I’m continuing to be impressed by all of the young people who are into Shape Note. A fair number younger than me (though not by much). It’s strange to have acquired another hobby that takes even more time that I don’t really feel like I have; however, it’s a blast, and I feel like it’s a good thing in my life.
I’ve made a number of post collections" for the tychoish site), which provide an overview of posts on a number of subjects that aren’t generated by some sort of tagging system, nor are they comprehensive. Rather, they represent a hand picked collection of topics that I’ve covered in some depth and the posts that best exemplify these subject areas. I expect this list to grow and shift slightly, but for the moment we have:

I hope you find these archives useful. If you have a suggestion for another topic or theme, or additional posts that you think might work well in one of these posts, do feel free to touch base with me about this.

The job. I’ve not written here a great deal about my job or my move in late June across the country. I figure I write a lot about work things most of the day anyway. Having said that, I know at least a couple c-workers (Hi M.G. & S.S.) and other work-related folks read this, so it’s no great secret. Some brief reflection:

It’s going well. My team, such as it is, has sort of found a rhythm that works for us, our project progresses, and even though it always feels like I’ve written to the end of the subject matter at hand, there’s always seems to be something captivating “up next to write about.

And I get to work with and write about free software stuff with a bunch of geeks. I’m learn stuff about writing all the time somehow, and while I think I probably work a bit too much (or so the cats seem to think) it’s going well.
The Cyborg Institute is a project that I need to figure out how to do better. I think having it as a “blog” was the wrong thing to do, an I’m glad I’ve stopped doing that. I started it when I was in a much different place in so many of my projects. It was long before I started the job, I was writing about different things on the tychoish blog, and a million things. Now, things are different, and I need to figure out a better way of doing things. Having said that, things haven’t been completely dead… News forthcoming.
We’re approaching the beginning of November, when I said in a previous post that I wanted to finish the novel I was working on by then. Well that didn’t happen. Not only did it not happen, but I didn’t even manage to finish the blasted scene that I’ve been working on for a while. Having said that, I have made progress (on that scene), and I have a passable plan for finishing the last of the novel. So that’s a good thing.
I gave up the ghost on my somewhat aged “frankenbuntu” (eg. a quirky variant of Ubuntu 9.04 Jaunty) installation on my laptop, and installed Arch Linux on the laptop. I’ve been remarkably pleased with this. There are quirks, but lord were there quirks on the old installation. I’m happy with the outcome, and I’ve basically switched to using this machine as my primary computer.

Interview with Judy Stein

Wed, 28 Oct 2009 00:00:00 +0000

Here’s another interview for the interview series file. Enjoy!

Who are you? What do you?

I’ve had lots of jobs, but at the moment I only have three: running a folk club, host= ing a radio show, and teaching ballad-singing at the Folk School. Two of these are volunteer jobs. I am also a 40/50-hour per week babysitter for my 2-year-old grandson and adult in charge at my house. I have recently acquired a sewing machine and a few art supplies, and got my concertina back from a friend who borrowed it; the projects, they will come.

Ongoing activities for years include folk dancing, especially Border Morris. I have been dancing, I reckon, for roughly 51 years. Singing, roughly 60; collecting songs maybe 53. Reading, since before I started school. (I read history, detective stories, and Kipling, mostly. I will also pick up= anything by Terry Pratchett, and have only been disappointed once there. I like Shakespeare, don’t care for Thomas Hardy or most poetry.)

Things I can do well enough to have made money at, either occasionally or on a regular basis: Drawing and painting. Sewing in a pants factory. General assistance in a doctor’s office. Singing. Belly-dancing. Salesclerk-type selling: candy, pictures and mirrors. Writing. Teaching mentally ill and/or learning disabled children. All these jobs have their ups and downs.

As for Intellectual stuff: I admire but do not have the temperament for heavy-duty intellectualism: teaching has made my natural instinct for pragmatism even stronger, and too much nitpicky defining and speculation becomes boring to= me fairly quickly. Also there is a heavy-handedness that goes with too muc= h categorizing; I like exceptions to rules.

Jet Packs or Hovercars?

They both sound like fun!
Lets talk about technology: What kind of technology do you use, and what’s the coolest thing that technology enables for you? What about your technology do you find frustrating?

I am a fairly backward person technology-wise: I’d say things like “the wheel” or maybe “cars” or “dishwashers” here. And I do like my computer. I also like binoculars. And shoes that actual ly fit your feet, that’s technology. Digital cameras have proved convenient but slightly disappointing: they don’t move fast enough. Cellphones are a mixed blessing too: they always need charging, and people expect you to HAVE one.
Favorite song at the moment? Tune? Who are the runners up?

Song, at the moment: Dick Gaughan singing “Fair Flooer of Northumberland” or Louis Killen singing “April Morning.” or Peter Bellamy singing “We Have Fed Our Sea,” or Pete Morton singing “Another Train.”

Tune: “Orange in Bloom/Sherbourne Waltz,” and there are millions of runners-up. I like shapenote hymns when someone else is singing them, but am too much of an anarchist to like following dots myself. Plus I am rather surly about church, and all the praise-the-Lord-ing gets to me sometimes. There is no why; there usually isn’t, with what I like. I could come up with one if I tried, but I am not by na= ture very introspective.
Favorite Website?

Wikipedia, for one. YouTube. Gutenberg Press. Sky and Telescope. Amazon.
What do you think was the most important event of the last 15 years?

The ongoing growth of the Internet--a new Wild West. :D
One thing that you’re most looking forward to in the next year?

Reclaiming a few lost skills (see the first question)
One thing that you wish you could learn?

Gardening. Actually, I’m not sure “wish” is the correct word: if I want to know something I can get a fair start on doing it. But I do intend to try my hand at making a proper garden next spring.
Doctor Who v. Red Dwarf?

Dr. Who: mostly because I’ve never seen Red Dwarf but maybe once…
Where can we find more about you/your projects?

The Focal Point; The LiveJournal; Folk School, or by asking me!

The End of Reusable Software

Tue, 27 Oct 2009 00:00:00 +0000

I wrote a post for the Cyborg Institute several weeks ago about the idea of “Reusable Software”, and I’ve thought for a while that it deserved a bit more attention. The first time around, I concentrated a lot about the idea of reusable software in the context of the kinds of computing that people like you and me do on a day to day basis. I was trying to think about the way we interact with computers and how this has changed in the last 30 years (or so) and how we might expect for this to change soon.

Well that was the attempt at any rate. I’m not sure how close I got to that.

More recently, I’ve been thinking about the plight of reusable software in the context of “bigger scale” information technology. I’d say this fits into my larger series of technology futurism posts, except that this is very much a work of presentism. So be it.

To back up for a moment I think we can summarize the argument against reusable software, which boils down to a couple of points:

1. With widely reusable software, most of the people who use computers on a regular basis can pretty much avoid ever having to write software. While it’s probably true most people end up doing a very small amount of programming without realizing it, gone are the days when using a computer meant that you had to know how to program it. While more people can slip into using computers than ever before, the argument is that people aren’t as good at using computers because they don’t know how they work as well.

Arguably this trend is one of the harbingers of the singularity, but that’s an aside.

2. Widely reusable software is often less good software than the single-use, or single-purpose stuff. When software doesn’t need to be reused, it only needs to do the exact things you need it to do well and can be optimized, tuned, and designed to fit into a single person’s or organization’s work-flow. When developers know that they’re developing a reusable application, they have to take into account possible variances in the environments where it will be deployed, a host of possible supported and unsupported uses. They have to design a feature set for a normalized population, and the end result is simply lower quality software.

So with the above rattling around in my head, I’ve been asking:

Are web applications, which are deployed centrally and often only on one machine (or a small cluster of machines), the beginning of a return to single use applications? Particularly since the specific economic goals of the sponsoring organization/company is often quite tightly coupled with the code itself?
One of the leading reasons that people give for avoiding open source release is embarrassment at the code base. While many would argue that this is avoidance of one sort or another, and it might be, I think it’s probably also true more often than not. I’m interested in thinking about what the impact of the open source movement’s focus on source code has had on the development of single use code versus multi use code in the larger scope.
What do we see people doing with web application frameworks in terms of code reuse? For starters, the frameworks themselves are all about code reuse and bout providing some basic tools to prevent developers from recreating the wheel over and over again. But then, the applications are (within some basic limitations) wildly different from each other and highly un-reusable.

Having said that, Rails/Django/Drupal sites suffer from poor performance in particularly high-volume situations for two reasons: Firstly, it’s possible to strangle yourself in database queries in the attempt to do something that you’d never do if you had to write the queries yourself. Secondly the frameworks are optimized to save developers time, rather than run blindingly fast on very little memory.

I suppose if I had the answers I wouldn’t be writing this here blog, but I think the questions are more interesting anyways, and besides, I bet you all know what I think about this stuff. Do be in touch with your questions and answers.

Onwards and Upwards!

Posts on tychoish

New Go Module: tychoish/godmenu

dmenu?

What?

Why?

How?

What if..?

The Mid Career Shuffle

The Most Forgotten CI Feature

Reintroducing Grip

Logging Trends

Infrastructure Libraries

Grip Reborn

New Go Module: tychoish/fun

New Go Modules

Why Have You Done This?

What Have You Learned?

Crafts at Scale

Why Spinning?

The Most Plain Knitting

Isolation Reading

Software Engineering for 2.0

Systems Administrators are the Problem

API Ergonomics

Common Gotchas

How to Choose a Programming Language

Spheres of Alignment

Easy Mode, Hard Mode

Methods of Adoption

Signs of Alignment

Tips for More Effective Multi-Tasking

Pave the On and Off Ramps

Finding Alignment

Against Testify

emt -- Golang Error Tools

Rescoping the Engineering Interview

Doubled Hat Pattern

Current Work

Emacs Stability

The Emacs Daemon GTK Bug, A Parable

Tips for Casting On a Sweater

Cool Things in the Alliance Union Stories

Opus Knitting

The Perfect Sleeve

Iterative Knitting

Plain Knitting Ahead

Knitting Catchup

Short Row Wrapping

The Org Mode Product

Docker Isn't A Build System

My Code Review Comments

Sleeve Survey

Values for Collaborative Codebases

The One Page Brainstorm

How to Become a Self-Taught Programmer

Notes

Quarantine Knitting Television

Knitting Gadgets

Test Multi-Execution

Pattern Fragment 4

A Selection of Foutain Pen Recommendations

Principles of Test Oriented Software Development

Pattern Fragment 3

Editor Themes

Does Anyone Actually Want Serverless?

The Kubernetes Cloud Mainframe

Observations

Impact and Predictions

Conclusion

Programming in the Common Lisp Ecosystem

Why Ecosystems Matter

Great Things

The Dark Underbelly

Quibbles

Bigger Projects

Pattern Fragment 2

Knitting off the Cone

How to Write Performant Software and Why You Shouldn't

Pattern Fragment 1

Finished a Sweater

`dmenu`?