tychoish

Git Feature Requests

Thu, 17 May 2012 00:00:00 -0400

The ability to mark a branch "diverged," to prevent (or warn) on attempted merges from master (for example) into a maintenance branch.
The ability to create and track dedicated topic branches, and complementary tooling to encourage rebasing commits in these sorts of branches. We might call them "patch sets" or "sets" rather than "branches." Also, it might be useful to think about using/displaying these commits, when published, in a different way.
Represent merge commits as hyperlinks to the user, when possible. I think GitHub's "network graph" and similar visualizations are great for showing how commits and branches interact and relate to each other.

This would probably require some additional or modifies output from "git log".
Named stashes.
Branched stashes (perhaps this is closer to what I'm thinking about for the request regarding topic branches.)
The ability to checkout "working copies," of different points/branches currently from a single repository at the same time, using "native" git utilities.

Related, "shelf" functionality is scriptable, but this too needs to be easier and more well supported.

I think legit is a step in the right direction, but it's weird and probably makes it more difficult to understand what's happening with git conceptually as opposed to the above features which would provide more appropriate conceptual metaphors for the work that would-be-git-users need.

The Limitiations of the GitHub Fork Model

Thu, 10 May 2012 00:00:00 -0400

Assumption:

git is pretty awesome, but it's conceptually complex. As a result using git demands a preexisting familiarity with git itself or some sort of wrapper to minimize the conceptual overhead.
The collaboration methods (i.e. hosting) provided by git, which are simple by design to allow maximum flexibility, do not provide enough structure to be practically useful. As a result providers like GitHub (and BitBucket and gitorious) offer a valuable service that makes it easier--or even possible--for people to use git.

Caveats:

there are problems with using centralized repository services controlled by third parties, particularly for open source/free software projects.

There are ways that GitHub succeeds an fails in this regard. but this dynamic is too complex to fully investigate within the scope of this post.
If you use GitHub as designed, and the way that most projects use GitHub, then you have a very specific and particular view of how Git works.

While this isn't a bad thing, it's less easy to use git in some more distributed workflows as a result. This isn't GitHub's fault so much as it is an artifact of people not really knowing how git itself works.

Assertion:

GitHub's "fork" model disincentives people from working in "topic" branches.
By making it really easy for people to publish their branches, GitHub disincentives the most productive use of the "git rebase" command that leads to clean and clear histories.
There's no distinction between a "soft fork" where you create a fork for the purpose of submitting a patch (i.e. a "pull request") and a "hard fork," where you actually want to break the relationship with the original project.

This is mostly meaningful in context of the other features that GitHub provides, notably the "Network" chart, and the issue tracker. In a soft-fork that I would intend to merge back in, I'd like the issues to "come with," the repository, or at least connect in some way to the "parent." For hard forks, it might make sense to leave the old issues behind. The same with the network chart, which is incredibly powerful, but it's not great at guessing how your repository relates to the rest of its "social network."

The solution: keep innovating, keep fighting lock-in, and don't let GitHub dictate how you work.

Making Things Easier

Tue, 08 May 2012 00:00:00 -0400

I spent a lot of time in the past few months thinking about "automation," as a project to take things that take a long time and require a lot of human intervention into things that just do themselves, and I think this is the wrong approach.

While total automation is an admirable, it's difficult, both because it requires more complex software to deal with edge cases, but also because it's hard to iterate into a fully automated solution.

Let's back up for a moment and talk about automation in general.

Computers are great at automating things. When you figure out how exactly to accomplish something digitally (i.e. polling an information source for an update, transforming data, testing a system or tool,) writing a program to perform this function is a great idea: not only does it reduce the workload on actual people (i.e. you.) I think the difference between people who are "good with computers," and people who are "great with computers," is the ability to spot opportunities for these kinds of automations, and potentially implement them..

To my mind the most important reason to automate tasks is to ensure consistency and to make it more likely that tedious tasks get done.

Having said this, rather than develop complete task automations for common functions, the better solution is probably to approach automation on the bottom up: instead of automating a complete process, automate smaller pieces particularly the most repetitive and invariable parts, and then provide a way for people to trigger the (now simplified) task.

The end result, is a system that's more flexible easier to write, and less prone to failure under weird edge cases. Perhaps this is a manifestation of "worse is better" also.

?Thoughts?

Onward and Upward!

Supporting Distributed Bug Tracking

Mon, 30 Apr 2012 00:00:00 -0400

The free software/open source/software development world needs a distributed bug tracking story. Because the current one sucks.

The State of the Art

There are a number of tools written between 2006 and 2010 or so that provide partial or incomplete solutions to the problem. Almost isn't quite good enough. The "resources" section of this post, contains an overview of the most important (my judgment,) representatives of the current work in the area with a bit of editorializing.

In general these solutions are good starts, and I think they allow us (or me) a good starting point for thinking about what distributed bug tracking could be like. Someday.

Bug tracking needs are diverse, which creates a signifigant design challenge for any system in this space. There are many existing solutions, that everyone hates, and I suspect most would-be developers and innovators in the space would like to avoid opening this can of worms.

Another factor is that, while most people have come to the conclusion that distributed source control tools are the "serious" contemporary tool for managing source code the benefits of distributed bug tracking hasn't yet propogated in the same way. Many folks have begun to come to terms with the fact that some amount of tactical centralization is inevitable, required, and even desirable1 in the context of a issue tracking systems.

Add to this the frequent requirement that non-developer users often need to track and create issues, and the result is that we've arrived at something of an impasse.

Requirements

A distributed bug tracking system would need:

A good way to provide short, unique identifiers for individual issues and comments so that users can discuss issues canonically.
An interface contained in a single application, script, or binary, that you could distribute with the application.
A simple/lightweight web-based interface so that users can (at least) review, search, and reference issues from a web browser.

Write access would also be good, but is less critical. Also, it might be more practical (both from a design and a workflow perspective,) to have users submit bugs on the web into a read-only "staging queue," that developers/administrators would then formally import into the project. This formalizes a certain type of triage approach that many projects may find useful.
To be separable from the source code history, either by using a branch, or by using pre-commit hooks to ensure that you never commit changes to code/content and the bugs at the same time.
To be editable, and to interact with commonly accessible tools that users already use. Email, command line tools, the version control systems, potentially documentation systems, build systems, testing frameworks and so forth.
Built on reliable tools.2
To provide an easy way to customize your "views" on bugs for a particular team or project. In other words, each team can freely decide which extra fields get attached to their bugs, along with which fields are visible by default, which are required, and so on -- without interfering with other projects.

The Future of the Art

We (all) need to work on building new and better tools to help solve the distributed issue tracking problem. This will involve:
- learning from the existing attempts,
- continuing to develop and solidify the above requirements,
- (potentially) test and develop a standard (yaml/json?) based data storage format that is easy to parse, and easily merged that multiple tools can use.
- Develop some simple prototype tools, potentially as a suite of related utilities (a la early versions of git.) that facilitate interaction with the git database. With an eye towards flexibility and extensible.
While there are implications for free software hosting as well as vendor independence and network service autonomy (a la Franklin Street Statement.) I think the primary reason to pursue distributed bug tracking has more to do with productivity and better engineering practices, and less with the policy. In summary:
- Bug database systems that run locally and are fast3 and always available.
- Tools that permit offline interaction with issue database.
- Tools that allow users to connect issues to branches.
- Tools that make it possible to component-ize bug databases in parallel with software

Resources

(With commentary,)

dist-bugs mailing list

This is the canonical source for discussion around distributed bug tracking.
Bugs Everywhere

This is among the most well developed solution speaking holistically. "be" is written in Python, can generate output for the web. It uses its own data format, and has a pretty good command line tool. The HTML output generate is probably not very fast at scale (none are,) but I have not tested it.
Ditz

Ditz is a very well developed solution. Ditz: implemented in Ruby, has a web interface, has a command line tool, uses a basic YAML data format, and stores data in branch. Current development is slow, getting it up and running is non-trivial, and my sense is that there isn't a very active community of contributors. There are reasons for this, likely but they are beyond the scope of this overview.
pitz

Pitz is a Python re-implementation of Ditz, and while the developer(s?) have produced a "release," the "interface" is a Python shell, and to interact with the database you have to, basically write commands in Python syntax. From a data perspective, however, Pitz, like Ditz is quite developed. Pitz while it stores data in-tree, I think it's important source of ideas/examples/scaffolding.
Artemis

This is a really clever solution that uses Maildirs to store issues. As a result you can interact with and integrate Artimis issues with your existing email client. Pull down changes, and see new bugs in your email, without any complicated email and list server setups.

The huge caveat is that it's implemented as a plugin for Mercurial, and so can't be used with git projects. Also, all data resides in the tree.
git-issues

In most ways, git-issues is my favorite: it's two Python files, 1700 lines of code, stores issues outside of the source branch, and has a good command line interface. On the downside, it uses XML (which shouldn't matter, but I think probably does, at least in terms of attracting developers,) and doesn't have a web-based interface. It's also currently un-maintained.
Prophet/sd

SD, which is based on a distributed database named Prophet, is a great solution. The primary issue is that it's currently unmentioned and is not as feature complete as it should be. Also a lot of SD focuses on synchronizing with existing centralized issue trackers, potentially at the expense of developing other tools.

It seems that you want centralized issue databases, or at least the fact that centralized issue databases appear canonical is a major selling point for issue tracking software in general. Otherwise, everyone would have their own text file with a bunch of issues, and that would suck. ↩
Because I don't program (much) and it's easy to criticize architectural decisions from afar, I don't want to explicitly say "we need to write this in Python for portability reasons" or something that would be similarly unfounded. At the same time, adoption and ease of use is crucial here, both for developers and users. Java and Ruby (and maybe Perl,) for various reasons, add friction to the adoption possibilities. ↩
"Is Jira/Bugzilla/etc. slow for you today?" ↩

New Haven All Day Singing 2012

Fri, 20 Apr 2012 00:00:00 -0400

I went to a fabulous all day Sacred Harp singing in New Haven Connecticut last weekend. It was great. Thoughts:

Size and Space

This wasn't a huge singing. There plenty of singings in the Northeast that have higher attendance, but that doesn't matter, there was something nice about getting to sing with the people assembled.

It helped that the room was great for singing, and it was the perfect size for the crowd. Sacred Harp singers are often big folklore geeks or big music geeks, but I think deep down, we're all really huge room geeks. Because a room that just sounds and feels good, makes all the difference in the world.

Note to self: Go to more awesome regional singings in great rooms in the future.

Heat

It was a pleasantly warm spring day, and with 80 or so people in a room, it got warm, and while this did subtract something from the comfort level, it also sets the mood somehow, and changes the tone of the day. Also, when the air is a bit more humid (but it only needs to be a bit,) and it's not as drafty and cold, its easier to keep your voice warmed up. The end result: I (or one) will sing better between April and September.

My Voice Part

In an unusual move for me, I spent 3 out of 4 sessions singing Tenor. While I don't have a "super bass" voice, I'm defiantly on bass side of baritone. In Sacred Harp, theoretically everyone can sing tenor, and it's fun to mix things up a bit and songs sound different in different parts. I am also finding that having a sense of another part makes it possible to have a more rich sense of the music. Highly recommended.

Also, every time I sing tenor it takes me 25 minutes to remember (or remind my body) how to do it, so it was particularly nice to have a good long spell of singing to both figure out how to sing the part but also to get more comfortable with it.

My local singing community has been a bit bereft of basses lately, so I haven't had much opportunity to actually sing tenor as much as I might like recently, so it was a particularly good change of pace.

Song Selection

I had something of an epiphany about leading and choosing your song at a singing.

While choosing a song that you like and enjoy hearing is obviously a part of the processes, I think song choice is more about choosing the right song for the moment, and figuring out what will sound best next, given the previous few songs.

I used to obsess a great deal about what song I would lead, and study it (at least some) before the singing even began. This weekend, I came with a few songs that I'd been thinking about but changed at the last moment when I thought that the song I had picked out wouldn't fit very well.

On the Origins of Morris Dancing

Mon, 09 Apr 2012 00:00:00 -0400

When you're out in the world Morris Dancing, everyone asks what you're doing. Actually, more typically people ask "What country is this from?" but I don't know what that means. In any case, this past weekend I witnessed the following exchange between elderly spectator to the foreman of a certain New York City Men's Team as the tour was moving between stands:

Spectator: What is this and where is it from?

J.D.: It's English Morris Dancing, do you know what that is?

Spectator: No.

J.D.: Do you care?

Spectator (pauses, unsure of how to continue) Yes, is it like cricket or football?

J.D. Cricket.

(At this point the spectator continued about his way satisfied and the tour continued to the next stand.)

This exchange, as these usually go, was pretty good. And the cricket part is totally right. There's actually, as I understand it, a lot of connection between the history of cricket and Morris Dancing: village's teams would play cricket and dance Morris, Morris Kits were often cricket uniforms with ribbons and bells, and before cricket, most Morris dancers wore black pants.

Still.

Lies Developers Tell About Documentation

Thu, 05 Apr 2012 00:00:00 -0400

All the documentation you'd need is in the test cases.
My comments are really clear and detailed.
I'm really interested and committed to having really good documentation.
This code is easy to read because its so procedural.
This doesn't really need documentation.
I've developed a really powerful way to extract documentation from this code.
The documentation is up to date.
We've tested this and nothing's changed.
This behavior hasn't changed, and wouldn't affect users anyway.
The error message is clear.
This entire document needs to be rewritten to account for this change.
You can document this structure with a pretty clear table.

Often this is true, more often these kinds of comments assume that it's possible to convey 3-5 dimension matrixes clearly on paper/computer screens.
I can do that.
I will do that.
No one should need to understand.

Task Updates

Tue, 03 Apr 2012 11:19:31 -0400

Life has been incredibly busy and full lately and that's been a great thing. I've also been focusing my time on big projects recently rather than posting updates here and updating the wiki. And then I have this day job which basically counts as a big project. While I like the opportunity to focus deeply on some subjects, I also miss the blog.

tycho is conflicted about something. Shocking.

In any case, I want to do something useful with this space more regularly. So here I am and expect me more around these parts.

I've been working on a total refresh of my Cyborg Institute project. I want it to be an umbrella for cool projects, nifty examples, great documentation, and smart people1 working on cool projects. If that's ever going to happen, I need to get something together myself. The first release will contain:

A book-like object, that provides an introduction to the basic principals of Systems Administration for developers, "web people," and other people who find themselves in charge of systems, without any real introduction to systems administration. (Status: 70% finished, with a couple more sections to draft and some editing left.)
A Makefile based tasklist aggregator, inspired by org-mode but largely tool agnostic. (Status: 95% finished, with documentation editing and some final testing remaining.)
A logging system for writers. I use it daily, and I think it's a vast improvement over some previous attempts at script writing, and I did a pretty good job of documenting it, but it's virtually impossible to manage/maintain. Having said that, I always wanted to rewrite it in Python (as a learning exercise,) so that might be a cool next step (Status: Finished save editing and an eventual rewrite.)
Emacs and StumpWM config files, packaged as "starter-kits" for new users. I have good build processes for both of these. I don't think that I need to document them fully, but I need to write some READMEs. Since there's a lot of redistribution of others code, I need to figure out the most compatible/appropriate license. (Status: Finished except for the work of free afternoon.)

Probably, all of these Cyborg Institute projects will get released at about the same time. The blockers will be finishing/editing the book and editing everything else. I might make the release a thing, we'll see.

Other than that, I:

Updated compilation.
Finished the first draft of this novel. Editing will commence in June. I've also started planning a fiction project, for a draft to begin in the fall?
Wrote a few paragraphs on the isd page, but I'm starting to think that as my time becomes more limited, that the critical-futures wiki project, as such, will probably be the first thing to fall on the floor, unless someone else is really interested in making that be a thing.

Onward and Upward!

My intention for the Cyborg Institute has always been (and shall remain,) as a sort of virtual think tank for cool projects put up by myself and others. You all, dearest readers, count in this group. ↩

Transit and NYC Subway Strategy

Thu, 29 Mar 2012 00:00:00 -0400

I have a question for game theory/urban planning/transit geeks, in part for practical reasons, and in part for a story I'm developing:

Is there some sort of resource that explains "most efferent" rapid transit rider strategies, perhaps from a game theory perspective?

I've been living in NYC for almost a year, with frequent visits for about 6 months before that and I've learned things like:

The physical layouts of a number of station complexes and transfer points, to facilitate quick/easy transfers.
A faltering sense of when to take and express and when to take a local, and when it makes sense to switch.
A decent sense of which route will be more direct/quicker in a given situation.
An acceptable sense of which part of the train you need to be on.

I'm interested in knowing if there is any work aimed at a general audience that addresses any of these questions, in particular:

The express/local decision making logic, particularly given situations like:
- When does it make sense to walk ~5-10 blocks to an express stop (possibly in the wrong direction,) rather than walk a shorter distance to a local stop that would require a transfer. (If your destination is an express stop?)
- At what point in a journey does it make the most sense to transfer between trains?
- How does time-of-day affect the logic.
Specific differences (if any) for navigating subway lines with multiple converging services. (e.g. the "M" with regards to the F and E, as well as the R with regards to the N in Manhattan.)
Logic for transferring between non-parallel train services that intersect at multiple points in a given journey (e.g. in Brooklyn, vs. Manhattan, if needed for inter-borough trips; E/M; 4,5,6 vs N, R.)
Logic for transferring between services that run on the same track (e.g. in NYC: N/R, E/C, F/M, etc.) with respect to how your journey overlays the route divergence and convergences.

The Million Words of Crap

Wed, 28 Mar 2012 00:00:00 -0400

There's this saying, I traced it down once but have forgotten the source again, that before you can write anything good, you have to write a million words of crap.

Well I've done it. I'm guessing that most writers hit this number without much difficulty fairly early on in their lives/careers, but it's hard keep records and do proper accounting of this data. And let's be frank, it's kind of a stupid thing to track. While I'm definitely a better writer these days than I was even 2 years ago, I know that I have a lot more to learn.

In any case, I have a firm account of a million words of crap that I've written in the last 10 years or so:

tychoish.com this wiki, mostly rhizome accounts for a bit more than 800,000 words.
I wrote a novel in 2002 and 2003 that now exists primarily as a paper volume in a zippered binder that I use as a foot rest. That was 100,000 words.
Since September 26, 2011 I've written a bit under 101,500 words for my work. The number is slightly inflated on account of code samples, some repetitive sections, and a few articles that colleagues wrote that I did some significant editing/rewriting.
Between June 2009 and May 2011, I wrote somewhere between 100,000 and 200,000 words for a previous job (again some measure of redundant content, other people's work, and code samples makes this figure hard to track.)
The Knowing Mars novella that I wrote between July 2007 and ~March 2008 is about 35,000 words.
The novel that I finished the first draft of few weeks ago is about 85,000 words.
I'm 35,000 words into the first draft of a technical book-like object.
I have 10,000 words in a couple of other projects.

Which puts me way way way, over (what? 1,300,000.) I wonder when I broke the million word mark?

Some observations on writing.

I still make all kinds of writing mistakes that I find embarrassing and difficult. Particularly with fiction (which probably only accounts for 200,000-250,000 words,) I feel incredibly clumsy.
At the same time, as long as I have a clear idea of what a piece of text needs to say, I'm reasonably comfortable pulling together a draft without much fuss. Sometimes figuring out what I need to say involves a bunch of reading and long walk, but it's possible.
I'm getting better at writing less. Writing concisely is hard work and it's easy to get into the habit of generating "word stew" that don't say anything useful and are impenetrable to read.
Writing is hard. I've always been an awful speller and I have some dyslexic moments, but--as longtime readers of the blog have surely noticed--I'm getting way better and writing cleaner copy.

Editing other peoples writing on a regular basis has also been enlightening because it allows me to put things in perspective, and be able to see value in what I can do with words. Also it's cool to be able to help other people write.
This is totally a correlation, but I pretty much never write anything longhand, and I think I'm a better writer for it. The downside is that I'm incredibly dependent on my own computer and its setup to do anything.

Onward and Upward!

Loops and Git Automation

Thu, 15 Mar 2012 00:00:00 -0400

This post provides a few quick overviews of cool bits of shell script that I've written or put together recently. Nothing earth shattering, but perhaps interesting nonetheless.

Making a Bunch of Symbolic Links.

I have two collection of Maildir folders to store my mail. One in a ~/work/mail folder for my work mail, and ~/mail for my personal projects. I want the work mailboxes to be symbolically linked to the personal ones (some work email, notably from GitHub comes to a personal address, and I want to be able to refile as necessary.) I moved the work maildirs last week, so I needed to create about 15 symbolic links, and so I wrote the following little loop:

 cd ~/mail
 for i in `find ~/work/mail -maxdepth 1 -mindepth 1`; do
     ln -s $i work.`echo $i | cut -d / -f 6`
 done

Should be useful, no?

Commit all Git Changes

For a long time, I used the following bit of code to provide the inverse operation of "git add .". Where "git add ." adds all uncommited changes to the staging area for the next commit, the following commit automatically removes all files that are no longer present on the file-system from the staging area for the next commit.

 if [ "`git ls-files -d | wc -l`" -gt "0" ]; then
     git rm --quiet `git ls-files -d`
 fi

This is great if you forget to use "git mv" or you delete a file using rm, you can run this operation and pretty quickly have git catch up with the state of reality. In retrospect I'm not really sure why I put the error checking if statement in there.

There are two other implementations of this basic idea that I'm aware of:

  for i in `git ls-files -d`; do
     git rm --quiet $i
  done

Turns out you can do pretty much the same thing with the following statement using the xargs command and you end up with something that's a bit more succinct:

 git ls-files --deleted -z | xargs -0 git rm --quiet

I'm not sure why, I think it's because I started being a Unix nerd after Linux dropped the argument number limit, and as a result I've never really gotten a chance to become familiar with xargs. While I sometimes sense that a problems is xargs shaped, I almost never run into "too many arguments" errors, and always attempt other solutions first.

A Note About `xargs`

If you're familiar with xargs skip this section. Otherwise, it's geeky story time.

While this isn't currently an issue on Linux, some older UNIX systems (including older versions of Linux,) had this limitation where you could only pass a limited number of arguments to a command. If you had too many, the command would produce an error, and you had to find another way.

I'm not sure what the number was, and the specific number isn't particularly important to the story. Generally, I understand that this problem would crop up when attempting to take the output of a command like find and piping or passing it to another command like grep or the like. I'm not sure if you can trigger "too many arguments" errors with globbing (i.e. *) but like I said this kind of thing is pretty uncommon these days.

One of the "other ways" was to use the xargs command which basically takes very long list of arguments and passes them one by one (or in batches?) to another command. My gut feeling is that xargs can do some things, like the above a bit more robustly, but that isn't experimentally grounded. Thoughts?

Onward and Upward!

Cron is the Wrong Solution

Thu, 08 Mar 2012 00:00:00 -0500

Cron is great, right? For the uninitiated, if there are any of you left, Cron is a task scheduler that makes it possible to run various scripts and programs at specified intervals. This means that you can write programs that "do a thing" in a stateless way, set them to run regularly, without having to consider any logic regarding when to run, or any kind of state tracking. Cron is simple and the right way to do a great deal of routine automation, but there are caveats.

At times I've had scads of cron jobs, and while they work, from time to time I find myself going through my list of cron tasks on various systems and removing most of them or finding better ways.

The problems with cron are simple:

Its often a sledge hammer, and it's very easy to put something in cron job that needs a little more delicacy.
While it's possible to capture the output of cron tasks (typically via email,) the feedback from cronjobs is hard to follow. So it's hard to detect errors, performance deterioration, inefficiencies, or bugs proactively.
Its too easy to cron something to run every minute or couple of minutes. A task that seems relatively lightweight when you run it once can end up being expensive in the aggregate when they have to run a thousand times a day.

This isn't to say that there aren't places where using cron isn't absolutely the right solution, but there are better solutions. For instance:

Include simple tests and logic for the cron task to determine if it needs to run before actually running.
Make things very easy to invoke or on demand rather than automatically running them regularly.

I've begun to find little scripts and dmenu, or an easily called emacs-lisp function to be preferable to a cron job for a lot of tasks that I'd otherwise set in a cron job.
Write real daemons. It's hard and you have to make sure that they don't error our or quit unexpectedly--which requires at least primitive monitoring--but a little bit of work here can go a long way.

Onward and Upward!

Git In Practice

Tue, 21 Feb 2012 00:00:00 -0500

Most people don't use git particularly well. It's a capable piece of software that supports a number of different workflows, but because it doesn't mandate any particular workflow it's possible to use git productively for years without ever really touching some features.

And some of the features--in my experience mostly those related to more manual branching, merging, and history manipulation operations--are woefully underutilized. Part of this is because Github, which is responsible for facilitating much of git's use, promotes a specific workflow that makes it possible to do most of the (minimal required) branch operations on the server side, with the help of a much constrained interface. Github makes git usable by making it possible to get most of the benefit of git without needing to mess with SHA1 hashes, or anything difficult on the command-line.

That's a good thing. Mostly.

Nevertheless, there are a few operations that remain hard with git: I sometimes encounter situations that I have to try a few times before I get it right, and there are commands that I always have to check the man page to figure out how to specify the references. And even then I'm sometimes still confused. So maybe I (or we?) can spend a little bit of time and figure out what processes remain hard with git and maybe try and see if there is a way to make the process a bit more streamlined.

Here's my list:

Reorder all commits since x commit.

This is basically: find the commit before the earliest one that you want to change, run "git rebase -i <commit hash>" reorder the commits even though git sorts the commits in the order that I find most un-intuitive.
Create local branches to track remote branches or repositories.

Setup the remotes, if necessary, and then run: "git branch --track <local-branch-name> <remote>/<branch-name>" and "git config branch.{name}.push {local-branch}:master".
Stash all local changes and switch branches.

It would also be nice if you could figure out way for git (or a helper) to see any open files in your text editor and save/close them if needed.
Pull a commit from the history of one branch into another branch without pulling anything else.

I think this is chery-pick? It might also be nice to pull a series of commits from one branch, rebase them into one commit in the destination branch, and then commit that.
Pretty much every time I've tried to use the merge command to get something other than what I would have expected to happen by using "pull," it ends tragically.

Reader suggestions:

Put your process/procedural frustrations with git here.

How about we work on figuring out how to solve these problems on the "comments page" page.

Assisted Editing

Thu, 09 Feb 2012 00:00:00 -0500

I learned about artbollocks-mode.el from Sacha Chua's post, and it's pretty freaking amazing.

Basically, it does some processing of your writing--while you work--to highlight passive sentences and affected jargon.1 And that's all. There are some functions for generating statistics about your writing, but I find I don't use that functionality often. You can enable it all of the time, or just turn it on when you're doing editing.

After a few weeks, I've noticed a marked improvement in the quality of my output. I leave it on all the time, but I'm pretty good at resisting the urge to edit while I'm writing. Or at least I'm pretty good at picking up again after going back to tweak a wording. In general it's hard to keep more than a few things in an editing pass at any time.

It turns out that the instant feedback on passive sentences, even though it's not perfect, is great for improving the quality of my content the first time out. And it's even better for doing editing work. It's harder to ignore a passive sentence when the editor is highlighting you see a screen full of them for you.

It's of course important to be able to ignore its suggestions from time to time, and it's no harder to ignore than "flyspell-mode" (the on-the-fly spell checker in emacs.)

This is perhaps the clumsiest part of the default distribution, as jargon is terribly specific to the kind of writing you're doing, and it turns out that one of the "art critic"/post-modern words (i.e. "node") is a word that I end up using (acceptably, I think) in a technical context when describing a clustered system. And there's a difference between a technical lexicon and a jargon, and regular expressions aren't terribly sensitive to this, so the actual list of words that you need to call yourself out on, varies a bit from person to person. But once you customize it, it's great. ↩

Allowable Complexity

Fri, 03 Feb 2012 00:00:00 -0500

I'm not sure I'd fully realized it before, but the key problems in systems administration--at least the kind that I interact with the most--are really manifestations of a tension between complexity and reliability.

Complex systems are often more capable flexible, so goes the theory. At the same time, complexity often leads to operational failure, as a larger number of moving parts leads to more potential points of failure. I think it's an age old engineering problem and I doubt that there are good practical answers.

I've been working on this writing project where I've been exploring a number of fundamental systems administration problem domains, so this kind of thing is on my mind. It seems, that the way to address the hard questions often come back to "what are the actual requirements, and are you willing to pay the premiums to make the complex systems reliable?"

Trade-offs around complexity also happen in software development proper: I've heard more than a few developers talk in the last few months weigh the complexity of using dynamic languages like Python for very large scale projects. While the quests and implications manifest differently for code, it seems like this is part of the same problem.

Rather than prattle on about various approaches, I'm just going to close out this post with a few open questions/thoughts:

What's the process for determining requirements that accounts for actual required complexity?
How do things that had previously been complex, become less complex?

Perhaps someone just has write the code in C or C++ and let it mature for a few years before administrators accept it as stable?
Is there an corresponding level of complexity threshold in software development and within software itself? (Likely yes,) and is it related to something intrinsic to particular design patterns, or to tooling (i.e. programming language implementations, compilers, and so forth.)

Might better developer tooling allow us to programs of larger scope in dynamic languages (perhaps?)

Reader submitted questions:

Your questions here.

Answers, or attempts thereat ?on the discussion page.

Persistent Emacs Daemons

Thu, 02 Feb 2012 00:00:00 -0500

I've been subject to a rather annoying emacs bug for months. Basically, when you start emacs with the --daemon switch, and the X11 session exits, and any emacs frames are open, then the emacs process dies. No warning. The whole point, to my mind, of the daemon mode is to allows emacs sessions to persist beyond the current X11 session.

This shouldn't happen. I think this is the relevant bug report, but I seem to remember that the issue has something to do with the way that GTK interacts with the X11 session and emacs's frames. It's something of a deadlock: the GTK has no real need to fix the bug (and/or it's a behavior that they rely on for other uses,) and it might not really be possible or feasible for emacs to work around this issue.1

I also think that it's probably fair to say that daemon mode represent a small minority all emacs usage.

Regardless, I've figured out the workaround:

.. don't use GTK.

Turns out, it's totally possible to build GNU emacs without GTK, by using the "Lucid" build. Which is to say, use the windowing system kit built for Lucid Emacs (i.e. XEmacs,) rather than GTK. I was able, using the code below, to get an emacs experience with the new build that seems identical2 to the one I used to get with GTK, except without the frustrating crashes every time that X11 spazzed when I decided to unplug a monitor or some such. A welcome improvement, indeed.

The following emacs-lisp covers all of the relevant configuration of the "look and feel" of my emacs session. Install the Inconsolata font if you haven't already, you'll be glad you did.

 (setq-default inhibit-startup-message 't
               initial-scratch-message 'nil
               save-place t
               scroll-bar-mode nil
               tool-bar-mode nil
               menu-bar-mode nil
               scroll-margin 0
               indent-tabs-mode nil
               flyspell-issue-message-flag 'nil
               size-indication-mode t
               scroll-conservatively 25
               scroll-preserve-screen-position 1
               cursor-in-non-selected-windows nil)

 (setq default-frame-alist '((font-backend . "xft")
                             (font . "Inconsolata-14")
                             (vertical-scroll-bars . 0)
                             (menu-bar-lines . 0)
                             (tool-bar-lines . 0)
                             (alpha 86 84)))

 (tool-bar-mode -1)
 (scroll-bar-mode -1)
 (menu-bar-mode -1)

Hope this helps you and/or anyone else that might have run into this problem.

I'd like to add the citation and more information here, but can't find it. ↩
To be fair, I mostly don't use the GUI elements in emacs, though having emacs instances outside of the terminal is nice for displaying images when using emacs-w3m, and for having a little bit of additional display flexibility for some more rich modes. ↩

The Laptop Riser that Changed My Life

Wed, 01 Feb 2012 00:00:00 -0500

tl;Dr> I got one of those nifty laptop risers that puts your laptop up closer to eye level, and it has pretty much improved all of my interactions with computers a thousand fold and it's made it possible for me to effectively use two screens. This post explores this.

One of my coworkers had a laptop stand she wasn't using and I asked to borrow it for an afternoon, and my neck stopped hurting. I never thought my neck hurt before, but apparently it does.

Or did.

But there's more: for years now I've kept an extra monitor around (and had one at work) but the truth is that I have never really felt like I've been able to get the most out of an external monitor.

Somehow, putting my laptop 4 inches in the air was the little change that made everything better. The laptop is generally on the left of the external monitor, and I have task lists, notes buffers, the chat window, and my status logging window on the laptop, and then three windows on the external (emacs buffer, terminal, emacs buffer) on the right. My primary focus centers between the monitors, but probably edges slightly toward the external, most of the time.

Also, I discovered that I--apparently--have a slight processing/attention defect whereby I find it painful and difficult to focus on things that are happening on the right side of the screen for any amount of time. Which is weird because my right eye has always been noticeably stronger. I'll ponder this more later.

My virtual desktops for email and web browsing are a bit less rigid, but the same basic idea. Somehow it seems to work. I've done a little bit of work recently to get the layouts right, to minimize the impact of the window management of most context switching (scripting various transitions, saving layouts, etc.) In all things are going great.

It strikes me that I've not posted here even a little about my setup in a while. The truth is that it's not terribly surprising and I've not changed very much recently. I'm back to one laptop, and as anxious as having one laptop makes me sometimes (I fear the lack of redundancy,) not having to keep it synced makes life easier. I've put some time into doing a little bit of polish on all of little bits of configuration/code that I have that makes my computing world go around, but mostly it's pretty good.

It's nice, and I'd write more about it, but I want to get back to getting things done around here. Exporting and exploring some of this stuff in greater depth is definitely on my list, so hang in there, and if there's something you particularly want to see, ?be in touch.

2011 Work/Spacetime Retrospective

Thu, 26 Jan 2012 00:00:00 -0500

For the most part, I'm quite happy with everything that I was able to accomplish last year. I've moved cities (for the second year in a row) and last year I changed jobs twice: in both cases, I think the current will stick for a while. And I'm working on other projects, with some impressive speed. Last year wasn't been great for finishing things, but I guess there's room for improvement this year.

After a fair amount of professional angst I'm finally doing pretty much exactly what I want to be doing: I'm writing a substantial/total revision of a software manual for a company developing an open source database system. I'll leave you to figure out the details, but it's great.

A couple years ago, I said to myself, that I wanted to be a "real technical writer," which is to say, work with engineering teams, write documentation and tutorials for a single product or group of products, and operate on a regular release schedule. I've done a great deal of writing for technology companies: from project proposals and journalism, to tutorials and content for distributors, to white papers, marketing, and sales materials. Delightfully, I've managed to get there, and in retrospect it's both somewhat amazing, and incredibly delightful.

A while back, I had dinner with a friend who's been doing the same thing I do for a long time (we know each other through folk dance and singing,) and by comparing our experiences it was great to learn that my experience is quite typical, both in terms of the work I'm doing and the procedural engineering practice frustrations (e.g. "What do you mean you changed the interface without telling me?!?!")

At work we have this thing where we send in an account of what we did during the day so that other people know what we're working on, and so that we can keep our team on the same page. After all, when you're all looking at computer screens all day, and in a few different time zones, it's easy to loose track of what people are working on.

At the bottom of these emails, we're prompted to ask "what are your blockers and impediments." Often I say something clever like "Compiler issue with Spacetime interface or Library." Or something to that effect. It feels like a good description of the last year.

Onward and Upward!

Aeron Woes

Tue, 24 Jan 2012 00:00:00 -0500

I have an Aeron chair at my desk at home. Confession.

I got it in April when I moved to New York City. The only piece of furniture that I had that I couldn't move in my (now former) car was my desk chair. I found a good deal on an Aeron chair and I rationalized to myself that the cost of the chair was actually about the cost of movers. Savings right?

It also helped, that I was leaving a job where I had an Aeron chair in my office, and I knew that in the short term I would be working from home. While my old desk chair was (and is) quite nice, it's not quite the same. Sit in an Aeron chair for a couple of two years, and it's hard to go back. I've sat in other chairs since then, and it's never quite the same.

Having said that, after a cleaning incident today, I would like to collect a few gripes about the Aeron chair for your consideration.

The assembly right beneath the chair collects dust and dirt in a proportion that doesn't seem quite possible. It's clearly an artifact of the mesh, and likely a commentary on the air circulation of my apartment.

Regardless, dusting nightmare.
The arms scuff and scratch on desks, if the bottom of the desk isn't completely smooth. This isn't an actual problem: the chair still works fine and is as comfortable as ever, but it's a annoying.

I've never looked at the underside of a desk before seriously. With every other chair I've either ordered a variant sans arms, or I've take then arms off as soon as possible.

The Aeron arms are low enough that they've never bothered me, so I thought "might as well." But it's still annoying.

That's all.

Documentation Emergence

Tue, 17 Jan 2012 00:00:00 -0500

I stumbled across a link somewhere along the way to a thread about the Pyramid project's documentation planning process. It's neat to see a community coming to what I think is the best possible technical outcome. In the course of this conversation Iain Duncan, said something that I think is worth exploring in a bit more depth. The following is directly from the list, edited only slightly:

I wonder whether some very high level tutorials on getting into Pyramid that look at the different ways you can use it would be useful? I sympathize with Chris and the other documenters because just thinking about this problem is hard: How do you introduce someone to Pyramid easily without putting blinders on them for Pyramid's flexibility? I almost feel like there need to 2 new kinds of docs:

easy to follow beginner docs for whatever the most common full stack scaffold is turning out to be (no idea what this is!)

some mile high docs on how you can lay out pyramid apps differently and why you want to be able to do that. For example, I feel like hardly anyone coming to Pyramid from the new docs groks why the zca under the hood is so powerful and how you can tap into it.

Different sets of users have different needs from documentation. I think my "Multi-Audience Documentation" post also addresses this issue.

I don't think there are good answers and good processes that always work for documentation projects. Targeted users and audience changes a lot depending on the kind of technology at play. The needs of users (and thus the documentation) varies in response to the technical complexity and nature every project/product varies. I think, as the above example demonstrates, there's additional complexity for software whose primary users are very technical adept (i.e. systems administrators) or even software developers themselves.

The impulse to have "beginner documentation," and "functional documentation," is a very common solution for many products and reflects two main user needs:

to understand how to use something. In other words, "getting started," documentation and tutorials.
to understand how something works. In other words the "real" documentation.

I think it's feasible to do both kinds of documentation within a single resource, but the struggle then revolves around making sure that the right kind of users find the content they need. That's a problem of ?documentation usability and structure. But it's not without challenges, lets think about those in the ?discussion.

I also find myself thinking a bit about the differences between web-based documentation resources and conventional manuals in PDF or dead-tree editions. (See, ?web-based-documentation eventually.) I'm not sure how to resolve these challenges, or even what the right answers are, but I think the questions are very much open.

9 Awesome Git Tricks

Fri, 06 Jan 2012 00:00:00 -0500

I'm sure that most "hacker bloggers" have probably done their own "N Git Tricks," post at this point. But git is one of those programs that has so much functionality and everyone uses it differently that there is a never ending supply of fresh posts on this topic. My use of git changes enough that I could probably write this post annaully and come up with a different 9 things. That said here's the best list right now.

See Staged Differences

The git diff command shows you the difference between the last commit and the state of the current working directory. That's really useful and you might not use it as much as you should. The --cached option shows you just the differences that you've staged.

This provides a way to preview your own patch, to make sure everything is in order. Crazy useful. See below for the example:

 git diff --cached

Eliminate Merge Commits

In most cases, if two or more people publish commits to a shard repository, and everyone commits to remote repositories more frequently then they publish changes, when they pull, git has to make "meta commits" that make it possible to view a branching (i.e. "tree-like") commit history in a linear form. This is good for making sure that the tool works, but it's kind of messy, and you get histories with these artificial events in them that you really ought to remove (but no one does.) The "--rebase" option to "git pull" does this automatically and subtally rewrites your own history in such a way as to remove the need for merge commits. It's way clever and it works. Use the following command:

 git pull --rebase

There are caveats:

You can't have uncommitted changes in your working copy when you run this command or else it will refuse to run. Make sure everything's committed, or use "git stash"
Sometimes the output isn't as clear as you'd want it to be, particularly when things don't go right. If you don't feel comfortable rescuing yourself in a hairy git rebase, you might want to avoid this one.
If the merge isn't clean, there has to be a merge commit anyway I believe.

Amend the Last Commit

This is a recent one for me..

If you commit something, but realized that you forgot to save one file, use the "--amend" switch (as below) and you get to add whatever changes you have staged to the previous commit.

 git commit --amend

Note: if you amend a commit that you've published, you might have to do a forced update (i.e. git push -f) which can mess with the state of your collaborators and your remote repository.

Stage all of Current State

I've been using a versing of this function for years now as part of my git-mail scheme. For some reason in my head, it's called "readd." In any case, the effect of this is simple:

If a file is deleted from the working copy of the repository, remove it (git rm) from the next commit.
Add all changes in the working copy to the next commit.

       git-stage-all(){
            if [ "`git ls-files -d | wc -l`" -gt "0" ]; then; git rm --quiet `git ls-files -d`; fi
            git add .
       }

So the truth of the matter is that you probably don't want to be this blasé about commits, but it's a great time saver if you use the rm/mv/cp commands on a git repo, and want to commit those changes, or a have a lot of small files that you want to process in one way and then snapshot the tree with git.

Editor Integration

The chances are that your text editor has some kind of git integration that makes it possible to interact with git without needing to drop into a shell.

If you use something other than emacs I leave this as an exercise for the reader. If you use emacs, get "magit," possibly from your distribution's repository, or from the upstream.

As an aside you probably want to add the following to your .emacs somewhere.

(setq magit-save-some-buffers nil)
(add-hook 'before-save-hook 'delete-trailing-whitespace)

Custom Git Command Aliases

In your user account's "~/.gitconfig" file or in a per-repository ".git/config" file, it's possible to define aliases that add bits of functionality to your git command. This is useful defining shortcuts, combinations, and for triggering arbitrary scripts. Consider the following:

 [alias]
         all-push  = "!git push origin master; git push secondary master"
         secondary = "!git push secondary master"

Then from the command line, you can use:

  git secondary
  git all-push

Git Stash

"git stash" takes all of the staged changes and stores them away somewhere. This is useful if you want to break apart a number of changes into several commits, or have changes that you don't want to get rid of (i.e. "git reset") but also don't want to commit. "git stash" puts staged changes onto the stash and "git stash pop" applies the changes to the current working copy. It operates as a FILO stack (e.g. "First In, Last Out") stack in the default operation.

To be honest, I'm not a git stash power user. For me it's just a stack that I put patches on and pull them off later. Apparently it's possible to pop things off the stash in any order you like, and I'm sure I'm missing other subtlety.

Everyone has room for growth.

Ignore Files

You can add files and directories to a .gitignore file in the top level of your repository, and git will automatically ignore these files. One "ignore pattern" per line, and it's possible to use shell-style globing.

This is great to avoid accidentally committing temporary files, but I also sometimes put entire sub-directories if I need to nest git repositories within git-repositories. Technically, you ought to use git's submodule support for this, but this is easier. Here's the list of temporary files that I use:

 .DS_Store
 *.swp
 *~
 \#*#
 .#*
 \#*
 *fasl
 *aux
 *log

Host Your Own Remotes

I've only once accidentally said "git" when I meant "github" (or vice versa) once or twice. With github providing public git-hosting services and a great compliment of additional tooling, it's easy forget how easy it is to host your own git repositories.

The problem is that, aside from making git dependent on one vendor, this ignores the "distributed" parts of git and all of the independence and flexibility that comes with that. If you're familiar with how Linux/GNU/Unix works, git hosting is entirely paradigmatic.

Issue the following commands to create a repository:

 mkdir -p /srv/git/repo.git
 cd /srv/git/repo.git
 git init --bare

Edit the .git/config file in your existing repository to include a remote block that resembles the following:

 [remote "origin"]
         fetch = +refs/heads/*:refs/remotes/origin/*
         url = [username]@[hostname]:/srv/git/repo.git

If you already have a remote named origin, change the occurrence of the word remote in the above snippet with the name of your remote. (In multi-remote situations, I prefer to use descriptive identifier like "public" or machine's hostnames.)

Then issue "git push origin master" on the local machine, and you're good. You can us a command in the following form to clone this repository at any time.

 git clone [username]@[hostname]:/srv/git/repo.git

Does anyone have git-tricks that they'd like to share with the group?

6 Awesome Arch Linux Tricks

Wed, 04 Jan 2012 00:00:00 -0500

A couple of years ago I wrote "Why Arch Linux Rocks" and "Getting the most from Arch Linux." I've made a number of attempts to get more involved in the Arch project and community, but mostly I've been too busy working and using Arch to do actual work. Then a few weeks ago when I needed to do something minor with my system--I forget what--and I found myself thinking "this Arch thing is pretty swell, really."

This post is a collection of the clever little things that make Arch great.

abs

I'm using abs as a macro for all of the things about the package build system that I enjoy.

Arch packages are easy to build for users: you download a few files read a bash script in the PKGBUILD file and run the makepkg command. Done. Arch packages are also easy to specify for developers: just specify a "build()" function and some variables int eh PKGBUILD file.

Arch may not have as many packages as Debian, but I think it's clear that you don't need comprehensive package coverage when making packages is trivially easy.

If you use Arch and you don't frequent that AUR, or if you ever find yourself doing "./configure; make; make install" then you're wasting your time or jeopardizing the stability of your server.

yaourt

The default package management tool for Arch Linux, pacman, is a completely sufficient utility. This puts pacman ahead of a number of other similar tools, but to be honest I'm not terribly wild about it. Having said that, I think that yaourt is a great thing. It provides a wrapper around all of pacman's functionality and adds support for AUR/ABS packages in a completely idiomatic manner. The reduction in cost of installing this software is quite welcome.

It's not "official" or supported, because it's theoretically possible to really screw up your system with yaourt but if you're cautious, you should be good.

yaourt -G

The main yaourt functions that I use regularly are the "-Ss" which provides a search of the AUR, and the -G option. -G just downloads the tarball with the package specification (e.g. the PKGBUILD and associated files) from the AUR and untars the archive into the current directory.

With that accomplished, it's trivial to build and install the package, but you get to keep a record of the build files for future reference and possible tweaking. So basically, you this is the way to take away the tedium of getting packages from the AUR, while giving you more control and oversight of package installation.

rc.conf

If you've installed Arch, then you're already familiar with the rc.conf file. In case you didn't catch how it works, rc.conf is bash script that defines certain global configuration values, which in turn controls certain aspects of the boot process and process initialization.

I like that it's centralized, that you can do all kinds of wild network configuration in the script, and I like that everything is in one place.

netcfg

In point of fact, one of primary reasons I switched to Arch Linux full time, was because of the network configuration tool, netcfg. Like the rc.conf setup, netcfg works by having a network configuration files which define a number of variables which are sourced by netcfg when imitating a network connection.

It's all in bash, of course, and it works incredibly well. I like having network management easy to configure, and setup in a way that doesn't require a management daemon.

Init System

Previous points have touched on this, but the "BSD-style" init system is perfect. It works quickly, and boot ups are stunningly fast: even without an SSD I got to a prompt in less than a minute, and probably not much more than 30 seconds. With an SSD: it's even better great. The points that you should know:

Daemon control scripts, (i.e. init scripts) are located in /etc/rc.d. There's a pretty useful "library" of shell functions in /etc/rc.d/function`` and a good template file in/etc/rc.d/skel` for use when building your own control scripts. The convention is to have clear and useful output and easy to understand scripts, and with the provided material this is pretty easy.
In /etc/rc.conf there's a DAEMON variable that holds an array. Place names, corresponding to the /etc/rc.d file name, of daemons in this array to start them at boot time. Daemons are started synchronously by default (i.e. order of items in this array matters and the control script must exit before running the next script.) However, if a daemon's name is prefixed by an @ sign, the process is started in the background and the init process moves to the next item in the array without waiting.

Start-up dependency issues are yours to address, but using order and background start-up this is trivial to implement. Background start ups lead to fast boot times.

Update Pending

Tue, 27 Dec 2011 00:00:00 -0500

It's been a while since I've written one of these "clip posts," but there's no time like the present to get started with that. I hope everyone out there in internet-land is having a good end of the year. I'll try and get a retrospective/new years out in the next few days, and avoid belaboring the point here.

As I said last friday it's my intent to focus here on shorter/quicker thoughts, and focus my free writing/project time for work on longer projects (fiction, non-fiction, perhaps some programming.) So far so good.

Other Cool Things on the Internet

Longer Forms

Fri, 23 Dec 2011 00:00:00 -0500

A friend asked me a question (several weeks ago by publication) on a technical topic and I spent most of the next few days writing a missive on database administration strategy. That seemed like a normal response. I was delighted to find that: I liked the voice, I enjoyed writing the longer document, and there are a dozen or so other related topics that I wanted to explore. So, apparently, I'm writing a book. This is exactly what I need: more projects. Not.

But it's a good thing: I find the writing inspiring and invigorating. I have a perspective and collection of knowledge that hasn't been collected and presented in a single place. I like long form writing. The larger piece might also be a good contribution to my portfolio (such as it is.)

I think this kind of writing suits my attention span.

This has left me without a lot of spare time for blogging, and (as I'm prone to do every so often,) rethinking the future of my efforts on tychoish.com and as a blogger. This is boring for all of you, but I'll give some higher level stuff here and we can follow up on the ?discourse page:

Blogging is fun, and even though I've not been posting regularly, I'm always writing blog posts. Sometimes I find myself writing posts in emails to friends, but I'm never really going to stop writing blog posts.
The general explosion of blog publishing that we saw a few years ago has declined. Audience fragmentation happened, readership got entrenched. I feel like I weathered the storm pretty well and I'm really happy with the site and readers I have, but I'm also pretty confident that blogging isn't going to be the means by which I "level up."1
eBooks have finally happened. For the last decade most people have been saying that ebooks are great for reference material (given search-ability,) and for providing an introduction to a text that people will eventually buy in a paper edition. That may be true, but I think it's changing rapidly, and with kindles and tablets and smart-phones, I think eBooks have effectively won, such as it is.

In another ten years, perhaps, we'll just call them books.
I'm pretty clear that keeping a blog, and perhaps most of the writing I do in my spare time is for my own enjoyment and betterment and helps to develop a personal portfolio and account of my work. I have no (real) interest in using my writing on tychoish.com or any other side that I maintain, as a way of supporting myself to any greater or lesser extent.

I want to be in the business of writing things and working with technology and ideas and people, not the business of publishing. While the line is not always clear between "writing projects that you publish yourself online," and "new media publisher," I want to stay away from the later as much as possible.

So I think this means that most of my "tychoish," writing time will go to writing this book project, and to fiction, and once my blog post backlog is fully depleted (heh,) most of my postings will either be announcements/?progress-reports or a bunch of shorter more off-the-cuff notes.

Here's hoping at least.

I can't really believe that I just used "level up" in this context. ↩

Whiteness and Diversity

Thu, 22 Dec 2011 00:00:00 -0500

This post is a follow up to my earlier post on diversity and representation. In short, while I think it's great that we're beginning to talk and write about race and representation in our fiction and field, I think we1 need to expand our analysis of whiteness.

Whiteness in Science Fiction

I'm still working on figuring out what this means, and I'm sorry that I haven't developed my thinking sufficiently to be more clear on this. In light of that here are a collection of my thoughts on representation:

Whiteness is multiple and I think it's possible (and important) to depict whiteness and white characters critically and without recapitulating normalization. At the same time, it's important to avoid falling victim to a lot of the normalization to which uncritical representations of racial diversity often fall pray.
The theory around race and representation must deal with issues around assimilation. More diversity is useful, but to move forward on issues of representation, the field needs to better understand the process of assimilation. I want to see stories that help us unpack assimilation.
Whiteness is complex and a major problem with stories that "don't do race well," is not just that the characters aren't explicitly of color, but that whiteness isn't portrayed very well. This is part of the struggle of privilege, but not only does science fiction need to be better about diversity and representation of non-white characters, but we the thinking on whiteness needs to continue to evolve apace.

Diversity and Quotas

Discussions about diversity and representation in fiction often lead the under-informed to ask "So what, do you want to impose some sort of quota system? Does that mean diversity is more important than quality?"

The answer is almost always no.

I'd also like to point out that this is one of those cases where whiteness and systematic bias conspire to define "quality," in unuseful ways. But this is another argument for another time.

The canonical answer is: there's a great deal of amazing work written by people of color and a lot of great fiction that incorporates and addresses the experiences of people of color. This is great, and if we've learned anything in the last couple of years, it's that if you look for this work it's there. The real challenge revolves around cultivating that work so that there's more of it, and promoting2 that work so that there's a large audience.

?Comments? Suggestions?

The science fiction writing/reading/editing community. ↩
Promoting and marketing literature is by no means a solved problem under any conditions. ↩

Technical Writing Fiction

Wed, 21 Dec 2011 00:00:00 -0500

On Outer Alliance Podcast #8, David Levine talked about having worked as a technical writer for some 15 years and then said something to the effect of "It's a point of great personal pride that I've never put a bulleted list in a piece of fiction."

I laughed out loud. Perhaps frightening a woman walking her dog nearby.

In most ways, the kind of writing that I do for work, API references, tutorials, administration overviews, best-practice descriptions, is very different from the kinds of things I write away from work, or at least I like to think so.

The truth is that I've learned a bunch about writing and about communicating in general from writing documentation. While my "professional background," doesn't include formal technological training, I definitely "broke in" because I was familiar with technology and could write, rather than being a particularly skilled or trained writer. Any more (just 2.5 years on,) I think the inverse is more true, but that's conjecture.

Technical writing has definitely shaped the evolution of my taste: a couple years ago, I found myself most drawn to complex tightly constructed prose in fiction. These days I mostly go for sparse clear concise prose that isn't particularly ornamented. Perhaps it's only really possible to tune the internal editor for one kind of style at a time.

Having said that, I will confess to feeling--and resisting--the urge to put a bulleted list or some other structured convention of software manuals in fiction.

It's the little things, really.

Xen and KVM: Failing Differently Together

Tue, 20 Dec 2011 00:00:00 -0500

When I bought what is now my primary laptop, I had intended to use the extra flexibility to learn the prevailing (industrial-grade) virtualization technology. While that project would have been edifying on its own, I also hoped to use the extra flexibility to some more consistent testing and development work.

This project spurned xen-laptop, but the truth is that Xen is incredibly difficult to get working, and eventually the "new laptop" just became the "every day laptop," and I let go of the laptop Xen project. In fact, until very recently I'd pretty much given up on doing virtualization things entirely, but for various reasons beyond the scope of this post I've been inspired to begin tinkering with virtualization solutions again.

As a matter of course, I found myself trying KVM in a serious way for the first time. This experience both generated a new list of annoyances and reminded me about all the things I didn't like about Xen. I've collected these annoyances and thoughts into the following post. I hope that these thoughts will be helpful for people thinking about virtualization pragmatically, and also help identify some of the larger to pain points with the current solution.

Xen Hardships: It's all about the Kernel

Xen is, without a doubt, the more elegant solution from a design perspective and it has a history of being the more robust and usable tool. Performance is great, Xen hosts can have up-times in excess of a year or two.

The problem is that dom0 support has, for the past 2-3 years, been in shambles, and the situation isn't improving very rapidly. For years, the only way to run a Xen box was to use an ancient kernel with a set of patches that was frightening, or a more recent kernel with ancient patches forward ported. Or you could use cutting edge kernel builds, with reasonably unstable Xen support.

A mess in other words.

Now that Debian Squeeze (6.0) has a pv-ops dom0 kernel, things might look up, but other than that kernel (which I've not had any success with, but that may be me,) basically the only way to run Xen is to pay Citrix1 or build your own kernel from scratch, again results will be mixed (particularly given the non-existent documentation,) maintenance costs are high, and a lot of energy will be duplicated.

What to do? Write documentation and work with the distributions so that if someone says "I want to try using Xen," they'll be able to get something that works.

KVM Struggles: It's all about the User Experience

The great thing about KVM is that it just works. "sudo modprobe kvm kvm-intel" is basically the only thing between most people and a KVM host. No reboot required. To be completely frank, the prospect of doing industrial-scale virtualization on-top of nothing but the Linux kernel and with a wild module in it, gives me the willies is inelegant as hell. For now, it's pretty much the best we have.

The problem is that it really only half works, which is to say that while you can have hypervisor functionality and a booted virtual machine, with a few commands, it's not incredibly functional in practical systems. There aren't really good management tools, and getting even basic networking configured off the bat, and qemu as the "front end" for KVM leaves me writhing in anger and frustration.2

Xen is also subject to these concerns, particularly around netowrking. At the same time, Xen's basic administrative tools make more sense, and domU's can be configured outside of interminable non-paradigmatic command line switches.

The core of this problem is that KVM isn't very Unix-like, and it's a problem that is rooted in it's core and pervades the entire tool, and it's probably rooted in the history of its development.

What to do? First, KVM does a wretched job of anticipating actual real-world use cases, and it needs to do better at that. For instances it sets up networking in a way that's pretty much only good for software testing and GUI interfaces but sticking the Kernel on the inside of the VM makes it horrible for Kernel testing. Sort out the use cases, and there ought to be associated tooling that makes common networking configurations easy.

Second, KVM needs to at least pretend to be Unix-like. I want config files with sane configurations, and I want otherwise mountable disk images that can be easily mounted by the host.

Easy right?

The commercial vendor behind Xen, under whose stewardship the project seems to have mostly stalled. And I suspect that the commercial distribution is Red Hat 5-based, which is pretty dead-end. Citrix doesn't seem to be very keen on using "open source," to generate a sales channel, and also seems somewhat hesitant to put energy into making Xen easier to run for existing Linux/Unix users. ↩
The libvirtd and Virt Manager works pretty well, though it's not particularly flexible, and it's not a simple command line interface and a configuration file system. ↩

tychoish

Git Feature Requests

The Limitiations of the GitHub Fork Model

Making Things Easier

Supporting Distributed Bug Tracking

The State of the Art

Requirements

The Future of the Art

Resources

New Haven All Day Singing 2012

Size and Space

Heat

My Voice Part

Song Selection

On the Origins of Morris Dancing

Lies Developers Tell About Documentation

Task Updates

Transit and NYC Subway Strategy

The Million Words of Crap

Loops and Git Automation

Making a Bunch of Symbolic Links.

Commit all Git Changes

A Note About xargs

Cron is the Wrong Solution

Git In Practice

Assisted Editing

Allowable Complexity

Persistent Emacs Daemons

The Laptop Riser that Changed My Life

2011 Work/Spacetime Retrospective

Aeron Woes

Documentation Emergence

9 Awesome Git Tricks

See Staged Differences

Eliminate Merge Commits

Amend the Last Commit

Stage all of Current State

Editor Integration

Custom Git Command Aliases

Git Stash

Ignore Files

Host Your Own Remotes

6 Awesome Arch Linux Tricks

abs

yaourt

yaourt -G

rc.conf

netcfg

Init System

Update Pending

Recent Posts Around Here

Other Cool Things on the Internet

Longer Forms

Whiteness and Diversity

Whiteness in Science Fiction

Diversity and Quotas

Technical Writing Fiction

Xen and KVM: Failing Differently Together

Xen Hardships: It's all about the Kernel

KVM Struggles: It's all about the User Experience

A Note About `xargs`