I’ve been a blogging slacker in the last few months. I’ve been working a lot (software releases! content migrations!) and spending my free-time singing and working on a few odds-and-ends projects. And not blogging.

But I did this /posts/delegated-builds project and it seemed like blogging about it would be good.

And it was…

I’ve had this blog, in one form or another for 10 years, and my relationship to this blog has grown and changed a lot in that time and I don’t think it’s useful to really think about all the turns too much, but the recent developments are novel:

• I write for a living, and have pretty consistently for the last 4-ish years (holy crap!) It’s not exactly that I’m burnt out on writing, but it does mean that I write differently now, which is a good thing, but I don’t always have the same ability to sit down after work and want to write more, for fun.

• I noticed that blogging rigorously meant that I didn’t really have time to work on “research” projects, which is to say, I was putting a lot of energy into writing about ideas and theories but not too much time into actually doing things.

  I love theory. I love working on theory, but I’m not sure I see use for theory that doesn’t interact with the world outside of it. For example, before I started my current job I wrote some about technical writing here, and those theories definitely guide what I’m working on today and I think I can stand by what I said, but I think my understanding and knowledge of documentation has grown a lot for the experience of working on it.

• To continue on this theme, the programming projects I’ve been working on, which I haven’t blogged about too much here, have been helpful in teaching me a lot about software development, (which I’ve long been fascinated by,) which improves the documentation I write and the way I approach problems.

  Furthermore, it means that things I used blog about and say “wouldn’t it be nice if a tool that did existed?” and now I spend a lot of time thinking about how to make them exist. Which isn’t to say that I’m a really fluent programmer (yet,) but I’m not helpless.

So in short, I’m back, and I’m hopeful that in the coming weeks and months I can use this space to talk about the things I’m working on and help build a little bit of (mostly personal) momentum behind these projects.

Onward and Upward!

This post accumulates what I thought would be the common questions about the /posts/delegated-builds post/tool. For more background see the /posts/build-woes post.

Couldn’t you just have a separate build-only repository?

Sure, but you’d still have to manage that repository which would probably require a non-trivial amount of code and wouldn’t support building/testing topic branches. Furthermore unless you linked the build directories in some way, which this solution does, you’d end up chronically overbuilding.

Doesn’t this use lot of disk space?

Sure, some. But I think in most competitions between disk space and improved productivity, productivity always wins. That not withstanding:

1. Little known fact: when run git clone and specify the remote as a “local” repository, git uses hardlinks, if possible, for it’s objects database. This means, that you’re only copying the source tree, indeed there are two or three copies of the source tree lying around as it is.

2. Most build processes aren’t terribly space intensive: the second checkout is only 7 megs, our .git directory is 7.3 megs (packed), which translates to a 5.4 meg source directory. By contrast the output of a full build of a branch is about 150 megabytes plus production staging.

At least in our case, the additional space costs are effectively trivial both given the size of contemporary hard drives and scale of other size requirements.

Source code may be larger: the MongoDB source tree has a 16 megabyte source tree (not counting 50+ megs of in-tree third party libraries) that becomes tens of gigabytes with build artifacts. Even so, given a project of this scale space costs wouldn’t be hard to justify. Having said that, most software build problems (that I’m aware of,) don’t face this kind of contention, so it’s pretty irrelevant.

This doesn’t make anything faster, so how does it help?

Indeed it probably makes things slower (tests are not yet conclusive,) but it means that any build process can happen entirely in the background and without possibly affecting your current work.

Sometimes the best way to optimize an inefficient process is to apply intelligence and actually make something slow faster. This is great, but it’s also quite hard (and time consuming) and often intelligence can only increase performance by a few percentage points. As a caveat, always make sure that things aren’t slow for a dumb and simple reason, but if an improvement isn’t obvious or there isn’t a simple easy to fix source of slowness, intelligence is often overrated in this regard.

Other times, perhaps even often, the best way to optimize an inefficient process is to make it not matter that it’s slow. Some things take a long time to do, and while it’s great to do things synchronously, it’s not always a real requirement.

This is a smart hack that falls into the second category: if builds are going to take 4 to 6 minutes to run, I don’t want that to prevent things from happening in that time. I don’t want to have to think about coordinating activities around a given period of dead time: this hack solves this handily.

Four to 6 minutes isn’t that long, but it’s starting to get to a point where it’s too long to maintain focus on a task and wait around for a build to finish, particularly for the longer ends.

With this I think we could tolerate ~15 minute builds without really causing a problem. Beyond that and we might need to reopen this case.

I thought I’d back up after the /posts/delegated-builds post and expand on the nature of the build engineering problem that I’ve been dealing with (at my day job) on a documentation related problem.

• we do roughly continuous deployment. All active development and editing happens in topic branches, and there’s no really good reason to leave typos and what not on the site any longer than we need to.
• we publish and maintain multiple versions of the same resource in parallel, and often backport basic changes to maintenance branches. This is great for users, and great for clarity, but is awful practically, because to deploy continuously, you have to be rebuilding.
• all build is self-contained. This isn’t strictly a requirement, and we do use some internal continuous integration tools for internal development, but at the core, for a number of reasons I think it’s important that all writers be able to build the project locally:
• as an open source project, it’s important that users can easily contribute at any level. We do lots of things to make it easy for people to submit patches, but if the build isn’t portable (within reason,) then it’s difficult for developers to work as peers.
• if it’s difficult to view rendered content while developing, it’s hard to develop well or efficiently. While I think the what-you-see-is-what-you-get model (WYSIWYG) is the wrong answer, good feedback loops are important and being able to build locally, after you make changes, whenever you want, regardless of the availability of a network connection, is terribly important.
• the tool we use, Sphinx, in combination with the size of our resource is a bottleneck. A single publication run takes anywhere from 4:30-6 depending on the hardware, and has grows on average 30 seconds every six months. I could rant about parallelism in documentation, but basically, if you want a system that handles cross referencing and internal links, and you want to generate static content, long compile times are mostly unavoidable.

Now there are a number of tricks that we’ve established to fight this underlying truth: Sphinx does some dependency checking to avoid “overbuilding,” which helps some, and I’ve done a lot of mangling in the Makefiles to make the build process more efficient for most common cases, but even so, long growing build times are inevitable.

The Sphinx part of the build has two qualities that are frustrating:

• each build is single threaded, so it has to read all the files one by one, and then write each file one by one. You can build other output formats in parallel (with a small hack from the default makefile,) but you can’t get around the speed of a single build. There is a patch in consideration for the next version that would allow the write-stage of the build to run concurrently, but that’s not live yet.
• during the read stage of the build, you can’t touch the source files, and extra files in the source tree can affect or break the build, which means that for the most part you can’t build and work at the same time, until now.

The solutions are often not much better than the problem:

• use a different build tool, that was built to do incremental builds. The problem is that there aren’t a lot of good options in this area, and the build is really the primary objectionable feature of the build.

• improve the build tool, or wait for it to improve. The aforementioned patch to let the write phase run concurrently will help a lot. Having said that, it’s important to keep the project on a standard public release of Sphinx and it’s difficult to modify core Sphinx behavior from the extension system.

  Perhaps I have Stockholm Syndrome with the build, but I tend to thing that on some level this is a pretty difficult problem, and building a safe concurrent build system is hard there aren’t a lot of extant solutions. At the same time, this blog is about 2.5 times as large as the documentation project and can do a complete rebuild in 20% of the time or less as much time. While the blog is probably a little less complex, they’re largely similar and it’s not 5-6 times less complex.

  I think the problem is that people writing new documentation systems have to target and support new users and smaller test projects, that by the time people have serious problems with the road blocks, the faulty designs are too baked in.

• brute force the problem by making use non-local build infrastructure that has faster less contentious processor and disk resources. This sounds good, except our test machines are pretty fast, and the gains made by better hardware don’t keep up with continued growth. While we might gain some speed by moving builds off of our local machines, the experience is quite worse. Furthermore, we do build-non locally, and that’s great, but it’s not a replacement.

There aren’t a lot of solutions and most of them seem to come down to “deal with it and build less,” which is hardly a solution.

This is the foundation of the /posts/delegated-builds script that I wrote, which addresses the problem by making it less intrusive. I’m also working on a brief FAQ, which might help address some of the big questions about this project.

Introduction

I cooked up something at work that I think is going to be awesome for building the project that I work on a day to day basis. Here’s the basic problem, in a different post, I’ll expand on these in more depth:

• we do continuous deployment.
• we maintain and publish multiple branches.
• our builds take a non-trivial amount of time (4-6 minutes depending on hardware,) and will continue to get longer.
• our documentation toolkit, Sphinx, lacks concurrency for some steps, which means builds take too long and leave most of a contemporary computer idle. Furthermore, given our use of topic-branches it can be hard to get work done during a build.

So there are a couple of notable hacks that I’ve come up with, over the past few months that help:

• duplicate some of the initial work so that different output formats can build in parallel (using Make’s job control) at the expense of disk some space.
• using a source proxy, (i.e. copying the source content into the build directory and building from this copy so that the actual source files can change during the build.)

These changes are simple and amount to some really minor changes to commands and Make files. This next fix required a non-trivial amount of code, but is really awesome:

• building content, if it’s already committed, from a local checkout in the build directory. This way you can build (and publish!) from a different branch without doing anything to your current working directory.

The Code

See the gist for a basic overview, and keep your eyes on the repositories:

• cyborg institute delegated-build repository
• github.com/tychoish/delegated-build/

Some implementation notes:

• it needs a bit more clean up and configuration with regards to a few hard-coded directory names, and assumptions about projects.

• in practice it should work fine with Python 2.7 and 3.0. If you have the backported argparse module for 2.6, that should work too.

• this plugs in really nicely with some existing infrastructure: becasuse we generate most of our Makefiles, it’s trivial to make this script smart and only permit sane things with regards to branch creation/management, and build targets.

  Building on this, I’ve written up a separate script to generate makefile targets to invoke these commands, which allows the script to fit more nicely into the existing idiom. That’s not included here, yet.

• this, so far, has been the best introduction I’ve gotten to the subprocess module, so perhaps this will be useful to you.

• there’s no good way to queue builds in make, except to use the blocking mode and use a bunch of make calls, which is reasonably inefficient. To get better at this, we’ll need to make some underlying build changes, but the gains could be pretty significant.

• finally, this is the first bit of Python I’ve written since I had the breakthrough where I finally understood classes of a moderate amount of complexity without classes. No harm, and it’s not like there’s any internal state; at the same time, a bit of encapsulation around the interactions with git might be useful.

Pull request, suggestions, are always welcome.

More On Delegated Builds

I’ve written a bunch more about this problem and script and will be posting some of that very soon!

(Also it’s good to return to blogging/posting. Thanks for sticking around!)

This is an awful pun, but I’ve recently written the following script to help with some of the work of back-porting patch sets to maintenance branches. Basically you pass it a bunch of commit identifiers and it cherry picks them all in order.

#!/usr/bin/python

import os import sys import subprocess

for commit in sys.argv[1:]: with open(os.devnull, “w”) as fnull: subprocess.call( [‘git’, ‘cherry-pick’, commit ], stdout=fnull, stderr=fnull )

What I’d been doing, previously is assembling commit hashes in an emacs buffer, and then copy-pasting git cherry-pick before each line and then pasting those lines into the shell and hoping nothing goes wrong. The script isn’t much better but it’s a start.

To use, save in a file named git-raspberry in your $PATH, chmod +x the file, and then just run “git raspberry”. Turns out git runs any program in the path that starts with git- as a sub command.

The more you know.

I’ve started a new project, much to my own surprise. After many years of looking at the merino/tencel blend “colrain” I ordered a cone of it, and have cast on a project: a plain tube using size 0s..

I think I may be crazy.

The thing is, I got one of these neck tubes a month or two ago, and it’s the most amazing thing ever. Looks good with most things, not weird, very comfortable, etc.

So I’m making myself one…

I’m calling it “Ballstown” after a tune in the Sacred Harp of the same name. The tune is named after the town in the capital region of New York State, now known as “Ballston Spa.” Why? Because I cast on 217 stitches.

It turns out, I’ve really rather missed plain knitting that you can just knit on for hours without really thinking about, or can knit on in the dark.

One of the reasons that I’ve not been knitting as much recently, other than available time is that I’ve found it difficult to actually wear or use the things I knit. Sweaters, even finer weight ones are too warm to wear inside, and not windproof enough to keep me warm outside without substantial jacket.

The answer is to knit finer fabrics, of course, but this has been easier said than done, for me. Mostly I’ve stuck to fair isle sweaters, which are great fun to knit, and reasonably wearable, but difficult to knit on casually: lots to lug around, and starting to knit something with a pattern requires some “spin up time,” as you remember where you were and what you’re supposed to be doing.

In most ways this plain tube is the perfect answer to this problem….

I’ll blog more about this (or not,) as I progress.

This is a rough sketch of some things that I’ve learned about the Sphinx documentation generation system. I should probably spend some time to collect what I’ve learned in a more coherent and durable format, but this will have to do for now:

• If you describe a type in parameter documentation it will automatically link to the Python documentation for that type when using the Python Domain and if you have intersphinx connected. That’s pretty cool.

• Sphinx let’s you define a scope for a file in some cases. If you’re documenting command-line options to a program. (i.e. with the “program” with subsidiary “option” directives,) or if you’re documenting Python objects and callables within the context of a module, the module and program directives have a scoping effect.

  Cool but it breaks the reStructuredText idiom, which only allows you to decorate and provide semantic context for specific nested blocks within the file. As in Python code, there’s no way to end a block except via white-space,¹ which produces some very confusing markup effects.

  The “default-domain” directive is similarly… odd.

• Sphinx cannot take advantage of multiple cores to render a single project, except when building multiple outputs (i.e. PDF/LaTeX, HTML with and/or directories.) if with a weird caveat that only one builder can touch the doctree directory at once. (So you either need to put each builder on its own doctree directory, or let one build complete and then build the reset in parallel.)

  For small documentation sets, under a few dozen pages/kb, this isn’t a huge problem, for larger documentation sets this can be quite frustrating.²

  This limitation means that while it’s possible to write extensions to Sphinx to do additional processing of the build, in most cases, it makes more sense to build custom content and extensions that modify or generate reStructuredText or that munge the output in some way. The include directive in reStructuredText and milking the hell out of make are good places to started.

• Be careful when instantiating objects in Sphinx’s conf.py file: since Sphinx stores the pickle (serialization) of conf.py and compares that stored pickle with the current file to ensure that configuration hasn’t changed (changed configuration files necessitate a full rebuild of the project.) Creating objects in this file will trigger full (and often unneeded) rebuilds.

• Delightfully, Sphinx produces .po file that you can use to power translated output, using the gettext sphinx builder. Specify a different doctree directory for this output to prevent some issues with overlapping builds. This is really cool.

Sphinx is great. Even though I’m always looking at different documentation systems and practices I can’t find anything that’s better. My hope is that the more I/we talk about these issues and the closer I/we’ll get to solutions, and the better the solutions will be.

Onward and Upward!

I don’t think the tension between having good, robust, and bug-free software and having software with new features and capabilities is solvable in the macro case. What follows is a musing on this subject, related in my mind to the On Installing Linux post.

I’m not exactly making the argument that we should all prefer to use unstable and untested software, but I think there is a way in which the stability¹ of the most prevalent Linux distributions is a crutch. Because developers can trust that the operating system will effectively never change, there’s no need to write code that expects that it might change.

The argument for this is largely economic: by spacing updates out to once a year or once every 18 months, you can batch “update” costs and save some amount of overhead. The downside here is that if you defer update costs, they tend to increase. Conversely, its difficult to move development forward if you’re continuously updating, and if your software is too “fresh,” you may loose time to working out bugs in your dependencies rather than your system itself.

The logic of both arguments holds, but I’m not aware of comparative numbers for the costs of either approach. I’m not sure that there are deployments of significant size that actually deploy on anything that isn’t reasonably stable. Other factors:

• automated updating and system management.
• testing infrastructure.
• size of deployment.
• number and variety of deployment configurations.

I (mostly) lurk on the markdown discussion list, which is a great collection of people who implement and develop projects that use markdown. And there’s been a recent spate of conversation about standardization of markdown implementations and syntax. This post is both a reflection on this topic and a brief overview of the oft-forgotten history.

A History of Markdown Standardization

Markdown is a simple project that takes the convention that most people have been using to convey text formatting and style in plain text email, and providing a very minimalist and lightweight script that translates this “markup” (i.e. “markdown”) into HTML. It’s a great idea, and having systems that make it possible for people to focus on writing rather than formating is a great thing.

People should never write XML, HTML, or XHTML by hand. *Ever*.

Here’s the problem: the initial implementation is a Perl script that uses a bunch of pattern matching regular expressions (as you’d expect) to parse the input. It’s slow, imprecise, there are a few minor logical bugs, there’s no formal specification, and the description of the markdown language are ambiguous on a few key questions. Furthermore, there are a number of features that are simple, frequently requested/desired, with no official description of the behavior.

As people have gone about developing markdown implementations and extensions in other languages, to fix up the inconsistencies, to provide markdown support in every programming language under the sun, without a formal specification and disambiguation of the open question, the result is fragmentation: all the implementations are subtly different. Often you’ll never notice, but if you use footnotes (which are non-standard,) or want to have nested lists, you will end up writing implementation-dependent markdown.

The result is that either you tie your text to a specific implementation, or you go blithely on with the knowledge that the markdown that you write or store today will require intervention of some sort in the future. If you need to extend markdown syntax, you can’t without becoming an implementer of markdown itself.

That’s an awful thing. And there’s no real path out of this: the originator of markdown has publicly stated that he has no interest in blessing a successor, continuing development of the reference implementation, or in contributing to a specification process. Insofar as he controls the authoritative definition of markdown, the project to standardize markdown is dead before it even begins.

The problem is that while most people involved (implementers, application developers, etc.) in markdown want some resolution to this problem: it’s bad for users and it makes implementing markdown difficult (which markdown flavor should you use? should you reimplemented bugs for consistency and compatibility, or provide a correct system that breaks compatibility?) At the same time, markdown implementations are not commercial products and were built to address their author’s needs, and none of those maintainers really have the time or a non-goodwill interest in a standardization process.

If markdown standardization weren’t doomed from the start, the fact that the only people with any real ability to rally community support for a standardized markdown, are not inclined to participate in a standardization process.

Markdown Isn’t For Text That Matters

If markdown were better, more clear, and more rigorously defined and implemented, this wouldn’t be a problem, but the truth is that markdown’s main role has been for README files, blog posts, wikis, and comments on blog posts and in discussion forms.

It’s a great “lowest common denominator” for multi-authored text that needs rich hypertext features but needs markdowns simplicity and intuitiveness. Big projects? Multi-file projects? Outputs beyond single files?

Sure you can hack it with things like maruku and multi-markdown to get LaTeX output, and footnotes, and more complex metadata. And there are some systems that make it possible to handle projects beyond the scope of a single file, but they’re not amazing, or particularly innovative, particularly at scale.

To recap, markdown is probably not an ideal archival format for important text, because:

• The implementation-dependency means that markdown often fails at genericism, which I think is supposed to be it’s primary features.

  Generic text representation formats are a must.

• If you need output formats beyond HTML/XHTML then markdown is probably not for you.

You can get other formats, but it’s even more implementation specific.

The Alternatives

Don’t standardize anything. While markdown isn’t perfect the way it is now, there’s no real change possible that wouldn’t make markdown worse. There are two paths forward, as I see it:

1. Give up and use reStructuredText for all new projects.

   RST is fussy, but has definite and clear solutions to the issues that plague markdown.

   • It has support for every major output format, and it wouldn’t be too hard to expand on that.
   • It’s fast.
   • In addition to the primary implementation, Pandoc supports python and there are early stage Java/PHP implementations. Most tools just wrap the Python implementation, which isn’t really a problem.
   • There are clear paths for extending rst as needed for new projects.

2. Design and implement a new markdown -like implementation. I think reMarkdown would be a good name. This will be a lot of work, and have the following components:

   • a complete test suite that other implementations could use to confirm compatibility with reMarkdown.
   • a formal specification.
   • a lexer/parser design and reference implementation. With an abstract XML-like output format. We want a realistic model implementation that isn’t overly dependent upon a single output format.

• an explicit and defined process for changing and improving the syntax.

(alternately, “Installing Linux the Hard Way”)

I’ve had the occasion to install Linux on three systems in the recent past. People don’t really install Linux anymore, it seems: with “cloud” instances and provisioning that’s based on images means that no one really has to install Linux as such. My experiences have been mostly awful:

• I couldn’t make my current laptop do a full LCM boot for the life of me. I partitioned the hard drive in the conventional way, and while the system works fine, I think non-abstracted disk volumes are bad practice.

  Disk partitioning and bootloaders remain the most difficult and frustrating aspect of the installation process, and there’s no automation to support this work. Furthermore, even if it takes you a day to get it right, usually you don’t have to mess with it for a year or two. Which makes it difficult to improve practically.

  The Debian installer will do this pretty well, but you can’t get the auto partitioning tool to not use the full disk. Or I can’t figure it out.

• I recently tried to install Arch Linux on an infrastructural system. Apparently in the last couple of months Arch totally did away with the installation system. So it dumps you into a mostly working shell and provides a couple of shell scripts to “automate” the installation.

It’s a great idea, as long as you never have to use it.

Conversely, it’s a great idea if you’re constantly running installations.

If you install Arch once every year or two, as I suspect is the most common case, good luck.

I need to do it again: to update an older laptop to the 64-bit version of Arch, and I fear this is going to be terribly painful. I’m left with two main questions:

1. Have we given up on the idea that desktop Linux may be viable for people who aren’t already familiar with Linux, or who aren’t software developers (or the next best thing?)

2. Does the desktop experience actually matter?

I’m asking this in a more narrow line of questioning. There’s computer usage that revolves around things that happen in the browser, which is (probably) better suited for embeded systems (i.e. Android and iOS based devices,) and it’s not clear where the line between that and “General Purpose” computing will fall.

If we end up using embeded systems for most of the computers that we actually touch, this fundamentally changes the desktop experience as we know it, particularly for things like installation.