I’ve been writing build system tool that allows users to specify concurrent build processes using a lightweight, Python-based system that minimizes overhead.

Progress is decent. I hope to use this to replace a hodgepodge of fabric and Makefile for my work and personal projects. I have a decent spec (3 hours), an initial implementation of the internal parts (3 hours,) a good first draft at a command line utility (1.5 hours,) internal/APO documentation (10 hours,) and none of the unit tests and procedural and conceptual documentation. In essence the hard stuff.

Basically what happened, is I spent a lot of time thinking about the problem, a little bit of time coding, and if all goes according to plan a lot of time writing rather droll code and good if uninteresting documentation.

Which is, all things considered, what all software boils down to.

Writing the core implementation is (often) this intense impassioned process that is necessarily flow-like, because there’s a bunch of state that you have to keep hot in your mind while solving hard problems, and if your attention drifts too far, you start breaking things.

Not that flow-like states are the best or only way to write code for core functionality, but it works and it’s enjoyable.

Everything else, is different:

• Writing documentation is an exercise in context switching: you have to read code, or poke at a running program to figure out how it works, and then turn that information on its head so you can tell people how to use it.

  It’s fun, but it’s much more fussy.

• Writing tests is similarly hard: it’s also about balancing “how it works” and “how its used,” but rather than describing something for future users, tests are about defining what constitutes “correctness” and what’s incorrect.¹

  Writing test-code is intellectually challenging work, and requires many of the same base skills as writing implementation-code but requires a different kind of focus and thinking.

• There’s a lot of code that remains once the core logic exists, including: user interfaces, logging, test, managing edge cases, optimization, and tuning the parameters of the behavior (business logic tweaks.)

Which isn’t to say that any particularly portion of the work is more or less difficult or important. But, if you don’t work in this world every day it’s easy to see the hard initial work as being “the real part of software development,” and allow all the other work to sort of fade into the background. Which is unfair, and I think is representative of a larger misunderstanding of how software works and gets made.

Another project for another day.

Onward and Upward!

I wrote a long time ago, about /technical-writing/atomicity which (more or less) is the same as topic based authoring. Both describe the process of breaking information into the smallest coherent blocks and then using the documentation toolkit to compile the kind resource.

Topic based approaches to documentation promise reduced maintenance costs and greater documentation reuse. I’m not sure if anyone’s used “ease of authorship,” as an argument in favor of topic based approaches (they’re conceptually a bit difficult for the author,) but you get the feeling that it was part of the intention.

The obvious parallel is object orientation in programming, and I think the comparison is useful: they both present with optimism about reuse and collaboration through modularity and modern tool chains. While object oriented programming predates topic based authoring, both have been around for a while and even if you aren’t an adherent of object orientation or topic-based authoring, I think it’s impossible to approach programming or documentation without being influenced by either of these paradigms.

Unless you’re working with a really small resource, without some topic-based you end up with redundant documentation that looses consistency and a maintenance nightmare.

The downfalls of “topics,” don’t negate it’s overall utility, but they are significant:

• topic based authoring makes it harder for non-writers to contribute to the documentation. This makes it more challenging to keep documentation up to date and can hurt overall accuracy.
• topics force writers to focus on the “micro” documentation at the expense of the “macro” documentation experience. The content is clear, the completeness is good, but the overall experience for users is awful.
• topic-centrism sometimes leads to deeper hierarchies which leads to duplicated content across the hierarchy as “cousin” nodes address related concepts.

What’s the solution? I’m not sure there is a single one, but:

• it’s important to avoid duplication, by having great support for “single sourcing” (inlining/inclusion,) and simple cross referencing.
• isolate all content in concrete topical units.
• start with flat global organization and add interpage hierarchy only when necessary.
• use as much intrapage organization and hierarchy as you need, and allow intrapage hierarchy.
• build great reference material first. Everything else is gloss, and you should layer the gloss on top of strong reference rather than try and build reference under an existing structure.

For work, I’ve been working on revising our build system so that less of the build definition happens in Makefiles and more of it happens in Python scripts. This post is an elaboration.

I’m a complete partisan of reusing standard tools, and so moving away from Make felt like a big/hard jump. However:

1. Process creation is expensive, and every “job” starts a new shell and process, which takes time.

2. Most of the build logic was in Python anyway: over time most shell lines called Python code rather than commands directly. This seems like a common artifact of more complex build processes.

Beyond this, the generation of the Makefiles itself was encoded in Python code.

For our project, at least, we were indirecting through Make, sort of through the hell of it.

3. It turns out that multiprocessing in Python is crazy easy to use.

The transition isn’t complete of course, we’re still using make to handle dependencies between groups of tasks, and there’s no particular rush or need to rid ourselves of Make, but the gains are huge. Things build faster one or two orders of magnitude in some cases. There’s less flakiness. Rebuild times are much faster, and there are fewer moving parts.

Great win!

This has me thinking about ways of doing build systems in a generic, maintainable way without relying on something like Make. I have a prototype on my Laptop at the moment that provides a way to specify build processes concurrently. The rest of this post will be a high level overview of the design of this system. Please provide feedback and enjoy!

Build systems are basically collections of tasks expressed in a graph structure. The tools exist to enforce and encode the graph structure, or less abstractly to ensure that tasks run in the proper order. If you’re paining a wall, the build system ensures that you spackle, apply the primer, and then apply the final coat, in that order.

There are, as near as I can tell, three different kinds of relationships among/between groups of tasks in a build process:

1. There are groups tasks that don’t depend on each other and can run concurrently with each other.

2. There are some tasks or groups of tasks that must not run before or after another group of tasks.

3. There are sequences of tasks that must run in a specific order, but can run at the same time as other tasks or sequence of tasks.

What Make, and related systems do is provide a mechanism to specify “dependency” relationships between files (and tasks after a fashion,) or groups of files/tasks. After a fashion, Make takes the dependency information and runs tasks more or less according to one of those patterns. In many ways, my project is an experiment to see if it’s possible to “outsmart Make,” by generalizing the kinds of operations and forcing users to specify the concurrency constraints of the tasks explicitly, rather than letting the concurrency emerge out of the dependency graph. Thoughts:

• This depends on users being able to intuit the actual dependencies abstractly, rather than rely on the emergence properties of Make. Arguably, Make also requires you to think abstractly about the potential concurrent modeling of the build, but allows you to avoid it in some situations.
• If some large portion of the compilation process relies external processes, the performance gains will probably be more modest. Process creation is still expensive, but it’s probably marginally cheaper to use subprocess than it is to start a full shell.

In addition to the basic machinery, I’ve written a few helper functions to read build definitions from a YAML file, which will produce a usable build system. I’ll release this once: I’ve written some tests, there’s better logging, and some basic README-level documentation.

Onward and Upward!

I spent a bunch of time this week taking a bunch of my work project’s build system. We’ve gone from having most of the heavy lifting done by Make, to having only doing the general high level orchestration with Make and doing all of the heavy lifting with (simple) custom Python code.

The logic in the previous system was:

• Make is everywhere, stable, and consistent. In the spirit of making the project as compatible and accessible to everyone it made sense to use common tools and restrict dependencies.
• Concurrency and parallelism are both super hard, and Make provided a way to model the build in a knowable way, and the parallels the build as much as possible. Before starting this project, I’d spent two years being a write working with a build system that was not concurrently and ran in one thread, and I was eager to avoid this problem.

It turns out:

• If you write Makefiles by hand they’re the inverse of portable. In the same way that “Portable Bash Script” is a thing that can lead only to insanity.

  As Make-based build systems grow, the only thing you can do to preserve your sanity is wrap up all build instructions as scripts of some kind and/or use some sort of meta-build tool to generate the Makefiles programatically, using some sort of meta-build tool.

  Complexity abounds.

• Forcing you to model your build process as a graph, is actually not a bad thing, and frankly is the strongest selling point. Make doesn’t enforce graphs (and how could you really?) but if you pay attention to the ordering and build performance it’s not hard to keep things running in parallel.

  By contrast, Make’s parallel execution is SO BAD. I think the problem is mostly shell/process creation overhead rather than scheduling. Regardless, for build systems with lots of small pieces, you loose a lot to overhead.

So I took the logic that’d I’d been using to generate the Makefiles, implemented some simple mtime based dependency checking, and used it to call the build functions directly in Python.

The results were huge. Speed gains of 300x, using 30% of the code and better processor utilization. Can’t argue with that. Even the steps that required external (i.e. non-Python) sub-processes components were considerably faster

So I was thinking: build systems tend to be big sources of blight, they tend to be hard to maintain and require a bunch of specialized knowledge that’s distinct from the actual domain knowledge of a project, and most generic build tools have problems, (like Make,) so what gives?

If the generic tools had better performance or were significantly easier to maintain, there’d be a rather convincing argument in their favor. As it is, I’m not really seeing it.

I’ve been enjoying blogging over on the micro tychoish site and thought I’d catalog these posts here.

• A Day in the Life
• Software Packaging
• Personal Software Change
• Small Data

More to come!

(Note: Because I’m terrible at remembering to post entries during the week, this post is actually from last week. But it’s still interesting!)

The past few weeks have been somewhat disjointed for me. I’d been working a lot to wrap up a long expected release, followed by a vacation without a project plan, and a few more busy weeks. On top of this, I spent a bunch of time working on wrapping up, or at least releasing a few personal project to assuage some guilt.

After all that, I found myself at loose ends: I didn’t have new projects because I hadn’t had enough time to think about them or more importantly I was so interested in finishing something that I’d been trying to suppress thinking about new projects.

Well that was a great idea. Not. Now, finally after spending too long lolling about and trying to restart the creative (and project planning) engines, I’ve actually done some things:

RstCloth

Basically this the first break at a very very simple API for generating reStructuredText. It’s modeled on the interface for my buildcloth which does the same sort of thing for generating Makefiles.

reStrucutredText exists to make text easier for humans to write well formed documents, which is great and useful for about 95% of use cases: human editable text formats for machine parsing are an amazing boon to documentation productivity.

There are cases, where it makes more sense to store content in a regular format, like JSON or YAML and build the content programatically, tabular data, integrating content from external sources. If most of your tool chain uses reStructuredText, then something like RstCloth is probably exactly what you need.

And because it’s a second-generation *Cloth tool, I already have most of the awkwardness worked out.

It’s still dev, and I’ll be getting documentation, a readme, and some examples nailed out in the next few weeks. In the meantime:

• rstcloth on pypi
• rstcloth on github

tumblr -- m.tycho.co

I reactivated my tumblr account, hooked it up to the awesome tumblesocks plugin for emacs, and have attached it to the concise m.tycho.co domain. I’m also mirroring the content at tychoish.com/micro.

I’m going to try to avoid over thinking this, but:

• While I’ve had some struggles with the emacs integration, it’s generally really idiomatic.

• I really like that you can queue posts. This is the feature that I miss the most about systems that I’ve used in the past to host tychoish.

• I like that there are some community features, and that the tagging isn’t worthless in light of being able to use tags to jump to relevant posts from other people.

  While I like self-hosted websites, and am kind of freaked out by the whole “my blog is a service,” I think that the connection to a community/audience is useful and powerful, and is not to be overrated.

• I like that tumblr does automatic integration with facebook and twitter. You can sort of do this manually, but baking things in leads to a better experience.

I’ve mentioned that I was working on a new sweater a few months ago, but I’ve neglected to post or write about the project at all. Let’s change that now:

In most respects it’s just like a number of existing sweaters that I’ve made: two color patterns, using a combination of mid-sized extrapolation of Scandinavian mitten patterns, with some influence of Turkish stocking patterns arranged in panels to convey strong vertical lines. The yarn is Harrisville Shetland, and another unidentified Shetland from a cone I got years ago and have now used in three sweaters. The plan is to have a simple fisherman’s-style drop shoulder construction with a simple short crew neck color.

The plan diverges somewhat from “tychoish standard” in two respects:

The biggest change is that it’s going to be a cardigan. I’ve never made a cardigan that I’d call a rocking success. I can do it, but the finishing always leaves something to be desired and it hangs funny or flares in a way that I don’t want.

The plan for finishing the cardigan opening this time around is to use the steek (the bit that you cut open) as the facing for a hem. the idea is minimal prep and let the yarn do its thing. For closure, I’ll do an attached i-cord band with room for buttons.

The slightly smaller change is that rather than use a hem, I used the “purl-when-you-can-and-want-to” for bottom hem treatment. The idea is that if you purl occasionally for the first few inches you can counteract the tendency of knitted fabric from rolling. It’s not perfect yet, but I’ve not steamed it, so we’ll see.

It’s fun to knit so far, and I look foraward to finally conquering my fear/avoidance of cardigans and perhaps finding the perfect lower edge finishing approach for stranded sweaters.

Onward and Upward!

… from blogging. See this post for the background.

I sometimes look at other people’s blogs, and think “wow, that’s sharp,” and while I really like the current tychoish theme, there’s a distinct lack of gradients, really polished typography, strong crisp lines, and elegant side bars.

Not that I have a clue what I’d put in a side bar: Hell, I can’t even find good things to put in the Cyborg Institute side bar. But it’s not just that my design has grown dated (I don’t think it has, that much,) and more that the practice of blogging has changed in a few ways:

The State of Blogging

• self-hosted blogs are the exception rather than the rule.

• it’s become increasingly difficult to aggregate content, the demise of Google Reader, both the removal of the product and the declining trend it its use point to the idea that RSS isn’t a user facing transmission method.

  People are getting content through other means, and publishers probably can’t depend that users will poll any content, which changes the role of the publishing system.

• In fact, I don’t have a real clue what the current state of the art for publishing tools for blogs is these days. My sense is that a greater portion of blogs are hosted on services like Tumblr and WordPress.com.

  The big “advancements,” in blogging technology are probably related to integration and distribution of content to third party systems, which services can probably do better than hosted solutions.

• There are fewer long-lived personal blogs, and even fewer that stray beyond a single niche.

Are there blogs that you read regularly? How do you know when there’s a new post?

Changes Afoot

Given these changes, and the chance to rethink how I approach this blog:

• I’m curious as to the state of commenting and discourse related to blogs. Do people actually comment, in anything other than exceptional situations? Are most conversations on hacker-news/reddit or other domain specific common space and other blogs?

  I’ve been thinking about the prospect of even turning off the discussion/discourse pages here. They don’t get used, they’re kind of weird, people don’t really know how to use them, and I’m not sure they get used. At the same time, providing a space for conversation seems essential. More on this on a later post.

  Edit: I totally did this, and while I have some regrets, I think it’s generally a good move.

• While it’d be nice to automate submitting content to various aggregation sites and social network-sites, I’ve added various browser extensions to do these submissions. It’s a pain in the ass, but I guess auto-submits makes for less useful content aggregator.

• Just as tagging systems are inefficient and broken for wikis and “real” technical resources (see /posts/taxonomic-failure/ for my thoughts,) they’re not all that great for blogs. I’m considering completely removing the tagging system on tychoish, and just letting the search tool (which is pretty good) make content easy to discover.

Onward and Upward!

In On my Return to Blogging post I attributed the fact that I’d taken a break from blogging because I wanted to get out and do things rather than just spend my free time writing and thinking about things.

A Critique

The problem with this kind of statement is that it evokes a certain kind of anti-intellectualism: thinking isn’t as good as doing things, which is counter productive. Actions, creation, feed and grow out of thinking (and vice versa.)

In light of this it’s difficult to re-calibrate ones practice without on the one hand taking an anti-intellectual stance or becoming too ungrounded in practice.

Cogitative Side Effects

I read an article a while back (source lost to the depths of the internet,) that mentioned the following effect; when you talk about something publicly the recognition and validation you get from talking about it is pretty much the same as the recognition and validation you’d get from actually doing something. The result is, if you talk about doing something, you become less likely to actually do it because you’ve already experienced most of the gratification of doing something.

(Sorry for the poor translation.)

In any case, it seems plausible, and certainly worth testing. So when I say “I want to spend time doing things,” rather than theorizing about possible future projects or talking about things I want to work on, as has been my wont, I’m just not.

This is an interesting conundrum for free software/open source: how do you start developing a project in a community centered way without shooting yourself in the proverbial foot. Sometimes it works (e.g. GNU MediaGoblin,) but often people hack a working prototype (and often a lot more) before talking about the project. There are too many examples to list.

There are also a large number of examples of projects that started that languish because they were clearly announced too soon. On the other hand, maybe early-public discussion or announcements is purely epiphenomenal and early public discussion is just a symptom of an always already weak project, that you’re more interested in talking about something that doing something. (Which might just prove the point?)

The Take Away

• Don’t blog about something until it exists, and is in a form that you’d be willing to share and discuss.

Corollary: code names are probably the same as real names.

• Strive for balance between “project work,” and meta-work. The ideal proportions are unclear.
• Avoid anti-intellectualism when possible.

Today I released the first version of Buildcloth which is a tool that I’ve been using at work to programatically (and in some cases) dynamically generate build systems (i.e. Makefiles.)

Background

It’s obviously been “production ready” in some sense for a while, but I recently finished the API documentation, and a lot of the infrastructure for packaging and distribution, so it seemed like this was a good starting point.

The initial idea was basically that while Make syntax can be really powerful, in a number of situations:

• to specify conditional elements,
• to generate build targets and procedures based on system configuration or project state,
• for large numbers similar of targets, and
• for build with where single targets have a group of related rules,

defining build systems programatically ends up producing a much more reliable and maintainable build system. The wins are pretty big in terms of maintainability, clarity, and flexibility.

The idea, and naming, is sort of: do what fabric does for shell scripts and deployment but for build system generators. Maybe this is exactly what you’re looking for.

More Information

Check it out:

• buildcloth on Pypi
• buildcloth github repo

Bugs go here, and patches/pull requests are always welcome.

Cool Improvements:

• full documentation.
• support for specifying targets/dependencies as a list.
• a build-rule abstraction called RuleCloth.
• improved ninja support.

The Roadmap

• making the tutorial and high level documentation better.
• improving the “RuleCloth.”
• adding some preliminary tools for managing data interactions.
• pypy support (why not?)