The backstory: I’m trying to learn Common Lisp. It’s sort of an arcane programming language with a few aspects that I rather like, and I’m viewing this as an exercise to generalize my programming experience/knowledge.

I’ve written some common lisp over the years, mostly because I use stumpwm, but I’ve been struggling to find a good project to start on my own or hack on an existing project.

A few weekends ago, I started hacking on coleslaw, which is a static site generator written in common lisp. The reasons are simple:

• I know something about the static site generator domain, so I’m only trying to learn one thing.
• Coleslaw is typical in many respects of site generators, but the architecture makes sense, and it’s reasonably simple and hasn’t been overly optimized.

There are three features I’m interested in adding:

1. I want make it so that Coleslaw builds content incrementally: there’s no reason that programs like this should have to rebuild all content on every build. There are two missing concepts “which pages changed,” and “if one page changes, what other pages must change.”
2. I want to make the entire build process more configurable. Thise includes expanding the options that are available in the site configuration, and allowing users to define and edit themes within their projects.
3. It feels wrong to leave the program signally threaded, particularly when build systems are inherently parallel, and Common Lisp real threads.

• I want to make this blog more blog-like. The current site works fine, but I’m growing restless, and I want to explore more.

So I began hacking and managed to learn a lot about the codebase, and am getting into the swing of this project. I played with a naive concurrency approach but it didn’t stick. For about a day of work, I:

• Made the docutils package usable from coleslaw and fixed some issues there. See pull request.

• Added a lot of configuration and made more things customizable. Pull requests.

  There’s a lot of little things that I like or feel like I will shortly like, but the sticking points at the moment are:

• A lot of things have started to make sense. let forms feel very natural. The object system is pretty great. I feel comfortable with the organization of code within a function. I don’t mind the parentheses.

• I remain pretty confused by the variable binding system outside of the basic let forms. I’m never sure about setf and setq (my policy thus far has just been to setf and troubleshoot if it doesn’t do what I want.) If Python is all about the power of the namespace, it feels like CL is all about the power of scopes and I haven’t yet learned all of them.

• Oddly, conditionals feel really hard and cumbersome. The if form is more limited and not particularly clear, cond is pretty useful but often feels like a sledge hammer. unless, and when are clear but not perfect.

I’ve not found the idiomatic way to perform “set a variable to the return of a function if it’s current value is 0/null/false, otherwise pass” which I find myself reaching for pretty frequently (and awkwardly.)

More to come, I’m sure!

Learning how to make computer software is hard. Not fundamentally hard: lots of people can do it, and even more people do things that are functionally equivalent to programming though they wouldn’t think of it as such. But teaching people how to write good computer software is a challenge, and one that I’m generally interested in exploring more.

For a long time, I’ve been interested in this problem from the outside: I didn’t really know how to program in any meaningful sort of way and I was interested in deconstructing the process of making software. Then something clicked and years of tinkering with systems administration and reading about programming languages and practices clicked and while I think I have a lot left to learn, I’ve started thinking about the problem from the other side.

We accumulate many skills and kinds of knowledge in an incremental sort of way: you study and practice and little by little our brains (and bodies) form new connections and we “learn.” Other kinds of learning follow a more “step-based” approach: we practice and study for a long period of time without much discernible change in understanding or skill until at some point we experience some sort of larger improvement in ability.

At least for me (and perhaps you as well,) things like dance, knitting, writing, and most structured/classroom-based topics tend to be incremental, mostly. Other things, like programming (at least initially) and singing/music tend to be step-based.

This isn’t to say that step-based areas of focus don’t require regular ongoing practice, just that the observable markers of progress may lag inconsistently behind effort and pedagogy.

When I was doing more non-professional writing, I was fond of the school of writing advice that said “the way to learn how to write (fiction) is to have a good story to tell;” when I think about learning to program I think the first step has to be a need to automate something on a computer.

I’ve even written up something on the topic

Hackers describe this as the “scratch your own itch” method (from CatB and elsewhere.)

Neither the idea of step-based versus incremental learning nor the notion of using a personal need to drive learning are new, but I think they illuminate eachother well.

Since I stopped being a student somewhat abruptly in 2007, I’ve become increasingly glad both that my education and personal development has continued and I’ve had the opportunity to explore things in ways that didn’t make sense in a structured context (e.g. “I want to learn about how databases work without formal CS/systems training,” or “I want to learn how to sing and withouta lot of music theory.”)

For most of the past year I’ve been pretty heads-down on the “learning to program” project, and I’ve had a number of interesting problems that I’ve used to help explore the topic:

• generalized `Sphinx <http://sphinx-doc.org>`_ publishing toolkit.

  This is both a work project and a personal project.

  Sphinx is a great tool for producing text, and I’m quite fond of it. At the same time, I’m not a fan of its architecture (and have a number of approaches to optimize the build process,) and there are a number of tasks: dependency resolution, version management, theme management, and deployment that any reasonably complex Sphinx-based project needs to address.

  While this project requires ongoing development and improvement, it’s basically feature complete, and it’s given

• I’ve had a couple of personal side projects that I’ve used to explore different kinds of programming problems, with greater and lesser success.

  Buildcloth, which is pretty cool and needs more work but I fear may be too complicated for the use-case.

  csc, which isn’t fully off the ground and may not provide a significant improvement upon Sphinx for most cases.

  dtf, which is a decent idea, but I’ve not had time to really implement and exercise the program and I fear that the core code quality isn’t great.

• More recently I’ve begun working on a dependency analytics package with a co-worker/friend to help him and his teammates understand and untangle a larger C++ project.

It’s been nice to be able to actively work on a project with another developer, and to be able to focus on performance and architecture issues while someone else focuses on feature prototyping and use-cases.

In a lot of ways this is a good “capstone” project for me because I’ve gotten to use and apply many of the things I’ve learned from writing concurrent/parallel Python, as well as moderate sized Python programs comes together well here.

Most of the projects that had been open and on my plate for the last few months have mostly wrapped up. There’s more work to be done on them, I could do a lot more work, and I think I will, but none of them are lacking a feature that I really need in order to accomplish something that I want to do.

I’m interested in learning more: about writing (documentation, science fiction, etc.), about software development and computing.

I’m interested in using metaphors and methods from programming and engineering to make documentation better. There are some obvious elements that are ripe for stealing in terms of process (scrum, iteration, etc.) as well as tooling (issue tracking, version control.) As I’ve continued to explore the connections and metaphors have become less obvious, but remain very helpful.

Recently I’ve been thinking about and using the idea of inheritance to help address content duplication issues. The new approach to tutorial content is one of these applications. Actually, this is a bit of retroactive intellectualizing: the need for reuse came first, and relating this back to inheritance is an idea that I want to explore.

I’m thinking about inheritance in the same way that we talk about the inheritance of classes in object oriented programs.

In the past, I’ve talked about the failure of the promise of single sourcing to solve the complexity problems in documentation as being related to the failure of object oriented programming styles to resolve code duplication and promote widespread code reuse. Or, if not related, symptoms of related causes.

For code, I think the issue is that while there are a couple of applications for inheritance (i.e. representing trees, some basic templating,) it’s not a universally applicable metaphor. The mantra composition over inheritance draws attention to the fact that “has a” relationships are more prevalent and useful than “is a” relationships.

Tactically, speaking, using inheritance rather than simple inlining or inclusion is quite helpful for thinking about content reuse in documentation. Inlining is technically easy to implement, but doesn’t actually help facilitate content reuse because it’s hard to write content for inclusion that’s sufficiently “context free,” whereas using inheritance makes it possible to reuse structures and portions of content without requiring writers to write context-free content.

Inheritance isn’t perfect of course: if you have to overload all or most inherited elements you end up with a confusing mush that’s hard to untangle, but it’s a decent starting point.

Onward and upward!

I spent some time yesterday evening dealing with a bug in some code I wrote/maintain, and I thought it would be a good exercise to just talk about the issue, and approaches problem solving. This is both an effort to demystify the act of programming and debugging and a brainstorming exercise. I explained a bunch of the background in an external page, to make the content a bit more accessible.

The System

To build a large complex documentation site we use Sphinx as the core “compiler” to translate source into outputs like HTML and PDFs, but this is really just the core operation, there’s a lot of other work: generating content, analyzing content to make sure that everything that needs to be recompiled is, and migrating content to staging and production environments.

Essentially the build happens in three-parts:

1. In the first step, we generate a bunch of content from structured sources. Then we have to assemble a dependency graph of all the content and update the time stamp (i.e. mime) all files that depend on files that have changed so that Sphinx will actually rebuild the files that need it.

   Most of this work happens in some sort of parallel execution model. I’ll rant about concurrency in Python in a bit. Hold your questions.

2. In the second step Sphinx runs. For testing purposes, this just builds one output (i.e. HTML), but our production builds need to build multiple output formats. When there are multiple builds, this runs each build in its own thread. Builds execute as sub-processes. Sphinx’s processing has a two-phase approach: there’s a read phase and a write phase. The read phase happens in one linear execution, but writing files can happen in a processing pool (if sphinx needs to write large numbers of multiple files.)

3. Finally we do a bunch of post-processing on the builds to move the files into the right places. This happens (typically) in a process pool that runs within the thread where the build executed.

See my overview of concurrency, and rant on Python concurrency for some background.

The Problem

There’s a deadlock. Sometimes, one of the Sphinx will just die, and stop doing anything, produces no messages and waits forever. (This is in step 2 above.) Everything else works fine.

Great.

The problem, or part of the problem, is that all of the parts of the system work fine in isolation. The code that starts the external process gets called extensively and works fine. The thread/process management code is also well exercised. In essence the question is:

“What could make code that works fine most of the time, fail oddly only sometimes.”

As I think about it, I’ve observed intermittent deadlocks from time to time, but they’ve gotten much worse recently:

• We weren’t doing automated building for a few weeks, for an unrelated reason and the problem shows up more frequently in automated builds.
• We moved the execution of the Sphinx process to thread pools from process pools: they were already running in yielding operations, no need to double the overhead.
• A few more operations moved to threads as well, again, mostly operations with lots of yielding, to reduce the overhead of processes and in one situation when a task was incompatible with processes (i.e. not pickelable.)

Attempts

• Make sure that the output of the sphinx build isn’t filling the output buffering that subprocess is doing. Write output to a temporary file rather than using the buffer. (see tempfile)
• Use Popen.poll() to figure out when an operation is done rather than using a blocking call which might prevent the thread from ever waking up.

Current state.

The bug is difficult to reproduce, but the above issues seems to reduce the incidence of the bug by a significant margin. My current theory is there were two deadlocks: one in sphinx itself that was pretty uncommon, and one with how we were calling sphinx that was more common and exacerbated by switching to the pool that ran sphinx from a process pool to a thread pool.

So the bug has probably been around for a while, but was much more rare given the different execution model. And regardless, was easily confused with another bug. Delightful.

Having resolved the bigger issue, it’s no longer a real problem for our automated testing, so I consider it a (minor) success. Longer term, I think we need to streamline the execution and process management, but this is not crucial now s we can continue to increment toward that solution.

What and Why

Concurrency is the term we use to think about operations that can happen at the same time. A computer program is just a list of operations that have some sort of ordering. If you run those operation in the order the programmer wrote them in, and the programmer wrote the right code, everything should work fine: If operation B depends on the outcome of operation A, then you just have to make sure that operation A happens before B.

There are two kinds of problems that this kind of model doesn’t address well:

1. Programs that respond to user or environmental interactions.

   Many programs don’t actually have a linear procedure, and depend on user input (e.g. a word processor or text editor must wait for you to type text).

   Let’s call these “event driven” programs.

2. Programs may consist of operations that don’t depend on each other.

   Consider a program that has operations A, B, C, and D. Sometimes, you may have to run D after C and C after B, and B after A; but sometimes B, C, and D are totally (or mostly) independent of each other and can run in any order.

   Let’s call these “potentially parallel workloads.”

While both of these kinds of models help us to think about ways that software can reflect and respond to operations happening at the same time, concurrency is more subtle than “parallelism.” Concurrency is about modeling the dependencies and relationships between operations, parallelism is really just an implementation detail.

Parallelism requires concurrency; but you can execute concurrent designs in parallel or not as needed or desired.

There are caveats both during development and at runtime:

• Concurrency makes some aspects of programming harder because if the order and timing of operations changes, the possibilities for conflicts and errors grows. It also makes it harder to follow the (possible) chain of operations.
• At runtime, if you use parallelism to execute a concurrent program, there’s an amount of overhead spent managing the more complex execution model.

Reference: <http://blog.golang.org/concurrency-is-not-parallelism>

Implementation Details (Python)

A single Python process does not support parallel execution: only one operation can execute at a time. Sort of. The rule is more complex:

• Python operations in Python code.

  This covers all Python code you write and depend on, and much of the standard library.

• (some) Python operations implemented in C.

  This covers some operations like computing hashes and reading and writing to files or network connections.

• Operations that run external processes. (e.g. “run this shell command”)

The second two operations can run in parallel with each other and with execution of Python code (i.e. these operations are “yielding”). These yielding operations typically account for the operations that take the most amount of time. The downside is that yielding operations account for a small percentage of the number of operations in a Python program.

Python provides two (native) parallelism metaphors: threads and processes.

• Threads are lightweight, have low start-up costs, and have access to the shared state of the master process. The downsides is that only yielding operations can actually run in parallel. Otherwise only one Python operation can run at once. Except that operations will interleave at some level, which means you can get some kinds of concurrency bugs (deadlocks/races) even though there’s limited parallel operation.
• Processes are less lightweight, have slightly higher start-up costs, but can all execute Python code at the same time. They also don’t have access to shared state, which means there are more costs associated with copying memory to-and-from the process. While there are more limitations on what can run in processes, because there’s isolation and no shared state, its more safe.

The best part is that the interfaces for working with threads and processes are the same, which makes testing easier.

The Rant

The problem isn’t that Python doesn’t have concurrency tools, it’s that no one started writing Python with the idea that parallelism and concurrency would be a defining element of most systems that people would need or want to write.

The result is that while it’s theoretically possible to modify Python itself to be more concurrent, one of the two things happen:

• Everything breaks. There’s a lot of Python that depends on the current behavior and concurrency semantics. The Python Standard Library is big. The ecosystem of software written in Python is even bigger and everything would break.
• In order to prevent everything from breaking, to make the changes required to support more intrinsic parallelism you actually end up slowing-down the arguably more common non-parallel operation.

The work on this is ongoing, of course, and eventually I suspect there will be some solution, but the change is unlikely to be revolutionary. In the mean time, it’s awkward and sometimes awful:

• You can write concurrent code, which is nice, but there is some awkwardness around these expression: calling lots of functions inside of [multiprocessing.Pool.apply_async()]{.title-ref} (or something similar) is pain; callbacks and passing passing function pointers around is awkward and prone to error.
• Because so little of the Python tooling expects thing to be running in parallel, there are huge warts: error handling blows; the documentation doesn’t really cover what yields or doesn’t, and what can or will block.
• In some situations, you can get pretty good parallel performance. This feels great, but often doesn’t feel predictable or reproducible.

What would make this better?

• There should be standard ways to express concurrency that feels less like a hack. This is a syntax/library deficiency.
• Errors in processes should bubble up more forcefully.
• Documentation of Python APIs should affirmatively describe the concurrency semantics of all operations.

a checklist

There’s a difference between software that works, software that’s brilliant, software that’s maintainable, and software that’s good. This is post that begins to enumerate the kinds of things that you can do as you write software to help make it sane and possibly good.

This isn’t about computer science, or really even about engineering principals. We all have a sense of what makes a physical object (furniture, buildings, electronics) feel like they are well made. This is about making software have the same feel, and what you can think about when you’re writing code to help give the finished product that feel.

I’ll expand on these items later, but as an outline.

1. Have logging everywhere.
2. Solid test code that mirrors app functionality.

If you feel like you can safely make a change to the code and be confident that your tests will catch regressions, then you’re good, otherwise; write more tests.

3. Never more hierarchy then you need.
4. Make the building infrastructure really robust.

5. Have internal abstractions for internal configuration and persistence.

Configuration and data persistence should be encapsulated by some internal interface and the application logic shouldn’t depend on the implementations of how the application is configured or how you persist the data.

6. Consider concurrency when possible.
7. Don’t confuse exceptions and conditions.
8. Break long sequences of sections into groups of logical operations.
9. Avoid unnecessarily tangled execution paths.

Please discuss!

At work I’ve introduced a new method for storing the content for procedural documents (i.e. tutorials/how-to guides/etc,) that is more structured. Rather than just writing a tutorial in a conventional way, we capture the steps in a specific structure that we use to generate the content in a specific way. The structure lets us do cool things with the presentation that would be hard to do well otherwise and help us focus theses kinds of documents on the core procedural or sequence-based nature of these documents.

Doing documentation well is really hard, but probably not for the reasons you think: Writing clear text about complex ideas isn’t easy, but it is straight forward. Figuring out what needs to be documented is harder, but it boils down to a business problem and if you talk to the right people it’s not a significant challenge. Ensuring that the documentation remains correct over time as the product develops, particularly with regards to duplicated content: really hard. Managing complexity of large texts that are always growing and changing, particularly with regards to correctness and content duplication: really fucking hard.

The problem is that the solutions to the hard problems are at odds with each other: you help address the complexity problems by decoupling the organization of the source material from the presentation, which never works well enough. You solve the duplication problem using a “single sourcing strategy” where you store common bits of information in one place and then inject that text as needed, which increases complexity.

There is no winning.

The new structured approach to procedures may not be winning, exactly, but it works pretty well. Essentially we put a huge core of our content into a YAML document and then render it out, but we get some nice benefits:

• For each step in a sequence we can inherit from any other step in the project and override any component of that step. This facilitates reuse, and forces us to think about potential reuse throughout the process.
• The compiler and parsers tell us when we get something wrong.
• The output is very regular, so we can be confident that all of the tutorials look the same and have the same structure.
• Minimal loss of editing clarity: YAML is great to edit, the structure is like a JSON document, but commentsare allowed and the syntax does smart things with newlines and requires less escaping.

Basically, at its core, we’re restricting the available structural possibles for some documents and using (simple) software to enforce those requirements. The results are quite good, but in some ways it makes the writing part a bit more difficult. There’s less room for imprecision, and there are some weak rhetorical formulations that become more important to avoid. For example: complex conditional structures don’t always work well and positional references (i.e. “as above”) are almost always unhelpful.

In some ways the effect is kind of like writing in a specific poetic form. Less metric, but the same kind of toying with squeezing an idea into a very specific form. The coolest side effect is that given the constraints that the new system imposes, the quality of everyone’s output has improved.

I can live with that!

In response to my Knitting Practices post, on Facebook my father commented “The word “practice” is apt. Is there an influence from yoga?”

The answer is obviously “yes,” though the route is somewhat indirect and travels through a story about programming. Stick with it for a little while.

Generators and Python Memory Efficiency

I was talking with a dancing friend about memory efficiency in Python programs, particularly with regards to loops and the range() function. Say you want to do something 1000 times, in Python the easiest way to do this is:

for it in range(1000):
   do_thing(it)

This says, “make a list of numbers from 1 to 1000,” and then call the do_thing() operation on that number (i.e. assigned to the variable it). range(1000) evaluates to a list, which Python stores in memory and the code above loops over.

The way to get this effect in other languages (JavaScript below) is to do something like this:

for (var it=0; it<= 1000; it++) {
   do_thing(it)
}

The for loop has three statements: an action to perform running the content of the loop (create a variable), a condition that will terminate the loop by returning false (variable is less than or equal to 1000), and an operation to run after each loop. The effect of the JavaScript is the same as the Python, except that the Python has to build this (potentially large) list for grins.

There’s a couple of quick answers to this specific question:

1. (In Python 2) use xrange() which is a special iterable type (i.e. it works with loops), that doesn’t need to build a list in memory, it just spits out values incrementally as needed.

   Let’s imagine that it does this by having a function that returns values starting at a certain point (i.e. 0) pausing after each value, and then returning the next value the next time it runs. Thes ea re called “generators” in Python.

2. Wait for the Python 3 switch to complete: In Python 3, range() is a generator of sorts, and it’s efficient in this way.

3. Not care about memory use so much. In most cases this will not be the bottle neck in your application. Really long lists of integers may take up megabytes of memory, but that’s not a huge deal.

Generators are great, and they’re worth using in most cases, but no one will laugh at you if you don’t use them in this situation.

The coolest thing, really is that you can really easily write generator functions. Here’s a silly example: :

def animals("input_animal"=None):
   for animal in [ 'cat', 'dog', 'cow']:
      if input_animal != animal:
          yield animal

This function returns an animal, as long as the input_animal isn’t in the list.

I explained how generators work to my friend and he said something like “Nifty, I guess they’re not really part of my Python practice yet.”

The phrase sort of stuck with me.

Programming is a Practice

There’s a bunch of theory to programming that’s grounded in how computers work, and a lot of things that are actually useful for programmers to know but in truth programming is a practice. Being able to look at a design or a program and understand how to write code to achieve some goal or connect two pieces of functionality, is really about the practice

Beyond understanding what makes good software good and how the machines work, most of programming is knowing the tools really wall and being able to identify the right tool for the job. Everything else is just typing and flow.

While the fundamental knowledge and knowledge of the tools is always important, at some point programming becomes more about being to figure out what the right solution is for any given obstacle or situation.

Knitting as Practice

Knitting is kind of the same. There are some fundamental skills (stitches, operations) and there are some basic fundamental modalities/patterns¹ that you use and combine to make some common objects.

Knitting itself is always about repetition, stitch after stitch, but it’s also about repetition on a higher level. Knitting the second sock, the second sleeve. Making pair of sock after pair of sock, or sweater after sweater.

At some point being a knitter stops being about the skills and modalities/patterns and starts being about actually making things and figuring out how to apply what you know about knitting to a new situation: the thing you want to make, the yarn your using, and the tools you’re using.

It’s all practice. All the way down.