Learning how to make computer software is hard. Not fundamentally hard: lots of people can do it, and even more people do things that are functionally equivalent to programming though they wouldn't think of it as such. But teaching people how to write good computer software is a challenge, and one that I'm generally interested in exploring more.
For a long time, I've been interested in this problem from the outside: I didn't really know how to program in any meaningful sort of way and I was interested in deconstructing the process of making software. Then something clicked and years of tinkering with systems administration and reading about programming languages and practices clicked and while I think I have a lot left to learn, I've started thinking about the problem from the other side.
We accumulate many skills and kinds of knowledge in an incremental sort of way: you study and practice and little by little our brains (and bodies) form new connections and we "learn." Other kinds of learning follow a more "step-based" approach: we practice and study for a long period of time without much discernible change in understanding or skill until at some point we experience some sort of larger improvement in ability.
At least for me (and perhaps you as well,) things like dance, knitting, writing, and most structured/classroom-based topics tend to be incremental, mostly. Other things, like programming (at least initially) and singing/music tend to be step-based.
This isn't to say that step-based areas of focus don't require regular ongoing practice, just that the observable markers of progress may lag inconsistently behind effort and pedagogy.
When I was doing more non-professional writing, I was fond of the school of writing advice that said "the way to learn how to write (fiction) is to have a good story to tell;" when I think about learning to program I think the first step has to be a need to automate something on a computer.
I've even started a wiki page on the topic.
Hackers describe this as the "scratch your own itch" method (from CatB and elsewhere.)
Neither the idea of step-based versus incremental learning nor the notion of using a personal need to drive learning are new, but I think they illuminate eachother well.
Since I stopped being a student somewhat abruptly in 2007, I've become increasingly glad both that my education and personal development has continued and I've had the opportunity to explore things in ways that didn't make sense in a structured context (e.g. "I want to learn about how databases work without formal CS/systems training," or "I want to learn how to sing and withouta lot of music theory.")
For most of the past year I've been pretty heads-down on the "learning to program" project, and I've had a number of interesting problems that I've used to help explore the topic:
generalized Sphinx publishing toolkit.
Sphinx is a great tool for producing text, and I'm quite fond of it. At the same time, I'm not a fan of its architecture (and have a number of approaches to optimize the build process,) and there are a number of tasks: dependency resolution, version management, theme management, and deployment that any reasonably complex Sphinx-based project needs to address.
While this project requires ongoing development and improvement, it's basically feature complete, and it's given
I've had a couple of personal side projects that I've used to explore different kinds of programming problems, with greater and lesser success.
Buildcloth, which is pretty cool and needs more work but I fear may be too complicated for the use-case.
csc, which isn't fully off the ground and may not provide a significant improvement upon Sphinx for most cases.
dtf, which is a decent idea, but I've not had time to really implement and exercise the program and I fear that the core code quality isn't great.
More recently I've begun working on a dependency analytics package with a co-worker/friend to help him and his teammates understand and untangle a larger C++ project.
It's been nice to be able to actively work on a project with another developer, and to be able to focus on performance and architecture issues while someone else focuses on feature prototyping and use-cases.
In a lot of ways this is a good "capstone" project for me because I've gotten to use and apply many of the things I've learned from writing concurrent/parallel Python, as well as moderate sized Python programs comes together well here.
Most of the projects that had been open and on my plate for the last few months have mostly wrapped up. There's more work to be done on them, I could do a lot more work, and I think I will, but none of them are lacking a feature that I really need in order to accomplish something that I want to do.
I'm interested in learning more: about writing (documentation, science fiction, etc.), about software development and computing.
I'm interested in using metaphors and methods from programming and engineering to make documentation better. There are some obvious elements that are ripe for stealing in terms of process (scrum, iteration, etc.) as well as tooling (issue tracking, version control.) As I've continued to explore the connections and metaphors have become less obvious, but remain very helpful.
Recently I've been thinking about and using the idea of inheritance to help address content duplication issues. The new approach tocontent is one of these applications. Actually, this is a bit of retroactive intellectualizing: the need for reuse came first, and relating this back to inheritance is an idea that I want to explore.
I'm thinking about inheritance in the same way that we talk about the inheritance of classes in object oriented programs.
In the past, I've talked about the failure of the promise of single sourcing to solve the complexity problems in documentation as being related to the failure of object oriented programming styles to resolve code duplication and promote widespread code reuse. Or, if not related, symptoms of related causes.
For code, I think the issue is that while there are a couple of applications for inheritance (i.e. representing trees, some basic templating,) it's not a universally applicable metaphor. The mantra composition over inheritance draws attention to the fact that "has a" relationships are more prevalent and useful than "is a" relationships.
Tactically, speaking, using inheritance rather than simple inlining or inclusion is quite helpful for thinking about content reuse in documentation. Inlining is technically easy to implement, but doesn't actually help facilitate content reuse because it's hard to write content for inclusion that's sufficiently "context free," whereas using inheritance makes it possible to reuse structures and portions of content without requiring writers to write context-free content.
Inheritance isn't perfect of course: if you have to overload all or most inherited elements you end up with a confusing mush that's hard to untangle, but it's a decent starting point.
Onward and upward!
I spent some time yesterday evening dealing with a bug in some code I wrote/maintain, and I thought it would be a good exercise to just talk about the issue, and approaches problem solving. This is both an effort to demystify the act of programming and debugging and a brainstorming exercise. I explained a bunch of the background in an external page, to make the content a bit more accessible.
To build a large complex documentation site we use Sphinx as the core "compiler" to translate source into outputs like HTML and PDFs, but this is really just the core operation, there's a lot of other work: generating content, analyzing content to make sure that everything that needs to be recompiled is, and migrating content to staging and production environments.
Essentially the build happens in three-parts:
In the first step, we generate a bunch of content from structured
sources. Then we have to assemble a dependency graph of all the
content and update the time stamp (i.e.
mime) all files that
depend on files that have changed so that Sphinx will actually
rebuild the files that need it.
Most of this work happens in some sort of parallel execution model. I'll rant about concurrency in Python in a bit. Hold your questions.
In the second step Sphinx runs. For testing purposes, this just builds one output (i.e. HTML), but our production builds need to build multiple output formats. When there are multiple builds, this runs each build in its own thread. Builds execute as sub-processes. Sphinx's processing has a two-phase approach: there's a read phase and a write phase. The read phase happens in one linear execution, but writing files can happen in a processing pool (if sphinx needs to write large numbers of multiple files.)
Finally we do a bunch of post-processing on the builds to move the files into the right places. This happens (typically) in a process pool that runs within the thread where the build executed.
See my overview of concurrency, and rant on Python concurrency for some background.
There's a deadlock. Sometimes, one of the Sphinx will just die, and stop doing anything, produces no messages and waits forever. (This is in step 2 above.) Everything else works fine.
The problem, or part of the problem, is that all of the parts of the system work fine in isolation. The code that starts the external process gets called extensively and works fine. The thread/process management code is also well exercised. In essence the question is:
"What could make code that works fine most of the time, fail oddly only sometimes."
As I think about it, I've observed intermittent deadlocks from time to time, but they've gotten much worse recently:
We weren't doing automated building for a few weeks, for an unrelated reason and the problem shows up more frequently in automated builds.
We moved the execution of the Sphinx process to thread pools from process pools: they were already running in yielding operations, no need to double the overhead.
A few more operations moved to threads as well, again, mostly operations with lots of yielding, to reduce the overhead of processes and in one situation when a task was incompatible with processes (i.e. not pickelable.)
Make sure that the output of the sphinx build isn't filling the output buffering that subprocess is doing. Write output to a temporary file rather than using the buffer. (see tempfile)
Use Popen.poll() to figure out when an operation is done rather than using a blocking call which might prevent the thread from ever waking up.
The bug is difficult to reproduce, but the above issues seems to reduce the incidence of the bug by a significant margin. My current theory is there were two deadlocks: one in sphinx itself that was pretty uncommon, and one with how we were calling sphinx that was more common and exacerbated by switching to the pool that ran sphinx from a process pool to a thread pool.
So the bug has probably been around for a while, but was much more rare given the different execution model. And regardless, was easily confused with another bug. Delightful.
Having resolved the bigger issue, it's no longer a real problem for our automated testing, so I consider it a (minor) success. Longer term, I think we need to streamline the execution and process management, but this is not crucial now s we can continue to increment toward that solution.
There's a difference between software that works, software that's brilliant, software that's maintainable, and software that's good. This is post that begins to enumerate the kinds of things that you can do as you write software to help make it sane and possibly good.
This isn't about computer science, or really even about engineering principals. We all have a sense of what makes a physical object (furniture, buildings, electronics) feel like they are well made. This is about making software have the same feel, and what you can think about when you're writing code to help give the finished product that feel.
I'll expand on these items later, but as an outline.
Have logging everywhere.
Solid test code that mirrors app functionality.
If you feel like you can safely make a change to the code and be confident that your tests will catch regressions, then you're good, otherwise; write more tests.
Never more hierarchy then you need.
Make the building infrastructure really robust.
Have internal abstractions for internal configuration and persistence.
Configuration and data persistence should be encapsulated by some internal interface and the application logic shouldn't depend on the implementations of how the application is configured or how you persist the data.
Consider concurrency when possible.
Don't confuse exceptions and conditions.
Break long sequences of sections into groups of logical operations.
Avoid unnecessarily tangled execution paths.
See the rhizome archive for more posts.