I wrote some notes about to write a post about a software project I worked on a year and a half ago, that I think is pretty cool, but I was on writing hiatus. Even better the specific code in question is now no longer in use. But I think it serves as a useful parable, but I will attempt to reflect.

Go’s logging¹ support in standard library works, and it successfully achieves its goals on its own terms. The problem is that it’s incredibly simple and lacks a number of features that are standard in most logging systems.² So as a result, I’m not surprised that most applications of consequence either use a couple of more fully featured logging packages or end up writing a large number of logging wrappers.

The fact that my project at work was using a special logging library is not particularly surprising, particularly because the project is old for a Go project. The logging library in question is a log4j-inspired package, that had been developed by a different group internally, but was no longer being used by that group. It worked, but there were a host of problems.³

I’d also written a logging package myself which was a definite improvement on the state of the art. I had two chief problems:

• how to convince teammates to make the change,
• how to make the change without disrupting ongoing work or the functioning of the system which had to be always deploy-able.

Here’s what I did…

First, I learned as much as I could about the existing system, it’s history and how we used it. I read a lot of code, documentation (such as it was,) and also related bug reports, feature requests, and history.

Second, I implemented wrappers for my system that (mostly) cloned the interfaces for the existing library in my own package. It’s called slogger, and it’s still there, though I hope to delete it soon. I wanted to make it possible to make the switch⁴ in the project initialization without needing to change every last logging statement.⁵

Then, we actually made the change so that logging used the new code internally but wrapped by the old interfaces. I think there were a couple of very obvious bugs early on, but frankly none of them are so memorable that I could describe them any more.

Finally, we went through and updated all of the logging statements. It was a big change, and impacted all of the code, but it happened quite late in the process and there were no bugs, because it was the least interesting or radical part of the project.

And then we had a new logger. It’s been great. With the new tool we’ve been able to easily add support for more structured approaches to logging and collecting log output in a variety of third party services.

In summary:

• replacing legacy subsystems can be a good way to improve the functionality of your project.
• change is hard, but there are ways to make changes easier and less disruptive. They often involve doing even more work.*
• write code to facilitate transitions, and then delete it later.
• the larger a change is, the less risky it should be. While there are lots of small-and-low risk changes you can make, the inverse should be true as rarely as possible.

I wrote a post about moving on from being a technical writer, and I’ve definitely written some since then about programming and various side projects, but I haven’t really done the kind of public reflection on this topic that I’ve done historically about, many other things.

When I switched to a programming team, I knew some things about computers, and I was a decent Python programmer. The goal, then was to teach myself a second programming language (Go,) and learn how to make “real” software with other people, or on teams with other people. Both of those projects are going well: I think I’ve become pretty solid as a Go programmer, although, it’s hard to say what “real” software is, or if I’m good at making it, but all indications are positive.

This weekend, for various reasons, I’ve been reviving a project that I did some work on this fall and winter, that I’ve abandoned for about 6 months. It’s been both troubling (there are parts that are truly terrible,) and kind of rewarding to see how much I’ve grown as a programmer just from looking at the code.

Queue then, I guess, the self reflective interlude.

My reason for wanting to learn--really learn--a second programming language, was to make sure that all the things I knew about system design, algorithms, and data structures was generalizable, and not rooted in the semantics of a specific language or even implementation of that language. I was also interested in learning more about the process of learning new programming languages so that I had some experience with the learning process, which may come in handy in the future.

Learning Go, I think helped me achieve or realize these goals. While I haven’t really set out to learn a third language yet, it feels tractable. I’ve also noticed some changes and differences in some other aspects of my interests.

I used to be really interested in programming qua programming, and I thought a lot about programming languages. While I still can evaluate programming languages, and have my own share of opinions about “the way things work,” I’m less concerned with the specific syntax or implementation. I think a lot about build tools, platform support, deployment models, and distributing methods and stories, rather than what it can do or how you have to write it. Or, how you make it ship it and run it.

I’ve also gotten less interested in UNIX-esque systems administration and operations, which is historically a thing I’ve been quite interested in. These days, I find myself thinking more about the following kinds of problems:

• build systems, the tools building software from source files, (and sometimes testing it!) and the ways to do this super efficiently and sensibly. Build systems are quite hard because in a lot of ways they’re the point through which your software (as software) interacts with all of the platforms it runs on. Efficient build systems have a huge impact on developer productivity, which is a big interest.
• developer productivity, this is a big catch all category, but it’s almost always true that people are more expensive than computers, so working on tools and features (like better build systems, or automating various aspects of the development process,)
• continuous integration and deployment, again connected to developer productivity, but taking the “automate building and testing,” story to its logical conclusion. CD environments mean you deploy changes much more often, but you also require and force yourself to trust the automated systems and make sure that project leadership and management is just as automated as the development experience.
• internal infrastructure, as in “internal services and tools that all applications need,” like logging, queuing systems, abstractions for persistence, deployment systems, testing, and exposed interfaces (e.g. RPC systems, REST/HTTP, or command line option option parsing). Having good tools for these generic aspects of the application make writing actual features for users easier. I’m also increasingly convinced that the way to improve applications and systems is to improve these lower level components and their interfaces.

Free Software and open source are still important, as is UNIX, but these kinds of developer productivity and automation issues are a level above that. I’ve changed in the last 5 years, software has changed in the last five years, the way we run software on systems has changed in the last 5 years. I’m super excited to see what kinds of things I can do in this space, and where I end up in 5 years.

I’m also interested in thinking about ways to write about this. I’d written drafts of a number of posts that were about learning how to program, about systems administration, and now that I’m finding and making more time for writing, one of the things I don’t really know about is what kind of writing on these topics I’m interested in doing, or how to do it in a way that anyone would be interested in reading.

We shall see. Regardless, I hope that I’m back, now.

This post is a follow up to the interlude in the /posts/programming-tutorials post, which part of an ongoing series of posts on programmer training and related issues in technological literacy and education.

In short, creating novel automations is hard. The process would have to look something like:

1. Realize that you have an unfulfilled software need.
2. Decide what the proper solution to that need is. Make sure the solution is sufficiently flexible to be able to support all required complexity.
3. Then sit down, open an empty buffer and begin writing code.

Not easy.¹

Something I’ve learned in the past few years is that the above process is relatively uncommon for actual working programmers: most of the time you’re adding a few lines here and there, testing various changes or adding small features built upon other existing systems and features.

If this is how programming work is actually done, then the kinds of methods we use to teach programmers how to program should hold some resemblance to the actual work that programmers do. As an attempt at a case study, my own recent experience:

I’ve been playing with Buildbot for a few weeks now for personal curiosity, and it may be useful to automate some stuff for the Cyborg Institute. Buildbot has its merits and frustrations, but this post isn’t really about buildbot. Rather, the experience of doing buildbot work has taught me something about programming and about “building things,” including:

• When you set up buildbot, it generates a python configuration file where all buildbot configuration and “programming” goes.

  As a bit of a sidebar, I’ve been using a base configuration derived from the buildbot configuration for buildbot itself, and the fact that the default configuration is less clean and a big and I’d assumed that I was configuring a buildbot in the “normal way.”

  Turns out I haven’t, and this hurts my (larger) argument slightly.

  I like the idea of having a very programmatic interface for systems that must integrate with other components, and I really like the idea of a system that produces a good starting template. I’m not sure what this does for overall maintainability in the long term, but it makes getting started and using the software in a meaningful way, much more possible.

• Using organizing my buildbot configuration as I have, modeled on the “metabuildbot,” has nicely illustrated the idea software is just a collection of modules that interact with each other in a defined way. Nothing more, nothing less.

• Distributed systems are incredibly difficult to get people to conceptualize properly, for anyone, and I think most of the frustration with buildbot stems from this.

• Buildbot provides an immediate object lesson on the trade-offs between simplicity and terseness on the one hand and maintainability and complexity on the other.

  This point relates to the previous one. Because distributed systems are hard, it’s easy to configure something that’s too complex and that isn’t what you want at all in your Buildbot before you realize that what you actually need is something else entirely.

  This doesn’t mean that there aren’t nightmarish Buildbot configs, and there are, but the lesson is quite valuable.

• There’s something interesting and instructive in the way that Buildbot’s user experience lies somewhere between “an application,” that you install and use, and a program that you write using a toolkit.

  It’s clearly not exactly either, and both at the same time.

I suspect some web-programming systems may be similar, but I have relatively little experience with systems like these. And frankly, I have little need for these kinds of systems in any of my current projects.

Thoughts?

This post is a follow up to my :doc`/posts/coding-pedagogy` post. This “series,” addresses how people learn how to program, the state of the technical materials that support this education process, and the role of programming in technology development.

I’ve wanted to learn how to program for a while and I’ve been perpetually frustrated by pretty much every lesson or document I’ve ever encountered in this search. This is hyperbolic, but it’s pretty close to the truth. Teaching people how to program is hard and the materials are either written by people who:

• don’t really remember how they learned to program.

Many programming tutorials were written by these kinds of programmers, and the resulting materials tend to be decent in and of themselves, but they fail to actually teach people how to program if they don’t know how to program already.

If you already know how to program, or have learned to program in a few different languages, it’s easy so substitute “learning how to program,” with “learn how to program in a new language” because that experience is more fresh, and easier to understand.

These kinds of materials will teach the novice programmer a lot about programming languages and fundamental computer science topics, but not anything that you really need to learn how to write code.

• people who don’t really know how to program.

People who don’t know how to program tend to assume that you can teach by example, using guided tutorials. You can’t really. Examples are good for demonstrating syntax and procedure, and answering tactical questions, but aren’t sufficient for teaching the required higher order problem solving skills. Focusing on the concrete aspects of programming syntax, the standard library, and the process for executing code isn’t enough.

These kinds of documents can be very instructive, and outsider perspective are quite useful, but if the document can’t convey how to solve real problems with code, you’ll be hard pressed to learn how to write useful programs from these guides.

In essence, we have a chicken and egg problem.

Interlude:

Even six months ago, when people asked me “are you a programmer?” (or engineer,) I’d often object strenuously. Now, I wave my hand back and forth and say “sorta, I program a bit, but I’m the technical writer.” I don’t write code on a daily basis and I’m not very nimble at starting to write programs from scratch, but sometimes when the need arises, I know enough to write code that works, to figure out the best solution to fix at least some of the problems I run into.

I still ask other people to write programs or fix problems I’m having, but it’s usually more because I don’t have time to figure out an existing system that I know they’re familiar with and less because I’m incapable of making the change myself.

Even despite these advances, I still find it hard to sit down with a blank buffer and write code from scratch, even if I have a pretty clear idea of what it needs to do. Increasingly, I’ve begun to believe that this is the case for most people who write code, even very skilled engineers.

This will be the subject of an upcoming post.

The solution(s):

1. Teach people how to code by forcing people to debug programs and make trivial modifications to code.

People pick up syntax pretty easily, but struggle more with the problem solving aspects of code. While there are some subtle aspects of syntax, the compiler or interpreter does enough to teach people syntax. The larger challenge is getting people to understand the relationship between their changes and behavior and any single change and the reset of a piece of code.

2. Teach people how to program by getting them to solve actual problems using actual tools, libraries, and packages.

Too often, programming tutorials and examples attempt to be self-contained or unrealistically simple. While this makes sense from a number of perspectives (easier to create, easier to explain, fewer dependency problems for users,) it’s incredibly uncommon and probably leads to people thinking that a lot of programming revolves around re-implementing solutions to solved problems.

I’m not making a real argument about computer science education, or formal engineering training, with which I have very little experience or interest. As contemporary, technically literate, actors in digital systems, programming is a relevant for most people.

I’m convinced that many people do a great deal of work that is effectively programming: manipulating tools, identifying and recording procedures, collecting information about the environment, performing analysis, and taking action based on collected data. Editing macros, mail filtering systems, and spreadsheets are obvious examples though there are others.

Would teaching these people how programming worked and how they could use programming tools improve their digital existences? Possibly.

Would general productivity improve if more people new how to think about automation and were able to do some of their own programming? Almost certainly.

Would having more casual programmers create additional problems and challenges in technology? Yes. These would be interesting problems to solve as well.

There are two parts to this post: first, the relationship or non-relationship between the ability to write code and technical literacy; and second, the pedagogical methods for teaching people how to program/code.

In some ways, I’ve been writing about this and related topics for quite a while: see /posts/objective-whatsis for an earlier iteration in this train of thought.

Programming and Technical Literacy

Programmers and other technical folks talk a lot about teaching young people to code as the central part of any young technical person’s education and basic computer literacy. Often this grows out of nostalgia for their own experience learning to program, but there are other factors at play.¹

In some cases, they even start or point to projects like Codecademy. Which are, in truth, really cool ideas, but I think that effectively equating the ability to write code with technical literacy is fraught:

• There are many different kinds of technical literacy and writing code is really such a small part. Sure code gives us a reasonable way to talk about things like design and architecture, but actually writing code is such a small part of developing technology.

• Writing code isn’t that important, really. In a lot of ways, code is just an implementation detail. Important as a way of describing some concepts pretty quickly, important because it’s impossible to iterate on ideas without something concrete to point to, but the implementation isn’t nearly as important as the behavior or the interface.

• For the last ~40 years, code has been the way that people design behavior and specify interfaces for software. While there are a lot of reasons why this predominantly takes the form of code, there’s not particular reason that we can’t express logic and describe interfaces using other modalities.

  There are many people who are very technically literate and productive who don’t write code, and I think that defining literacy as being able to write code, is somewhat short sighted. Also, there is another group of people who are actually programmers who don’t think of the things they do as “programming,” like people who do crazy things with spreadsheets, most librarians, among others. These non-coding programmers may shy away from programming or are mostly interested in the output of the program they write and less interested in the programming itself.

This is a huge problem. I hope that this /posts/computer-literacy-project that I’ve been planning will start to address some of these issues, but there’s even more work to do.

How to Teach People to Code

(This section of the post derives from and summaries the “How to Teach People to Program” wiki page.)

Most of the way that programming books and courses teach programming are frustrating and somewhat dire, for a few reasons:

• Most examples in programming books are dumb.
• Basic computer science/engineering knowledge is fundamental to the way that accomplished programmers think about programming but aren’t always required to teach people how to program.
• Syntax isn’t that important, but you can’t ignore it either.
• Slow reveals are really frustrating.
• The kinds of code that you write when learning to programming bear little resemblance to the actual work that programmers do.

The solutions to these problems are complex and there are many possible solutions. As a starting point:

• Separate the way you present core concepts (i.e. data structures, typing, functions, classes, etc.) from actual code examples and from actual explanations of the syntax.

  Interlink/cross reference everything, but if you give people the tools to answer their own questions they’ll learn what they actually need to know, and you can then do a better job of explaining the syntax, basic concepts, and practical examples.

• Provide longer examples that aren’t contrived.

  Examples don’t need to start from first principals, and don’t need to be entirely self contained. Programming work rarely starts from first principals (relative,) and is rarely actually self contained. It’s foolish, then to use these sorts of pedagogical tools.

Thoughts?

In no particular order:

Org Mode Guilt and a Lisp Function

I have some guilt about having mostly forsaken org-mode,¹ in particular because I was watching Sacha Chua’s chat with John Wiegley, and I think both are such nifty hackers, and have done so many things that are pretty darn nifty.

I liked what I heard about johnw’s org mode setup so much that I might give it a try again. But in the mean time, I wanted to make my “recompile my tasklist function” to be a bit more clean. The result is follows:

(defun tychoish-todo-compile ()
   (interactive)
   (if (get-buffer "*todo-compile*")
       (progn
          (switch-to-buffer-other-window (get-buffer "*todo-compile*"))
          (recompile))
       (progn
          (compile "make -j -k -C ~/wiki")
          (switch-to-buffer-other-window "*compilation*")
          (rename-buffer "*todo-compile*")))
       (revbufs))

Notables:

• This is the first time I’ve used progn which is somewhat embarrassing, but it’s a great thing to have in the toolkit now. Link: progn
• I hadn’t realized until now that there wasn’t an else-if form in emacs lisp. Weird, but it makes sense.
• Compilation Mode is pretty much my current favorite thing in emacs.
• revbufs is this amazing thing that reverts buffers if there aren’t local modifications, and also reports to you if a buffer has changed outside of emacs and there are local modifications. So basically “does everything you want without destroying anything and then tells you what you need to do manually.” Smart. Simple. Perfect.

I might need to “macro-ize” this, as I have a lot of little compile processes for which I’d like to be able to trigger/maintain unique compile buffers. That’s a project for another day.

Emacs Thoughts

I’m even thinking about putting together a post about how, although I’m a diehard emacs user, and I’ve spent a fair bit of time learning how to do really great things with emacs, there are a lot of vim-ish things in my workflow:

• I read email with mutt and I’ve tried to get into GNUS, and I try it again every now and then, but I always find it so unbelievably gnarly. At least the transition. Same with Notmuch, which I like a lot more (in theory,) but I find the fact that Notmuch and mutt have this fundamental misunderstanding about what constitutes a “read” email, to be tragic.

• I use a crazy ikiwiki + deft + makefile setup for task tracking. As (obliquely) referenced above.

  I might give org another shot, and I’ve been looking at task warrior, but the sad truth is that what I have works incredibly well for in most cases, and switching is hard.

• I tend jump to a shell window to do version control and other things, even though I’m familiar with magit and dired, my use of these tools is somewhat spotty.

This is an entry in my loose series of posts about build systems.

I’ve been thinking recently about why I’ve come to think that build systems are so important, and this post is mostly just me thinking aloud about this issue and related questions.

Making Builds Efficient

Writing a build systems for a project is often relatively trivial, once you capture the process, and figure out the base dependencies, you can write scripts and make files to automate this process. The problem is that the most rudimentary build systems aren’t terribly efficient, for two main reasons:

1. It’s difficult to stumble into a build process that is easy to parallelize, so these rudimentary solutions often depend on a series of step happening in a specific order.

2. It’s easier to write a build system that rebuilds too much rather than too little for subsequent builds. From the perspective of build tool designers, this is the correct behavior; but it means that it takes more work to ensure that you only rebuild what you need to.

As a corollary, you need to test build systems and approaches with significantly large systems, where “rebuilding too much,” can be detectable.

Making a build system efficient isn’t too hard, but it does require some amount of testing and experimentation, and often it centers on having explicit dependencies, so that the build tool (i.e. Make, SCons, Ninja, etc.) can build output files in the correct order and only build when a dependency changes.¹

The Benefits of a Fast Build

1. Fast builds increase overall personal productivity.

   You don’t have to wait for a build to complete, and you’re not tempted to context switch during the build, so you stay focused on your work.

2. Fast builds increase quality.

   If your build system (and to a similar extent, your test system,) run efficiently, it’s possible to detect errors earlier in the development process, which will prevent errors and defects. A tighter feedback loop on the code you write is helpful.

3. Fast builds democratize the development process.

   If builds are easy to run, and require minimal cajoling and intervention, it becomes much more likely that many people

   This is obviously most prevalent in open source communities and projects, this is probably true of all development teams.

4. Fast builds promote freshness.

   If the build process is frustrating, then anyone who might run the build will avoid it and run the build less frequently, and on the whole the development effort looses important feedback and data.

   Continuous integration systems help with this, but they require significant resources, are clumsy solutions, and above all, CI attempts to solve a slightly different problem.

Optimizing Builds

Steps you can take to optimizing builds:

(Note: I’m by no means an expert in this, so feel free to add or edit these suggestions.)

• A large number of smaller jobs that can complete independently of other tools, are easy to run in parallel. If the jobs that create a product take longer and are more difficult to split into components, then the build will be slower, particularly on more powerful hardware.
• Incremental builds are a huge win, particularly for larger processes. Most of the reasons why you want “fast builds,” only require fast rebuilds and partial builds, not necessarily the full “clean builds.” While fast initial builds are not unimportant, they account for a small percentage of use.
• Manage complexity.

There are a lot of things you can do to make builds smarter, which should theoretically make builds faster.

Examples of this kind of complexity include storing dependency information in a database, or using hashing rather than “mtime” to detect staleness, or integrating the build automation with other parts of the development tool chain, or using a more limited method to specify build processes.

The problem, or the potential problem is that you lose simplicity, and it’s possible that something in this “smarter and more complex” system can break or slow down under certain pressures, or can have enough overhead to render them unproductive optimizations.

I’ve found myself writing a fair number of Makefiles in the last few weeks: In part because it was a tool, hell a class of tools, that I didn’t really understand and I’m a big sucker for learning new things, and in part because I had a lot of build process-related tasks to automate. But I think my interest is a bit deeper than that.

Make and related tools provide a good metaphor for thinking about certain kinds of tasks and processes. Build systems are less about making something more efficient (though often it does do that,) and more about making processes reproducible and consistent. In some respects I think it’s appropriate to think of build tools as.

I’ve written here before about the merits of /technical-writing/compilation for documentation, and I think that still holds true: build processes add necessary procedural structure. Indirectly, having formalized build process, also makes it very easy to extend and develop processes as needs change. There’s some up-front work, but it nearly always pays off.

While I want to avoid thinking that everything is a Makefile-shaped nail, I think it’s also probably true that there are a lot of common tasks in general purpose computing that are make shaped: format conversion, extracting and importing data, typesetting (and all sorts of publication related tasks,) archiving, system configuration, etc. Perhaps, more generic build tools need to be part of basic computer literacy. That’s another topic for a much larger discussion.

Finally, I want to raise (or re-raise) the question, that another function of build systems is reduce friction on common tasks and increase the likelihood that tasks will get done, and that people will need less technical background to do fundamentally mundane tasks. Build systems are absolutely essential for producing output from any really complex process because it’s hard to reliably produce builds without them; for less complex processes they’re essential because no one (or fewer people) do those tasks without some kind of support.

Rough thoughts as always.

• The ability to mark a branch “diverged,” to prevent (or warn) on attempted merges from master (for example) into a maintenance branch.

• The ability to create and track dedicated topic branches, and complementary tooling to encourage rebasing commits in these sorts of branches. We might call them “patch sets” or “sets” rather than “branches.” Also, it might be useful to think about using/displaying these commits, when published, in a different way.

• Represent merge commits as hyperlinks to the user, when possible. I think GitHub’s “network graph” and similar visualizations are great for showing how commits and branches interact and relate to each other.

  This would probably require some additional or modifies output from “git log”.

• Named stashes.

• Branched stashes (perhaps this is closer to what I’m thinking about for the request regarding topic branches.)

• The ability to checkout “working copies,” of different points/branches currently from a single repository at the same time, using “native” git utilities.

  Related, “shelf” functionality is scriptable, but this too needs to be easier and more well supported.

  I think legit is a step in the right direction, but it’s weird and probably makes it more difficult to understand what’s happening with git conceptually as opposed to the above features which would provide more appropriate conceptual metaphors for the work that would-be-git-users need.

Assumption:

1. git is pretty awesome, but it’s conceptually complex. As a result using git demands a preexisting familiarity with git itself or some sort of wrapper to minimize the conceptual overhead.
2. The collaboration methods (i.e. hosting) provided by git, which are simple by design to allow maximum flexibility, do not provide enough structure to be practically useful. As a result providers like GitHub (and BitBucket and gitorious) offer a valuable service that makes it easier--or even possible--for people to use git.

Caveats:

• there are problems with using centralized repository services controlled by third parties, particularly for open source/free software projects.

  There are ways that GitHub succeeds an fails in this regard. but this dynamic is too complex to fully investigate within the scope of this post.

• If you use GitHub as designed, and the way that most projects use nGitHub, then you have a very specific and particular view of how Git works.

  While this isn’t a bad thing, it’s less easy to use git in some more distributed workflows as a result. This isn’t GitHub’s fault so much as it is an artifact of people not really knowing how git itself works.

Assertions:

1. GitHub’s “fork” model[^fork] disincentives people from working in “topic” branches.

2. By making it really easy for people to publish their branches, GitHub disincentives the most productive use of the “git rebase” command that leads to clean and clear histories.

3. There’s no distinction between a “soft fork” where you create a fork for the purpose of submitting a patch (i.e. a “pull request”) and a “hard fork,” where you actually want to break the relationship with the original project.

   This is mostly meaningful in context of the other features that GitHub provides, notably the “Network” chart, and the issue tracker. In a soft-fork that I would intend to merge back in, I’d like the issues to “come with,” the repository, or at least connect in some way to the “parent.” For hard forks, it might make sense to leave the old issues behind. The same with the network chart, which is incredibly powerful, but it’s not great at guessing how your repository relates to the rest of its “social network.”

The solution: keep innovating, keep fighting lock-in, and don’t let GitHub dictate how you work.