Practicing

In response to my Knitting Practices post, on Facebook my father commented “The word “practice” is apt. Is there an influence from yoga?”

The answer is obviously “yes,” though the route is somewhat indirect and travels through a story about programming. Stick with it for a little while.

Generators and Python Memory Efficiency

I was talking with a dancing friend about memory efficiency in Python programs, particularly with regards to loops and the range() function. Say you want to do something 1000 times, in Python the easiest way to do this is:

for it in range(1000):
   do_thing(it)

This says, “make a list of numbers from 1 to 1000,” and then call the do_thing() operation on that number (i.e. assigned to the variable it). range(1000) evaluates to a list, which Python stores in memory and the code above loops over.

The way to get this effect in other languages (JavaScript below) is to do something like this:

for (var it=0; it<= 1000; it++) {
   do_thing(it)
}

The for loop has three statements: an action to perform running the content of the loop (create a variable), a condition that will terminate the loop by returning false (variable is less than or equal to 1000), and an operation to run after each loop. The effect of the JavaScript is the same as the Python, except that the Python has to build this (potentially large) list for grins.

There’s a couple of quick answers to this specific question:

  1. (In Python 2) use xrange() which is a special iterable type (i.e. it works with loops), that doesn’t need to build a list in memory, it just spits out values incrementally as needed.

    Let’s imagine that it does this by having a function that returns values starting at a certain point (i.e. 0) pausing after each value, and then returning the next value the next time it runs. Thes ea re called “generators” in Python.

  2. Wait for the Python 3 switch to complete: In Python 3, range() is a generator of sorts, and it’s efficient in this way.

  3. Not care about memory use so much. In most cases this will not be the bottle neck in your application. Really long lists of integers may take up megabytes of memory, but that’s not a huge deal.

Generators are great, and they’re worth using in most cases, but no one will laugh at you if you don’t use them in this situation.

The coolest thing, really is that you can really easily write generator functions. Here’s a silly example: :

def animals("input_animal"=None):
   for animal in [ 'cat', 'dog', 'cow']:
      if input_animal != animal:
          yield animal

This function returns an animal, as long as the input_animal isn’t in the list.

I explained how generators work to my friend and he said something like “Nifty, I guess they’re not really part of my Python practice yet.”

The phrase sort of stuck with me.

Programming is a Practice

There’s a bunch of theory to programming that’s grounded in how computers work, and a lot of things that are actually useful for programmers to know but in truth programming is a practice. Being able to look at a design or a program and understand how to write code to achieve some goal or connect two pieces of functionality, is really about the practice

Beyond understanding what makes good software good and how the machines work, most of programming is knowing the tools really wall and being able to identify the right tool for the job. Everything else is just typing and flow.

While the fundamental knowledge and knowledge of the tools is always important, at some point programming becomes more about being to figure out what the right solution is for any given obstacle or situation.

Knitting as Practice

Knitting is kind of the same. There are some fundamental skills (stitches, operations) and there are some basic fundamental modalities/patterns1 that you use and combine to make some common objects.

Knitting itself is always about repetition, stitch after stitch, but it’s also about repetition on a higher level. Knitting the second sock, the second sleeve. Making pair of sock after pair of sock, or sweater after sweater.

At some point being a knitter stops being about the skills and modalities/patterns and starts being about actually making things and figuring out how to apply what you know about knitting to a new situation: the thing you want to make, the yarn your using, and the tools you’re using.

It’s all practice. All the way down.


  1. In programming terms, these would be patterns. I’m using the term modality/pattern to disambiguate. I’m thinking about things like “how to shape a neck opening on a sweater,” or “turn a heel on a sock.” Not the pattern for an entire object, but the method for knitting a particular object. ↩︎

The First Ankle Sock

I recently finished my first knitting project after my [[return to knitting|/knitting/a-knitting-practice/]]: a pair of ankle socks in a medium (for socks) weight yarn. The socks are plain and I’m pleased with the result, but there were some unexpected parts of the project, and some useful lessons:

These Aren’t The Socks You Meant To Make

For the most part I’m interested in knitting pretty plain socks in plain (solid) yarn: probably even undated yarn that I can soak in tea or henna post-knitting. So the fact that this is one of those patterned, superwash sock yarn is something of a misnomer.

I also intended to get 7 inch cuffs out of the yarn (in a 50 gram ball and I even knit most of a sock: but I ran out of yarn. My assumption that I could knit a 64 stitch sock with sport weight yarn was a touch over the mark, but it was close.

So having failed that project I took a break from the yarn and ripped it all back, and attempted to do a 60 stitch sock (right choice!) and knit a 2 inch cuff.

I could have gotten away with a slightly longer cuff, but I’ve been wanting to have a couple of short pairs of socks to wear while dancing, and to be honest the yarn looked better on the ball than knitted up.

The first moral: Given enough time, I figured it out.

The second moral: It’s ok to rip something out if it doesn’t work out. It’s also ok to not rip it out immediately after you realize it doesn’t work out.

I Kitchnered Toes All By Myself

I’ve never really been successful at kitchnering anything. I find every instruction to be incomprehensible, and every attempt to kitchner something has ended in disaster. My solution for the past couple of years has been to send the occasional sock off to my mother for her to do the honors.

The thing is, I understand the concept of what’s happening, which makes the fact that I couldn’t do it so frustrating. While I was knitting the first sock, I decided that I could probably think my way through the problem, and after a bit of fussing, I was ale to actually complete the sock toe.

I had to turn the sock inside out, and the ends weren’t perfect, but it was close.

The Wearing Report

I wore these socks to a big Balkan Dance festival, and they worked! I thought the cuffs were a bit too short, but the sizing was correct in every other regard. Next pair will have slightly longer cuffs.

Huzzah!

The Perfect Rib

I knit socks on four (5 inch) double pointed needles, these days from the cuff down, and I enjoy socks with some amount of ribbing at the cuff.

Seems reasonable. I also tend to have two other restrictions:

  1. Each needle should have the same number of stitches.
  2. In the ribbing, each needle should begin with a knit stitch and end with a purl stitch.

This somewhat constrains the possibilities. Also given a preference for rubbings that are biased towards knit stitches, I tend to veto options that might otherwise be workable. Here’s a quick cheat sheet of compliant patterns. Patterns are for one needle, repeat for each

  • 12 stitches (48):
    • Knit 2, purl 2 till end.
    • Knit 3, purl 1 till end.
  • 13 stitches (52): Knit 3, purl 2 twice; then knit 2, purl 1.
  • 14 stitches (56): Knit 3, purl 2 twice; then knit 3, purl 1.
  • 15 stitches (60):
    • Knit 3, purl 2 till end.
    • Knit 2, purl 1 till end.
  • 16 stitches (64):
    • Knit 2, purl 2 till end.
    • Knit 3, purl 1 till end.
  • 17 stitches (68): Knit 2, purl 2 twice; then knit 2, purl 1 three times.
  • 18 stitches (72):
    • Knit 2, purl 1 till end.
    • Knit 3, purl 2 three times; then knit 2, purl 1.
  • 19 stitches (76): Knit 3, purl 2 three times; then knit 3, purl 1.
  • 20 stitches (80):
    • Knit 2, purl 2 till end.
    • Knit 3, purl 1 till end.
    • Knit 2, purl 1 till end.
    • Knit 3, purl 2 three times; then knit 2, purl 1.
  • 21 stitches (84):
    • Knit 2, purl 1 till end.
    • Knit 3, purl 2 three times; then knit 2, purl 2.
  • 22 stitches (88):
    • Knit 2, purl 1 six times, then knit 2, purl 2.
    • Knit 3, purl 3; then, knit 2, purl 2 till end..
  • 23 stitches (92):
    • Knit 2, purl 2 five times; then knit 2, purl 1
    • Knit 3, purl 1 five times; then knit 2, purl 1
    • Knit 3, purl 2 four times; then knit 2, purl 1
  • 24 stitches (96):
    • Knit 2, purl 2 till end.
    • Knit 3, purl 1 till end.
    • Knit 2, purl 1 till end.
    • Knit 3, purl 2 three times; then knit 2, purl 1.
  • 25 stitches (100):
    • Knit 3, purl 2 till end.

A Knitting Practice

I’ve not done much knitting for the last year or so. I have a sweater in progress (a cardigan,) and I have an in progress scarf thing, and that’s about it.

Or was.

I enjoy knitting, and find it both relaxing (the rhythm) and stimulating (the meditative aspects, the project planning and design). And I’m pretty good at it. I’ve been knitting (mostly on) for the last 10 years (or so,) and am very technically competent (I think.)

At the same time I lost a lot of interest in knitting. There were and are other things in my life: learning how to make software became (and is) more challenging than making sweaters and shawls, and then there’s singing, a full time (and then some) job, and what not.

I’ve also burnt out on knitting culture. For a long time, part of knitting involved, blogging about knitting, reading knitting blogs, taking pictures of projects and yarn, shopping for yarn, hanging out on ravelry, working in yarn shops, and while knitting itself is a big part of knitting culture I’m realizing a few things:

  1. I don’t really need more sweaters.

    I have a huge pile of sweaters that I never wear in my apartment (where storage is a premium,) and have a huge stack of sweaters in my parents house. While a few exceptions, most of the sweaters that I have (the ones here) aren’t particularly wearable in my day to day life. (They’re odd, or a bit loud, or more commonly too damn warm.)

    Indeed, I bought this Merino cycling zip-up sweater (from Chrome) and it’s basically my new uniform. This has proven a couple of things, first that I can wear wool against my skin without discomfort and second that the more plain and fitted a sweater is the more likely I am to wear it.

  2. The thing that I like the most about knitting is being able to sit down, and let my hands work and let my mind go thinking about the knitting or about something that I’m writing, or nothing at all.

    While I’m good at designing things, and good at doing complicated sweaters, I find that I tend to avoid projects where I end up focusing too much on what needs to happen next, or worry about running out of yarn, or need to follow a pattern.

  3. Knitting is a personal rather than a social act.

    I can knit with other people, but it’s not the kind of thing that I want to do when having a conversation, or actively doing something else. I can concentrate on things that are happening around me, but I can’t really read and I can’t really talk much.

The solution seems pretty clear: don’t knit things you don’t want to do, and knit in a way that makes sense for your interests. I’ll continue to work on the projects that I have in progress, and I’ll probably still make sweaters, but I think I’ll be oing a lot more plain knitting I think a lot more sock knitting.

Sock Knitting

Historically I’ve not knit very many socks, I’ve made a half dozen pairs, some more successful than others, most wearable. I can knit a sock completely off book, and am have a basic sock pattern memorized and don’t even really need to do much test knitting.

I finished the pair of socks that I started knitting (probably six months in the running) worsted socks with a wierd ribbing pattern in an evening and then and cast on for a couple of new pairs. Given that I’ve discovered how awesome wool socks are with a new pair of boots that I have (and that I can wear them on my skin without wanting to die.) I’m even more inspired to knit socks.

The new socks are:

  • a pair of sport weight socks using some striping German sock superwash yarn. 64 stitches, planed 7 inch cuff, 2.5mm (US 1.5) needles, cuff down, 2x2 ribbing.
  • a pair of fingering weight socks (undyed LB 1878) 80 stitches, planed 7 inch cuff, 2.5mm (US 1.5) needles, cuff down, 2x2 ribbing.

The details probably won’t change much. My hope is to be able to figure out a handknit sock pattern that’s suitable for year round wear.

Knitting Blogging Here

I’ll probably continue to write about knitting here, I like writing, and I’m thinking about it, but I’ll probably address this space more like I would a software or philosophy blog, rather than the more typical knitting blog format. I hope to see you around!

2013 Reflections

I have a hard time passing up a significant calendar event as an opportunity to reflect and synthesize past experiences. Having said that, while I had a pretty good year, it was not a year of big changes. There were a lot of pretty interesting and fun smaller changes and developments in my world and my projects in the last year. In no particular order:

  • This was the year that I went from being “someone who tinkers on code” to being someone who can write code when the situation calls for it. This felt like an revolutionary change as it happened, but I recognize now that it has been a long time coming and in retrospect

  • This was the year where I finally slayed my copy quality woes. I’ve long struggled with producing clean copy (English text) in a reasonable amount of time (without spelling/grammar errors.) A couple years ago, I got to a point where I could write clean copy but it felt like I had to spend hours being neurotic. Now, I’ve trained myself to catch things in much more quickly. I’m not perfect, but it’s no longer something that I feel like I struggle with constantly.

  • I reevaluated my relationship with my hobbies and avocational pursuits. I decided to spend more time learning about technology and computer science, I decided to spend less time writing science fiction and less time knitting. I decided to prioritize my social/folk times to go to more singings both in town and out of town, and focus my dancing around spending time with my friends rather than dancing for the sake of dancing.

  • This was the year that I decided to stay in New York City. I’ve been here for almost three years now, and I like living here. While I can envision living somewhere else in the right situation, I think I’ve decided to stay for the long haul. This has meant doing things like furniture shopping (though not buying,) and saving for the eventual purchase of an apartment.

  • I worked too much. But I think I accomplished cool things. In most ways, I think we’re doing some really innovative things with documentation and documentation processes (and writing strong documentation for a cool product.) There have been struggles, as always, but it’s been good. And a lot of work. I don’t think that I would go back and change anything, but I do look forward to trying to establish a little more sanity here.

    There are other things, of course, but that covers it. It seems hard to avoid making resolutions, or at least articulating goals at his juncture:

  • Keep doing the cool things that I’m doing: learning, coding, doing cool things at work.

  • Develop a reading practice. I love reading things, but I’m not very good at reading regularly and keeping up with my reading habit. I’d like to read novels as well as stay generally on top of Asimov’s (and potentially the New York Review of Science Fiction.)

  • Develop a more extensive knitting practice. Full post on the subject forth coming, but I want to knit because I enjoy the activity of knitting and the cadence of making things like socks.

    I want to stop feeling like my knitting needs to have an increasing complexity to make it interesting, or that every thing I knit needs to be unique in some specific way.

  • Become proficient in a programming language that isn’t Python.

  • Start or restart a non-day-job related writing project, and implement a project plan that centers on regular ongoing progress rather than binges.

Coming Soon: Thoughts on implementing goals.

Happy New Year!

Static Site Compiler Design Notes

For a year or more I’ve been playing with the idea of writing my own static site generator. I’ve been producing websites using static site generators for the last 4 or 5 years. For an overwhelming majority of cases, static generation is the right modality; however, there are some significant ways in which the tool chain is not mature.

I’ve been working on an initial pass at a better kind of static site generator, which is a ways away from being “production quality,” but the initial idea is in place, so I feel comfortable discussing some of the details.


Most static site compilers retain simple designs where the process of building the site is non-incremental: all pages are always rebuilt even if the content of the page has not changed. Furthermore, there’s no parallelism, so pages must be built one at a time.

Typically the program will need to process all pages twice, because the rendering phase may require links between content, which complicates the requirements slightly.

The simple design is a huge win for three reasons: it’s easy to understand how these build systems work. There are never any false negatives where pages aren’t rebuilt that should be. Best of all, for small and moderately sized sites, serial/non-incremental approaches are faster.

The downside is pretty big: the amount of time to build the site grows linearly with the number of pages, and there’s no real way to improve performance, particularly as sites become larger.


The design of the “improved” solution is pretty obvious. Iterate through all pages to build and “ingest” their content and metadata. You can do this in parallel, and if possible it makes sense to as much here as you can: build pages that don’t depend on other pages if possible. Also tag pages that include aggregated content here for latter processing.

As you collect the results from these operations, you have enough information to generate the pages that include aggregated information, and you can run and do any remaining rendering.

The only other efficiency is hashing the input page upon initial read as well as the rendered page when you’re done so that you can completely skip pages that don’t need rendering.

For most builds, you might only need to render changed pages; unfortunately, there’s still a fixed cost per-page, which can be amortized somewhat.


For smaller collections of pages, doing everything in a row is faster, and requires less code to get to a proof of concept. As a result, most static site generator implementations follow the serial/non-incremental pattern. The problem is, of course, that this doesn’t hold up as sites grow.

The current state of the art isn’t great:

  • My current blogging solution, ikiwiki does incremental rebuilds but doesn’t rebuild in parallel.
  • Sphinx does incremental rebuilds, though it’s spotty in cases, and for thewrite phase only works in parallel (latest version only). The tinkerer blog tool is built on top of Sphinx.

I think it’s possible to do most of the work of the build in parallel, and I’m interested in writing a site compiler that uses a concurent design, targeted not at raw speed but maximum efficiency. My hope is that such a site generator could make these systems more plausible for general content management for organizations and groups that need flexible systems but don’t need the overhead or complexity of database-driven applications.

I’m using the python job runner as a starting point, building using reStructuredText pages in using the general format of Jekyll pages. Supporting markdown doesn’t seem beyond the realm of possibility. The initial implementation is, of course, in Python on account of my personal comfort and affection for reStructuredText, though I think it would be useful to attempt to port the system to Go at some point, no plans for that at the moment.

Onward and Upward!

Buildcloth v0.3 Planning

I spent a lot of time at the end of the summer working on finishing out the basic buildcloth functionality, and haven’t really gotten the chance to use it properly. There were some flaws:

  • a dumb oversight means that the hash-based dependency checking doesn’t work.
  • Buildcloth is a bit complicated and designed for a general purpose. In practical terms, I made buildcloth to perform a task that I’ve been able to accomplish with 10% or less of the code.
  • There’s no good separation between “the management of a build system” and “the build system data” in the system as it currently exists.

Buildcloth is a nifty idea, and one that I’d like to expand upon. Also since the project is still pre-1.0, it seems reasonable to take these lessons and work on building a more usable implementation.

This post is a collection of thoughts on what I’d like to accomplish for 0.3:

  • pull out the job queue/running system from the build data organization.
  • collect more state in the dependency checking system/infrastructure.
  • separate data ingestion from build system organization.
  • remove ingestion and processing logic from the command line tool.
  • impose sub-module structure to make the interfaces for all of the different aspects of the program.

No clue about time frame. Feel free (and encouraged) to leave comments if you’re interested in helping or have a feature that you’d really like to see.

Onward and Upward!

Documentation Tooling

I’ve spent a little bit of time building some non-critical tools for my teammates on my work project, which has got me thinking about tooling for documentation systems.

This collection of tools is something that we’ve started to take for granted, but I think it’s pretty novel and worth talking about a bit more.


Documentation toolkits, traditionally refer to the system that deals with the production of the documentation for end use, which typically means taking the source text and rendering it into web sites, pdfs, ebooks, and embedded “online help” text.

These toolkits are really important and I think one of the best things you can do for a documentation tool kit is to produce documentation using a tool specifically designed to address the needs of documentation projects and technical writers.

Unfortunately, documentation production is mostly for readers and business owners of documentation and not really for the writers. Tooling for documentation, particularly the kind that I’ve been spending time on recently is about making documentation easier to maintain, and easier to improve at scale. For example:

  • tools to make common textual chunks easier to reuse.
  • tools that enforce common structures.
  • tools that detect common patterns of weak constructions.
  • tools that that analyize text for common readability conventions.

I was going to write this up as a blog post, but I think it makes sense as a collection of wiki pages:

Onward and Upward!