One of the things I’ve realized about myself is a knitter is that I really like epic projects, or sequences of projects.

Before my knitting hiatus, I bought a kilo of undyed sock yarn--because I liked the fiber content, and I wanted to explore knitting sweaters at finer gauges, because heavier weight yarns always made sweaters that felt too warm. Seemed like a good project. Due to a sizing error, I was only really able to get 2 sweaters out of the cone of yarn, but I think in the future three sweaters from a kilo seems useful. There’s something epic about this as a project, and I definitely intend to knit more sweaters in this vein. I was thinking about “epic knitting projects,” and came up with the following ideas:

• 10 identical socks: After many years of having a somewhat hodgepodge and recently well worn collection of socks, this year, I purchased 14 pairs of new socks and it’s been great, both because nothing is teetering on the edge of being worn out, and I decided to buy 5 pairs of two different kinds of socks that I like a lot, plus 4 of a heavier weight, and I like that every day is a “good sock” day. I also like that having a bunch of identical socks socks makes it easier to wash and pair them up. I’ve never really knit more than one pair of socks that matched so it seems like a fun challenge.
• Knit all of Elizabeth Zimmerman’s Yoke Sweater shaping variations. I think I’ve made about three of them, and none of these yoke sweaters have really ever entered more regular rotation. I think knitting them in lighter weights, and reducing the yoke depth a bit to fit more could help a lot.
• Knit a collection of sweater’s inspired by classic Alice Starmore patterns, but modified for modern sensibilities in terms of fit and shaping. I’ve knit three patterns (Henry VIII, Norway, and Faroe) and would gladly knit them all again and I’d love to try the cabled sweaters as well. While the designs and stitch patterns of these sweaters are compelling the fit is not, which is probably a feature of these patterns being written in the late 80s and early 90s. I think the changes would be mostly to knit things at a finer gauge (Faroe, Norway, and the cabled sweaters,) but also to modify the shaping for better fits at the shoulder, and maybe modify neck shape for v-necks or open necklines.

I’ve also thought about knitting a bunch of lace, but I lack the floor space in my current apartment to actually block any kind of lace shawl reasonably.

Anyone else have Epic-scale knitting projects for this list?

It seems obligatory to mark the new year, and what a year it was, for me and everyone else: pandemic, quarantine, changing jobs, new hobbies, totally different routines. The backdrop of the pandemic makes looking back (and forward!) so weird, and I find myself asking: will the things that changed in my life still be true when there’s less quarantine? did the things that happen in 2020 happen because of the pandemic or would they have happened anyway? The certainty that most (but not all!) of 2021 will, in practical terms, look a lot like 2020 is intense and makes this whole “obligatory new years” thing a bit harder. Instead of doing something like “resolutions”/“goals”--which are always fraught--or some kind of lofty synthetic review of the last year, I think I’d like to muse on features of 2020 that I hope and expect to continue in 2021

• Knitting: I took a lot of time off of knitting, but I found that I’d been missing it, and I think there are ways that it fits well into my life quarantine or no. It’s also been quite fun to write about knitting and knitting projects, and become more engaged with other knitters, and I look forward to knitting things for friends and family. I have list in my head of some nifty ideas for sweaters and some other things to knit, so I expect this to stay.
• Blogging: I’ve been writing blog posts for years and while I’ve always found it rewarding, but everything else related to blogging has been hard. During the summer, while I was interviewing for jobs, I wrote blog posts most days, and sort of fell down on posting them, and still have 25 (or so) in the draft folder. Writing is the easy part, it’s editing (or letting go!) promotion, and remembering to move things out of draft. I’ve spruced up the blog and done some work to automate regular publishing, and it seems like I might be able to make this work! I definitely want to.
• Pickling: I’ve really enjoyed pickling things, and if anything I expect that I’ll enjoy doing this more after quarnainte when it’s easier to share these things. Right now I have some cranberries on the go, and a lot of sauerkraut in the fridge that I’m slowly eating. It’ll be fun to have more people to share it with! I’m excited to explore radishes as well as nappa cabbage.
• Coffee: I started drinking coffee in 2014 and have mostly had coffee made by other people: tech jobs in NYC have pretty great office coffee, and the process was something of a mystery to me at the beginning. While I had a Chemex and made coffee for myself sometimes, it was definitely a special occasion sort of thing and not part of my regular routine. Now making a pot of coffee is part of my morning routine, and I find it pretty satisfying, and while I definitely look forward to drinking coffee that other people make more often in the future, I really like getting up making a pot of coffee and sitting down to write nearly every morning.

While I’ve been working on this knitting book project, I’ve realized that I’ve developed something of a backlog of sweaters that I want to knit (for this project, and others,) and I thought I’d write them all down for our collective enjoyment.

• A second version of the favorite sweater I knit during college: a plain sweater with a black body, gray at shoulders, drop shoulder, steeks, and gray sleeves. I want to do this out of fingering weight wool, probably HD Shetland. [in progress]
• The same sweater as above, except with set in sleeves, and probably light blue as the contrasting color at the sleeve.
• Sock yarn sweater, superwash, with v-neck, and some kind of textrued stitch pattern at the yoke that would be knitted without. I made one of these sweaters already, but it came out a touch smaller than I think I really want, and there are a collection of small modifications that I’d like to make, again for verification purposes.
• A fully gray sweater, also out of HD Shetland, with the sleeves knit cuff-up, and with the yoke knit with “set-in sleeves,” in the round. I knit a sweater like this in the fall, and I want to verify that a basic a set of changes would make the sweater really wearable.
• I have the first 3-5 inches of a color work cardigan that I started before my hiatus from knitting: I think there are a number of flaws with this sweater: the sizing is off, I didn’t handle the bottom hem correctly, and I don’t think I have enough yarn in these specific collors, but I think I’d like to attempt the pattern again.
• A color work sweater with set in sleeves. Because knitting rows (back and forth) in stranded is annoying and a bit fussy, most patterns use drop shoulder shaping, but I’d like to experiment and see if I can perfect the technique a more fitted style for these sweaters.
• Using a “garter rib” stitch for the full body of a sweater, both because I like the idea of a gentle rib pulling in to provide more fit, while also being fun and keeping a fairly simple shape over the entire length of the sweater.

I think that’s enough for now!

While I mostly run emacs as a daemon managed by systemd I like to keep my configuration nimble enough that I can just start emacs and have an ad-hoc session for editing some files.¹ Or that’s the goal. This has been a long running project for me, but early on in quarantine times, I’ve gotten things tightened up to the point that I feel comfortable using EDITOR=emacs just about all the time. On my (very old) computers, emacs starts and in well under 1 or 2 seconds, and does better on faster machines. If your emacs setup takes a long time to start up, this article is for you! Let’s see if we can get the program to start faster.

Measure!

The emacs-init-time function in emacs will tell you how long it took emacs to start up. This tends to be a bit under the actual start time, because some start-up activity gets pushed into various hooks, after startup. Just to keep an eye on things, I have a variant of the following as the first line in my config file: :

(add-to-list 'after-init-hook
      (lambda ()
        (message (concat "emacs (" (number-to-string (emacs-pid)) ") started in " (emacs-init-time)))))

It’s also possible to put additional information in this hook (and in reality I use something that pushes to a desktop notification tool, rather than just to message/logging.

You can also lower this number by putting more work into the after-init hook, but that sort of defeats the purpose: if you get the init time down to a second, but have 10 seconds of init-hook runtime, then that’s probably not much of a win, particularly if the init-hook tasks are blocking.

The second thing to do is have a macro² available for use that allows to measure the runtime of a block of code, something like: :

(defmacro with-timer (name &rest body)
  `(let ((time (current-time)))
 ,@body
 (message "%s: %.06f" ,name (float-time (time-since time)))))

You can then do something like: :

(with-timer "mode-line-setup"
  (set-up-mode-line))

And then look in the *Message* buffer to see how long various things take so you know where focus your efforts.

Strategies

• use-package makes it possible to do much less during applciation startup, and makes it possible to load big lisp packages only when you’re going to use them, and registers everything so you never feel the difference. Audit your use-package forms and ensure that they all declare one of the following forms, which insures that things are loaded upon use, rather than at start time:
  • :bind registers specific keybindings, but as autoloads so that the pacakge is only loaded when you use the keybinding.
  • :command registers function names as autoloads, like keybinding, so the commands/functions are available but again the package isn’t loaded until it’s used.
  • :mode attaches the package to a specific file extension or extensions so again, the package doesn’t get loaded until you open a file of that type.
  • :after for loading a package after another package it depends on or is only used after.
  • :hook this allows you to associate something from this package into another package’s hook, with the same effect of deferring loading of one package until after another is used.
• Avoid having the default-scratch buffer trigger loading as much as possible. It’s very tempting to have *scratch* default to org-mode or something, but if you keep it in fundamental-mode you can avoid loading most of your packages until you actually use them, or can avoid pulling in too much too quickly.
• Keep an eye on startup both for GUI, terminal, and if you use daemons, daemon instances of emacs. There are some subtleties that may be useful or important. It’s also the case that terminal startup times are often much less than GUI times.
• Think about strategies for managing state-management (e.g. recentf session-mode and desktop-mode.) I’ve definitley had times when I really wanted emacs to be able to restart and leave me exactly where I was when I shut down. These session features are good, but often it’s just crufty, and causes a lot of expense at start up time. I still use desktop and session, but less so.
• Fancy themes and modelines come at some cost. I recently was able to save a lot of start up time by omiting a call to spaceline-compile in my configuration. Theme setup also can take a bit of time in GUI mode (I think!), but I dropped automatic theme configuration so that my instances would play better with terminal mode.³

Helpful Settings

I think of these four settings as the “start up behavior” settings from my config: :

(setq inhibit-startup-echo-area-message "tychoish")
(setq inhibit-startup-message 't)
(setq initial-major-mode 'fundamental-mode)
(setq initial-scratch-message 'nil)

I’ve had the following jit settings in my config for a long time, and they tend to make things more peppy, which can only help during startup. This may also just be cargo-cult, stuff: :

(setq jit-lock-stealth-time nil)
(setq jit-lock-defer-time nil)
(setq jit-lock-defer-time 0.05)
(setq jit-lock-stealth-load 200)

If you use desktop-mode to save state and re-open buffers, excluding some modes from this behavior is good. In my case, avoiding reopening (and rereading) potentially large org-mode buffers was helpful for me. The desktop-modes-not-to-save value is useful here, as in the following snippet from my configuration: :

(add-to-list 'desktop-modes-not-to-save 'dired-mode)
(add-to-list 'desktop-modes-not-to-save 'Info-mode)
(add-to-list 'desktop-modes-not-to-save 'org-mode)
(add-to-list 'desktop-modes-not-to-save 'info-lookup-mode)
(add-to-list 'desktop-modes-not-to-save 'fundamental-mode)

Notes

A couple of months ago I started writing a thing that I think will be a book, or something close, about knitting. A few weeks ago, I jokingly called it “Gestalt Knitting,” and the name stuck, at least for myself for now. The idea is to write a book, that’s mostly about knitting, that’s less a collection of patterns, and more of a meditation on the process of designing and knitting some sweaters, hats, and socks.

As a knitter, I’ve never really followed patterns: even when I see a design that I like, I always end up changing it somehow: changing the process to be something that I’d enjoy more (e.g. knitting in the round,) or modifying the size a bit (I’ve always been slim and I know I like to avoid wearing things that feel like tents,) or adding features that I know increase the comfort (e.g. making the neckline more open, or the sleeves a touch wider.) Patterns, thus, for me are sort of an opening volley in a conversation about a project rather than a description of a project, and I think in practice this is pretty common, though I think that often, knitters think about their modification as minor and inconsequential when really they’re super cool and important.

I’m less interested in any specific design or pattern, and more interested in these kinds of basic patterns that we can reach to when faced with a pile of yarn and a desire to knit a sweater (or socks, etc.). Sort of the knitting equivalent of a family recipe that you can make without looking at any instructions and maybe without really measuring the ingredients. After a while, I suspect many knitters have a few things that they can just make on their own--like a hat or some simple socks--and one of my goals of this project is to help expand the collection of “house patterns” by explaining and exploring some of the basic garments that I tend to knit.

As a writer, I have a lot of practice writing about the details of complicated ideas and processes in unambiguous and clear terms. I’m interested in seeing if I can take my style for writing about procedures and concepts can translate to another subject area. Knitting patterns are, by convention, super concise and linear in a way that gives people just enough information to reproduce a specific garment, but ends up leaving out so much useful information about why you would knit something in a specific way, options for modification, or interactions between the pattern, the shape, and the process of knitting. When I’m knitting I get a lot of delight from the way all of details of a project come together, and I want to frame and present knitting projects in ways that really highlight that aspect of knitting. I also want to write something that’s knitters would find engaging to read all on its own, even without overlapping any of my specific knitting projects.

While I’ve made a lot of progress on the draft, I still have a bunch of work to do on this project: both in terms of more writing and also knitting “research.” I suspect I’ll write a bit about it here. Stay tuned!

I made a new instagram account, because that seems to be a thing, for knitting specific things: @gestaltknitting

While I knit a lot in college, and for a few years afterwords, my attention to knitting as a hobby kind of trailed off, and I have basically not knit at all in the last five years, but suddenly a bit before Thanksgiving, I found myself listening to an audio book and playing a silly game on my phone really wishing that I could knit, and given that it was 2020 I did, and it’s been a lot of fun and pretty satisfying to get back into knitting.

There were lots of reasons for the hiatus, but the leading reasons were:

• My professional work, first technical writing for software engineers, and then software engineering itself, was actually quite similar to knitting at least conceptually: knitting and software require lots of problem solving and similar kinds of iterative math. Particularly for the first few years where I was teaching myself how to program, I felt like knitting ended up being more like work than I wanted.
• I lived in southern Wisconsin in college, and it was actually cold. Between climate change, and ending up living in New York City--being a huge city that holds heat, and being on a island with the harbor as a heat sink, most buildings in the City have very aggressive heating systems--it felt like I never really wanted to wear wool, because I was often too warm.

These problems seem solveable: my learning curve as a programmer has become less steep, and my day-to-day engineering work tends has shifted to be higher level and organizational in some ways that feel less like knitting. I think the “always too warm” feeling about my city has changed a bit as I’ve acclimated and I think it will be possible to just knit very light weight things and combine them with light weight jackets for increased wearability.

So I’m back.

I’ve finished the sweater I started in 2015 (it’s not exceptional, but I know how to fix it for the future,) knit another sweater that I’m pretty pleased with, and I’ve started a third and I’ve been working on a book about knitting that I’m quite excited about. I never kept a really extensive collection of yarn, but I have enough to keep me busy for a while, and using the yarn I already have has been interesting as a constraining function in planning new projects.

I suspect I’ll be blogging about knitting a bit more over the next little bit, about both specific projects and maybe some higher level things. I hope you don’t mind!

In August of 2019 I became a Staff Engineer, which is what a lot of companies are calling their “level above Senior Engineer” role these days. Engineering leveling is a weird beast, which probably a post onto itself. Despite my odd entry into a career in tech, my path in the last 4 or 5 years has been pretty conventional; however, somehow, despite having an increasingly normal career trajectory, explaining what I do on a day to day basis has not gotten easier.

Staff Engineers are important for scaling engineering teams, but lots of teams get by with out them, and unlike more junior engineers who have broadly similar job roles, there are a lot of different ways to be a Staff Engineer, which only muddies things. This post is a reflection on some key aspects of my experience organized in to topics that I hope will be useful for people who may be interested in becoming staff engineers or managing such a person. If you’re also a Staff Engineer and your experience is different, I wouldn’t be particularly surprised.

Staff Engineers Help Teams Build Great Software

Lots of teams function just fine without Staff Engineers and teams can build great products without having contributors in Staff-type roles. Indeed, because Staff Engineers vary a lot, the utility of having more senior individual contributors on a team depends a lot of the specific engineer and the team in question: finding a good fit is even harder than usual. In general, having Senior Technical leadership can help teams by:

• giving people managers more space and time to focus on the team organization, processes, and people. Particularly in small organizations, team managers often pick up technical leadership.
• providing connections and collaborations between groups and efforts. While almost all senior engineers have a “home team” and are directly involved in a few specific projects, they also tend to have broader scope, and so can help coordinate efforts between different projects and groups.
• increasing the parallelism of teams, and can provide the kind of infrastructure that allows a team to persue multiple streams of development at one time.
• supporting the career path and growth of more junior engineers, both as a result of direct mentoring, but also by enabling the team to be more successful by having more technical leadership capacity creates opportunities for growth for everyone on the team.

Staff Promotions Reflect Organizational Capacity

In addition to experience and a history of effusiveness, like other promotions, getting promoted to Staff Engineer is less straight forward than other promotions. This is in part because the ways we think about leveling and job roles (i.e. to describe the professional activities and capabilities along several dimensions for each level,) become complicated when there are lots of different ways to be a Staff Engineer. Pragmatically, these kind of promotions often depend on other factors:

• the existence of other Staff Engineers in the organization make it more likely that there’s an easy comparison for a candidate.
• past experience of managers getting Staff+ promotions for engineers. Enginering Managers without this kind of experience may have difficulty creating the kinds of opportunities within their organizations and for advocating these kinds of promotions.
• organizational maturity and breadth to support the workload of a Staff Engineer: there are ways to partition teams and organizations that preclude some of the kinds of higher level concerns that justify having Staff Engineers, and while having senior technical leadership is often useful, if the organization can’t support it, it won’t happen.
• teams with a sizable population of more junior engineers, particularly where the team is growing, will have more opportunity and need for Staff Engineers. Teams that are on the balance more senior, or are small and relatively static tend to have less opportunity for the kind of broadly synthetic work that tends to lead to Staff promotions.

There are also, of course, some kinds of technical achievements and professional characteristics that Staff Engineers often have, and I’m not saying that anyone in the right organizational context can be promoted, exactly. However, without the right kind of organizational support and context, even the most exceptional engineers will never be promoted.

Staff Promotions are Harder to Get Than Equivalent Management Promotions

In many organizations its true that Staff promotions are often much harder to get than equivalent promotions to peer-level management postions: the organizational contexts required to support the promotion of Engineers into management roles are much easier to create, particularly as organizations grow. As you hire more engineers you need more Engineering Managers. There are other factors:

• managers control promotions, and it’s easier for them to recapitulate their own career paths in their reports than to think about the Staff role, and so more Engineers tend to be pushed towards management than Senior IC roles. It’s also probably that meta-managers benefit organizationally from having more front-line managers in their organizations than more senior ICs, which exacerbates this bias.
• from an output perspective, Senior Engineers can write the code that Staff Engineers would otherwise write, in a way that Engineering Management tends to be difficult to avoid or do without. In other terms, management promotions are often more critical from the organization’s perspective and therefore prioritized over Staff promotions, particularly during growth.
• cost. Staff Engineers are expensive, often more expensive than managers particularly at the bottom of the brackets, and it’s difficult to imagine that the timing of Staff promotions are not impacted by budgetary requirements.

Promoting a Staff Engineer is Easier than Hiring One

Because there are many valid ways to do the Staff job, and so much of the job is about leveraging context and building broader connections between different projects, people with more organizational experience and history often have an advantage over fresh industry hires. In general:

• Success as a Staff Engineer in one organization does not necessarily translate to success at another.
• The conventions within the process for industry hiring, are good at selecting junior engineers, and there are fewer conventions for more Senior roles, which means that candidates are not assessed for skills and experiences that are relevant to their day-to-day while also being penalized for (often) being unexceptional at the kind of problems that junior engineering interviews focus on. While interview processes are imperfect assessment tools in all cases, they’re particularly bad at more senior levels.
• Senior engineering contributors have a the potential to have huge impact on product development, engineering outcomes, all of which requires a bunch of trust on the part of the organization, and that kind of trust is often easier to build with someone who already has organizational experience

This isn’t to say that it’s impossible to hire Staff engineers, I’m just deeply dubious of the hiring process for these kinds of roles having both interviewed for these kinds of roles and also interviewed candidates for them. I’ve also watched more than one senior contributor not really get along well with a team or other leadership after being hired externally, and for reasons that end up making sense in retrospect. It’s really hard.

Staff Engineers Don’t Not Manage

Most companies have a clear distinction between the career trajectories of people involved in “management” and senior “individual contributor” roles (like Staff Engineers,) with managers involved in leadership for teams and humans, with ICs involved in technical aspects. This seems really clear on paper but incredibly messy in practice. The decisions that managers make about team organization and prioritization have necessary technical implications; while it’s difficult to organize larger scale technical initiatives without awareness of the people and teams. Sometimes Staff Engineers end up doing actual management on a temporary basis in order to fill gaps as organizations change or cover parental leave

It’s also the case that a huge part of the job for many Staff Engineer’s involves direct mentorship of junior engineers, which can involve leading specific projects, conversations about career trajectories and growth, as well as conversations about specific technical topics. This has a lot of overlap with management, and that’s fine. The major differences is that senior contributors share responsibility for the people they mentor with their actual managers, and tend to focus mentoring on smaller groups of contributors.

Staff Engineers aren’t (or shouldn’t be!) managers, even when they are involved in broader leadership work, even if the specific engineer is capable of doing management work: putting ICs in management roles, takes time away from their (likely more valuable) technical projects.

Staff Engineers Write Boring and Tedious But Easy Code

While this is perhaps not a universal view, I feel pretty safe in suggesting that Staff Engineers should be directly involved in development projects. While there are lots of ways to be involved in development: technical design, architecture, reviewing code and documents, project planning and development, and so fort, I think it’s really important that Staff Engineers be involved with code-writing, and similar activies. This makes it easy to stay grounded and relevant, and also makes it possible to do a better job at all of the other kinds of engineering work.

Having said that, it’s almost inevitable that the kinds of contribution to the code that you make as a Staff Engineer are not the same kinds of contributions that you make at other points in your career. Your attention is probably pulled in different directions. Where a junior engineer can spend most of their day focusing on a few projects and writing code, Staff Engineers:

• consult with other teams.
• mentor other engineers.
• build long and medium term plans for teams and products.
• breaking larger projects apart and designing APIs between components.

All of this “other engineering work” takes time, and the broader portfolio of concerns means that more junior engineers often have more time and attention to focus on specific programming tasks. The result is that the kind of code you end up writing tends to be different:

• fixing problems and bugs in systems that require a lot of context. The bugs are often not very complicated themselves, but require understanding the implication of one component with regards to other components, which can make them difficult.
• projects to enable future development work, including building infrastructure or designing an interface and connecting an existing implementation to that interface ahead of some larger effort. This kind of “refactor things to make it possible to write a new implementation.”
• writing small isolated components to support broader initiatives, such as exposing existing information via new APIs, and building libraries to facilitate connections between different projects or components.
• projects that support the work of the team as a whole: tools, build and deployment systems, code clean up, performance tuning, test infrastructure, and so forth.

These kinds of projects can amount to rather a lot of development work, but they definitely have their own flavor. As I approached Staff and certainly since, the kind of projects I had attention for definitely shifted. I actually like this kind of work rather a lot, so that’s been quite good for me, but the change is real.

There’s definitely a temptation to give Staff Engineers big projects that they can go off and work on alone, and I’ve seen lots of teams and engineers attempt this: sometimes these projects work out, though more often the successes feel like an exception. There’s no “right kind” of way to write software as a Staff Engineer, sometimes senior engineer’s get to work on bigger “core projects.” Having said that, if a Staff Engineer is working on the “other engineering” aspects of the job, there’s just limited time to do big development projects in a reasonable time frame.

Related Reading

• More than coding: What is Exploratory Work
• StaffEng.com

I left my job at MongoDB (8.5 years!) at the beginning of the summer, and started a new job at the beginning of the month. I’ll be writing and posting more about my new gig, career paths in general, reflections on what I accomplished on my old team, the process of interviewing as a software engineer, as well as the profession and industry over time. For now, though, I want to write about one of the things I’ve been working on this summer: making a bunch of the open source libraries that I worked on more generally useable. I’ve been calling this the deciduous platform,¹ which now has its own github organization! So it must be real.

The main modification in these forks, aside from adding a few features that had been on my list for a while, has been to update the buildsystem to use go modules² and rewrite the history of the repository to remove all of the old vendoring. I expect to continue development on some aspects of these over time, though the truth is that these libraries were quite stable and were nearly in maintenance mode anyway.

Background

The team was responsible for a big monolith (or so) application: development had begun in 2013, which was early for Go, and while everything worked, it was a bit weird. My efforts when I joined in 2015 focused mostly on stabilization, architecture, and reliability. While the application worked, mostly, it was clear that it suffered from a few problem, which I believe were the result of originating early in the history of Go: First, because no one had tried to write big applications yet, the patterns weren’t well established, and so the team ended up writing code that worked but that was difficult to maintain, and ended up with bespoke solutions to a number of generic problems like running workloads in the background or managing Apia. Second, Go’s standard library tends to be really solid, but also tends towards being a little low level for most day-to-day tasks, so things like logging and process management end up requiring more code³ than is reasonable.

I taught myself to write Go by working on a logging library, and worked on a distributed queue library. One of the things that I realized early, was that breaking the application into “microservices,” would have been both difficult and offered minimal benefit,⁴ so I went with the approach of creating a well factored monolith, which included a lot of application specific work, but also building a small collection of libraries and internal services to provide useful abstractions and separations for application developers and projects.

This allowed for a certain level of focus, both for the team creating the infrastructure, but also for the application itself: the developers working on the application mostly focused on the kind of high level core business logic that you’d expect, while the infrastructure/platform team really focused on these libraries and various integration problems. The focus wasn’t just organizational: the codebases became easier to maintain and features became easier to develop.

This experience has lead me to think that architecture decisions may not be well captured by the monolith/microservice dichotomy, but rather there’s’ this third option that centers on internal architecture, platforms, and the possibility for developer focus and velocity.

Platform Overview

While there are 13 or so repositories in the platform, really there are 4 major libraries: grip, a logging library; jasper, a process management framework; amboy, a (possibly distributed) worker queue; and gimlet, a collection of tools for building HTTP/REST services.

The tools all work pretty well together, and combine to provide an environment where you can focus on writing the business logic for your HTTP services and background tasks, with minimal boilerplate to get it all running. It’s pretty swell, and makes it possible to spin up (or spin out) well factored services with similar internal architectures, and robust internal infrastructure.

I wanted to write a bit about each of the major components, addressing why I think these libraries are compelling and the kinds of features that I’m excited to add in the future.

Grip

Grip is a structured-logging friendly library, and is broadly similar to other third-party logging systems. There are two main underlying interfaces, representing logging targets (Sender) and messages, as well as a higher level “journal” interface for use during programming. It’s pretty easy to write new message or bakcends, which means you can use grip to capture all kinds of arbitrary messages in consistent manners, and also send those messages wherever they’re needed.

Internally, it’s quite nice to be able to just send messages to specific log targets, using configuration within an application rather than needing to operationally manage log output. Operations folks shouldn’t be stuck dealing with just managing logs, after all, and it’s quite nice to just send data directly to Splunk or Sumologic. We also used the same grip fundamentals to send notifications and alerts to Slack channels, email lists, or even to create Jira Issues, minimizing the amount of clunky integration code.

There are some pretty cool projects in and around grip:

• support for additional logging targets. The decudous version of grip adds twitter as an output format as well as creating desktop notifications (e.g. growl/libnotify,) but I think it would also be interesting to add fluent/logstash connections that don’t have to transit via standard error.'
• While structured logging is great, I noticed that we ended up logging messages automatically in the background as a method of metrics collection. It would be cool to be able to add some kind of “intercepting sender” that handled some of these structured metrics, and was able to expose this data in a format that the conventional tools these days (prometheus, others,) can handle. Some of this code would clearly need to be in Grip, and other aspects clearly fall into other tools/libraries.

Amboy

Amboy is an interface for doing things with queues. The interfaces are simple, and you have:

• a queue that has some way of storing and dispatching jobs.
• implementations of jobs which are responsible for executing your business logic, and with a base implemention that you can easily compose, into your job types, all you need to implement, really is a Run() method.
• a queue “group” which provides a higher level abstraction on top of queues to support segregating workflows/queues in a single system to improve quality of service. Group queues function like other queues but can be automatically managed by the processes.
• a runner/pool implementation that provides the actual thread pool.

There’s a type registry for job implementations and versioning in the schema for jobs so that you can safely round-trip a job between machines and update the implementation safely without ensuring the queue is empty.

This turns out to be incredibly powerful for managing background and asynchronous work in applications. The package includes a number of in-memory queues for managing workloads in ephemeral utilities, as well as a distributed MongoDB backed-queue for running multiple copies of an application with a shared queue(s). There’s also a layer of management tools for introspecting, managing, the state of jobs.

While Amboy is quite stable, there is a small collection of work that I’m interested in:

• a queue implementation that store jobs to a local Badger database on-disk to provide single-system restartabilty for jobs.
• a queue implementation that stores jobs in a PostgreSQL, mirroring the MongoDB job functionality, to be able to meet job backends.
• queue implementations that use messaging systems (Kafka, AMPQ) for backends. There exists an SQS implementation, but all of these systems have less strict semantics for process restarts than the database options, and database can easily handle on the order of a hundred of thousand of jobs an hour.
• changes to the queue API to remove a few legacy methods that return channels instead of iterators.
• improve the semantics for closing a queue.

While Amboy has provisions for building architectures with workers running on multiple processes, rather than having queues running multiple threads within the same process, it would be interesting to develop more fully-fledged examples of this.

Jasper

Jasper provides a high level set of tools for managing subprocesses in Go, adding a highly ergonomic API (in Go,) as well as exposing process management as a service to facilitate running processes on remote machines. Jasper also manages/tracks the state of running processes, and can reduce pressures on calling code to track the state of processes.

The package currently exposes Jasper services over REST, gRPC, and MongoDB’s wire protocol, and there is also code to support using SSH as a transport so that you don’t need to expose remote these services publically.

Jasper is, perhaps, the most stable of the libraries, but I am interested in thinking about a couple of extensions:

• using jasper as PID 1 within a container to be able to orchestrate workloads running on containers, and contain (some) support for lower level container orchestration.
• write configuration file-based tools for using jasper to orchestrate buildsystems and distributed test orchestration.

I’m also interested in cleaning up some of the MongoDB-specific code (i.e. the code that downloads MongoDB versions for use in test harnesses,) and perhaps reinvisioning that as client code that uses Jasper rather than as a part of Jasper.

Gimlet

I’ve written about gimlet here before when I started the project, and it remains a pretty useful and ergonomic way to define and regester HTTP APIs, in the past few years, its grown to add more authentication features, as well as a new “framework” for defining routes. This makes it possible to define routes by implementing an interface that:

• makes it very easy to produce paginated routes, and provides some helpers for managing content
• separates the parsing of inputs from executing the results, which can make route definitions easy to test without integration tests.
• rehome functionality on top of chi router. The current implementation uses Negroni and gorilla mux (but neither are exposed in the interface), but I think it’d be nice to have this be optional, and chi looks pretty nice.

Other Great Tools

The following libraries are defiantly smaller, but I think they’re really cool:

• birch is a builder for programatically building BSON documents, and MongoDB’s extended JSON format. It’s built upon an earlier version of the BSON library. While it’s unlikely to be as fast at scale, for many operations (like finding a key in a document), the interface is great for constructing payloads.
• ftdc provides a way to generate (and read,) MongoDB’s diagnostic data format, which is a highly compressed timeseries data format. While this implementation could drift from the internal implementation over time, the format and tool remain useful for arbitrary timeseries data.
• certdepot provides a way to manage a certificate authority with the certificates stored in a centralized store. I’d like to add other storage backends over time.

And more…

Notes