I wrote this post nearly 5 years ago, and have been sitting on the draft for a long time: not for any reason, I think it’s actually pretty good post. For non-knitters, this is kind of a “ask a great cook for their comfort food recpie,” but in a narrative form.

In any case, I haven’t really been knitting very much recently, and while I enjoy writing knitting patterns there’s a lot of work in writing a well formed knitting pattern that I’m poorly positioned for ring now (test knitting! good photography! talking with knitters!) But, perhaps someone will find it useful… Enjoy!

Part of my recent return to knitting has been about taking a much more simple approach to yarn. I think yarn is cool and working with good yarn is awesome but at a certain point, I think yarn distracts from the things that I like most about knitting: the consistency, the dependability, the rhythm of the activity, and coordination of parallel activities.

Novel yarns and yarn variety actually makes the process of knitting less enjoyable for me. It also doesn’t really jive with my taste in clothing: I like plain things that fit well without a lot of adornment. While I enjoy knitting patterned sweaters for the rhythm, I don’t really wear them much. I also, live and spend my time in a climate where a I’m almost always wearing a light sweater (during the cold months) and inside during the rest of the months.

The result of this is that I’ve mostly been working on sock knitting. I like wearing wool socks, and after a period of not wearing them for a few reasons, I didn’t actually have that many wool socks. Which has lead me to get acclimated to knitting fingering weight yarn with size 0 needles.

So I want to make a sweater in this mold: fingering weight, very plain lines, probably knit in the round using the Elizabeth Zimmerman system. Starting from the bottom, I’ve been leaning away from ribbings at the bottom, and have tended to like hems though they sometimes flare. Ususally, I just cast on provisionally and add the hem (or whatever) at the end anyway. There’s time yet to decide.

More importantly, I’m quite interested in having a rolled collar for the neck, but I tend to think that rolled ends mostly have a flare look anyway. I can defer this decision for a while.

For shaping and even most of the styling I intend to copy the Chrome Cobra Zip Up, which is, by far, my favorite article of clothing.

I think really subtle increases (so that the sweater tapers to the waist) is a good feature and might choose to do some of those, particularly if the model sweater has them. The model sweater has a really long back, and I think I might moderate this slightly.

The shoulders are an open question. If this goes well, I think I’d like to knit my way though most of the standard EZ shoulder constructions: I think I’ve knit all of the options at least once, but I’ve not done all of them in plain knitting, and most of the sweaters are a bit odd in one way or another. There’s a long project. I want to start with, and hopefully master, the set-in-sleeve.

For those of you playing along at home, set-in-sleeves are probably what you think of as “normal” sleeves, the garment fits in the shoulders, and the sleeves angle gently down from the shoulders. Most shirts have this shaping but the shapes aren’t terribly natural for knitting.

To knit set in sleeves in the round, you join the sleeves to the body, setting some stitches aside where the pieces meet and then decreasing body snitches into the sleeves as you knit until the body is just as wide as the shoulders. Then decrease the sleeve stitches into the body until you have about 3 inches of sleeve stitches left. Finally, knit knit short rows across the front and back (or just the front) stitches, decreasing the remaining stitches in the short rows ending with a 3 needle bind-off at the appropriate moment. To get a good crew neck, begin shaping the front of the neck every row 1.5 inches before you start the shoulder short-rows, and shape the back of the neck every other row when you start the shoulder short rows.

The shaping and body of the knitting is pretty straightforward from design perspective. The hard part from the perspective of the success of the sweater is the hems and/or ribbing, and figuring out the right thing to for each hem. It’s always something.

The longer that I have this job, the more difficult it is to explain what I do. I say, “I’m a programmer,” and you’d think that I write code all day, but that doesn’t map onto what my days look like, and the longer it seems the less code I actually end up writing. I think the complexity of this seemingly simple question grows from the fact that building software involves a lot more than writing code, particularly as projects become more complex.

I’d venture to say that most code is written and maintained by one person, and typically used by a very small number of pepole (often on behalf of many more people,) though this is difficult to quantify. Single maintainer software is still software, and there are lots of interesting problems, but as much as anything else I’m interested in the problems adjacent to multi-author code-bases and multi-operator software development.¹

Fundamentally, I’m interested in the following questions:

• How can (sometimes larger) groups of people collaborate to build something that’s bigger than the scope of any of their work?
• How can we build software in a way that lets individual developers focus most of the time on the features and concerns that are the most important to them and their users.²

The software development process, regardless of the scope of the problem, has a number of different aspects:

• Operations: How does is this software execute and how do we know that its successful when it runs?
• Behavior: What does it do, and how do we ensure it has the correct behavior?
• Interface: How will users interact with the process, and how do we ensure a consistent experience across versions and users' environment?
• Product: Who are the users? What features do they want? Which features are the most important?

Sometimes we can address these questions by writing code, but often there’s a lot of talking to users, other developers, and other people who work in software development organizations (e.g. product managers, support, etc.) not to mention writing a lot of English (documentation, specs, and the like.)

I still don’t think that I’ve successfully answered the framing question, except to paint a large picture of what kinds of work goes into making software, and described some of my specific domain interests. This ends up boiling down to:

• I write a lot of documents describing new features and improvements to our software. [product]
• I think a lot about how our product works as it grows (scaling), and what kinds of changes we can make now to make that process more smooth. [operations]
• How can I help the more junior members of my team focus on the aspects of their jobs that they enjoy the most, and help illustrate broader contexts to them. [mentoring]
• How can we take the problems we’re solving today and build the solution that balances the immediate requirements with longer term maintainability and reuse. [operations/infrastructure]

The actual “what” I’m spending my time boils down to reading a bunch of code, meeting with my teamates, meeting with users (who are also coworkers.) And sometimes writing code. If I’m lucky.

A lot of my work, these days, focuses on figuring out how to improve how people develop software in ways that reduces the amount of time developers have to spend doing work outside of development and that improves the quality of their work. This post, has been sitting in my drafts folder for the last year, and does a good job of explaining how I locate my workand* makes a case for high quality generic build system tooling that I continue to feel is compelling.*

Incidentally, it turns out that I wrote an introductory post about buildsystems 6 years ago. Go past me.

Canonically, build systems described the steps required to produce artifacts, as system (graph) of dependencies¹ and these dependencies are between source files (code) and artifacts (programs and packages) with intermediate artifacts all in terms of the files they are or create. Though different development environments, programming languages, and kinds of software have different.

While the canonical “build systems are about producing files,” the truth is that the challenge of contemporary _software development isn’t really just about producing files. Everything from test automation to deployment is something that we can think about as a kind of build system problem.

Let’s unwind for a moment. The work of “making software,” breaks down into a collection of--reasonably disparate--tasks, which include:

• collecting requirements (figuring out what people want,)
• project planning (figuring out how to break larger collections of functionality into more reasonable units.)
• writing new code in existing code bases.
• exploring unfamiliar code and making changes.
• writing tests for code you’ve recently written, or areas of the code base that have recently chaining.
• rewriting existing code with functionally equivalent code (refactoring,)
• fixing bugs discovered by users.
• fixing bugs discovered by an automated test suite.
• releasing software (deploying code.)

Within these tasks developers do a lot of little experiments and tests. Make a change, see what it’s impact is by doing something like compiling the code, running the program or running a test program. The goal, therefore, of the project of developer productivity projects is to automate these processes and shorten the time it takes to do any of these tasks. In particular the feedback loop between “making a change” and seeing if that change had an impact. The more complex the system that you’re developing, with regards to distinct components, dependencies, target platforms, compilation model, and integration’s, them the more time you spend in any one of these loops and the less productive you can be.

Build systems are uniquely positioned to suport the development process: they’re typically purpose built per-project (sharing common infrastructure,) most projects already have one, and they provide an ideal environment to provide the kind of incremental development of additional functionality and tooling. The model of build systems: the description of processes in terms of dependency graphs and the optimization for local environments means.

The problem, I think, is that build systems tend to be pretty terrible, or at least many suffer a number of classic flaws:

• implicit assumptions about the build or development environment which make it difficult to start using.
• unexpressed dependencies on services or systems that the build requires to be running in a certain configuration.
• improperly configured dependency graphs which end up requiring repeated work.
• incomplete expression of dependencies which require users to manually complete operations in particular orders.
• poorly configured defaults which make for overly complex invocations for common tasks.
• operations distributed among a collection of tools with little integration so that compilation, test automation, release automation, and other functions.

By improving the quality, correctness, and usability of build systems, we:

• improve the experience for developers every day,
• make it really easy to optimize basically every aspect of the development process,
• reduce the friction for including new developers in a project’s development process.

I’m not saying “we need to spend more time writing build automation tools” (like make, ninja, cmake, and friends,) or that the existing ones are bad and hard to use (they, by and large are,) but that they’re the first and best hook we have into developer workflows. A high quality, trustable, tested, and easy to use build system for a project make development easier, continuous integration easy and maintainable, and ultimately improve the ability of developers to spend more of their time focusing on important problems.

I wrote this thing about monitoring in software, and monitoring in web applications (and similar) about a year ago, and I sort of forgot about it, but as I was cleaning up recently I found this, and think that I mostly still agree with the point. Enjoy!

It is almost always the case that writing software that does what you want it to do is the easy part and everything else is the hard part.

As your software does more a number of common features emerge:

• other people are responsible for operating your software.
• multiple instances of the program are running at once, on different computers.
• you may not be able to connect to all of the running instances of the program when something goes wrong.
• people will observe behaviors that you don’t expect and that you won’t be able to understand by observing the program’s inputs or outputs''

There are many things you can do to take a good proof of concept program and turn it into a production-ready program, but I think logging and introspection abilities are among the most powerful: they give you the most bang for your buck, as it were. It’s also true that observability (monitoring) is a hot area of software development that’s seeing a lot of development and thought at the moment.

While your application can have its own internal reporting system, its almost always easier to collect data in logs first rather than

Aggregate Logs

Conventionally operators and developers interact with logs using standard unix stream processing tools: tail, less, and grep and sometimes wc, awk, and sed. This is great when you have one (or a small number) process running on one machine. When applications get bigger, stream processing begins to break down.

Mostly you can’t stream process because of volume there’s too much data, it’s hard to justify spending disk space on all of your application servers on logs, and there’s too much of it to look at and do useful things. It’s also true that once you have multiple machines, its really helpful to be able to look at all of the logs in a single place.

At the lowest level the syslog protocol and associated infrastructure solves this problem by providing a common way for services to send log data via a network (UDP, etc.) It works but you still only have stream processing tools, which may be fine, depending on your use case and users.

Additionally there are services and applications that solve this problem: splunk (commercial/enterprise software ) sumologic (commercial/cloud software) and the ELK stack (an amalgamation of open source tools.) that provide really powerful ways to do log search, reporting, and even build visualizations. There are probably others as well.

Use them.

Structure Logs

The most common interview question for systems administrators that my colleagues give is a “log sawing” question. This seems pretty standard, and is a pretty common exercise for parsing information out of well known streams of log data. Like “find a running average request time,” or figure out the request rate.

The hard part is that most logs, in this example are unstructured in the sense that they are just line-wise printed strings, and so the exercise is in figuring out the structure of the messages, parsing data from the string, and then tracking data over the course of the logs. Common exercise, definitely a thing that you have to do, and also totally infuriating and basically impossible to generalize.

If you’re writing software, don’t make your users do this kind of thing. Capture events (log messages) in your program and output them with the information already parsed. The easiest way is to make your log messages mapping types, and then write them out in JASON, but there are other options.

In short, construct your log messages so that they’re easy to consume by other tools: strongly (and stably) type your messages, provide easy way to group and filter similar messages. Report operations in reasonable units (e.g. seconds rather than nanoseconds) to avoid complex calculations during processing, and think about how a given data point would beinteresting to track over time.

Annotate Events

Building on the power of structured logs, it’s often useful to be able to determine the flow of traffic or operations through the system to make it possible to understand the drivers of different kinds of load, and the impact of certain kinds of traffic on overall performance. Because a single operation may impact multiple areas of the system, annotating messages appropriately makes it possible to draw more concrete conclusions based on the data you collect.

For example when a client makes a user request for data, your system probably has a request-started and request-ended event. In addition this operation may retrieve data, do some application-level manipulation, modify other data, and then return it to the user. If there’s any logging between the start and end of a request, then it’s useful to tie these specific events together, and annotations can help.

Unlike other observability strategies, there’s not a single software feature that you can use to annotate messages once you have structured capabilities, although the ability of your logging systems to have some kind of middleware to inject annotations is quite useful.

Collect Metrics

In addition to events produced by your system, it may be useful to have a background data collection thread to report on your application and system’s resource utilization. Things like, runtime resource utilization, garbage collector stats, and system IO/CPU/Memory use can all be useful.

There are ways to collect this data via other means, and there are a host of observability tools that support this kind of metrics collection. But using multiple providers complicates actually using this data, and makes it harder to understand what’s going in the course of running a system. If your application is already reporting other stats, consider bundling these metrics in your existing approach.

By making your application responsible for system metrics you immediately increase the collaboration between the people working on development and operations, if such a divide exists.

Conclusion

In short:

• collect more data,
• increase the fidelity and richness of the data you collect,
• aggregate potentially related data in the same systems to maximize value,
• annotate messages to add value, and provide increasingly high level details.

I’ve been thinking recently about the way that my interest in writing and publishing blog posts has waned, and while there are a lot of factors that have contributed to this, I think there’s a big part of me that questions what the purpose of writing is or should be, and because I mostly write about the things I’m learning or thinking about, my posts end up being off-the-cuff explanations of things I’ve learned or musings on a theoretical point which aren’t particularly well referenced, and while they’re fun to write and useful for my own process they’re not particularly useful to anyone else. Realizing this puts me at something of a crossroads with my own writing, and has me thinking a lot about the practice of citation.¹

Mechanically, citation anchors text in relationship to other work,² but it also allows discussion to happen in and between texts. Also, the convention for citation in the context of informal writing is a link or an informal reference, so it’s difficult to track over time, and hard to be systemtic in the way that one text interacts with its sources.

Blogs bring out confessional writing with ambling³ structure and the freedom to say just about anything, which I have found liberating and generally instructive, but it’s also limiting. For writing that comes out of personal experience, it’s difficult to extrapolate and contextualize your argument, or even to form an argument, particularly in the context of a blog where you’re writing a larger number of shorter pieces. It’s also probably true that by framing discussions in personal experience its hard for people with different experiences to relate to the content, and more importantly the concepts within.

I’m not arguing against journaling: journals are greatgreat, but sometimes, I think journals might be best unpublished. I’m also not arguing against the personal essay as a form: there are many topics that are well served by that genre of writing. I do want to think about what else is possible⁴ and how to write things that are stronger, more grounded, and easier to relate to and interact with. I think more citations and references are the key, but I’m left with two problems:

1. Style. There aren’t great conventions for referencing things in informal writing. Throwing a link in the right context works, and is clear, but it might not be enough as it’s hard to know what’s a citation-typed-reference and other kinds of links. Also links don’t hold up well over time. The more formal approaches are rooted in out of date technologies and tactics. Citations often reference page numbers, footnotes don’t often make sense in informal situations,⁵ and bibliography conventions are mostly non-existant.
2. Tooling. I’m pretty sure that well cited texts are well-cited, because their authors have great memories for things they’ve read, but because researchers often have tooling that supports managing a database of references, notes and bibliographic information. If you have a record of the resources you’ve read (or otherwise consumed), it becomes easier to pull out citations as you write and edit.⁶

Neither of these are insurmountable, but I think would require a good deal of work both on figuring out better citation formats and patterns, as well as developing better tooling. I don’t have answers yet, but I do want to think more about it, and probably play with writing some tools.

I think the queue is a really powerful metaphor for organizing and orchestrating the internal architecture of an application. Once you have a queue, and tasks that are running in that queue, making that system run well requires some attention. This post is very much a sequel to the application framework post.

Factors of your queue implementation and system may impact the applicability of any of these suggestions for your particular application. Additionally, there is lots of work on queue theory so there are formally described properties of queues, and this is really just a collection of informal knowledge that I’ve collected on this subject. I hope you find it useful!

As the operator of a queue there are two properties: latency, or time to completion, for work in the queue and throughput, or total amount of work completed. These properties are generally trade-offs with each other, and often work to improve throughput will impact latency, and vice versa. It turns out, however, that the theoretical limits of your system’s capacity for either latency or throughput are below the actual requirements of your application, so you can generally just focus on improving one area or the other without really feeling like you’re trading latency for throughput.

All tasks in the queue should, generally, of similar size in terms of execution time and resource usage. When there are tasks that run slowly or take a long time and tasks that run quickly, you can easily end up in situations where long running tasks group together. Indeed, this isn’t just a possibility, but a near certainty. If you can’t break work into similar sized units, then you main recourse is to either separate the work into different queues and proportion resources as needed to ensure that both queues are making progress. You generally want to run longer tasks before shorter tasks, but the impact on overall performance depends on other characteristics and the way that your application expects certain kinds of latency and throughput.

Always monitor task runtime (by type,) as well as overall queue depth, and if possible currently running operations. When something goes wrong, or there’s an external event that impacts queue performance, you’ll need these data to understand the state of your world and debug the underlying issue. Don’t wait for something to go wrong to set this up.

Idempotentcy, or the ability of a task to run more than once without chaining the final outcome is a great property in any distributed system, but the more idempotent your operations are the less you have to worry about edge cases where you might run them more than once, particularly around process restarts and deployments. While you generally only want to run things once for efficiency sake, it’s important to be able to know that you can run things more than once without causing a crisis.

While it’s easy to think about the time a task spends waiting in a queue when tasks are ordered in the queue in a first-in-first-out model, other ordering mechanisms can easily lead to items getting stuck in the queue. Consider the following behaviors:

• if there are dependencies between tasks, or cases where doing work leads to the creation of more tasks, always run these tasks earlier before other tasks.
• consider boosting the relative priority of tasks that have been waiting longer relative to other tasks in the queue. If tasks have a priority, and new tasks come in that have higher priority than older tasks, then some lower priority tasks may never get done. While priority of tasks is important, if its important that all tasks get done, balance wait time with priority.
• alternatively, consider elimiting tasks that have been waiting in the queue for longer than a specified period. These “TTL” for queue items can avoid wasting resources doing work that is not useful.
• separate desperate priority or types of work into seperate queues to reduce latency. Having additional queues often incurs some per-queue/per-worker resource overhead. When worker infrastructure canbe shared between queues, and both queues are not consistently running at capacity (e.g. have backlogs).
• quantify the job dispatching overhead. While generally breaking apart larger tasks into smaller execution units improves efficiency, if the overhead of creating, dispatching, and running jobs is a significant portion of a job’s runtime, then your system is probably spending too many resources on overhead and you can increase throughput by increasing the overall task size.

There’s more, but this definitely covers the basics.

It happened gradually, and it wasn’t entirely an intentional thing, but at some point I became a software engineer. While a lot of people become software engineers, many of them have formal backgrounds in engineering, or have taken classes or done programs to support this retooling (e.g. bootcamps or programming institutes.)

I skipped that part.

I wrote scripts from time to time for myself, because there were things I wanted to automate. Then I was working as a technical writer and had to read code that other people had written for my job. Somewhere in there I was responsible for managing the publication workflow, and write a couple of build systems.

And then it happened.

I don’t think it’s the kind of thing that is right for everyone, but I was your typical, nerdy/bookish kid who wasn’t great in math class, and I suspect that making software is the kind of thing that a lot of people could do. I don’t think that my experience is particularly replicable, but I have learned a number of useful (and important) things, and I realize as I’ve started writing more about what I’m working on now, I realize that I’ve missed some of the fundamentals¹

Formal education in programming, from what I’ve been able to gather strikes me as really weird: there are sort of two main ways of teaching people about software and computer science: Option one is that you start with a very theoretical background that focuses on data structures, the performance of algorithms, or the internals of how core technologies function (operating systems, compilers, databases, etc.) Option two, is that you spend a lot of time learning about (a) programming language and about how to solve problems using programming.

The first is difficult, because the theory² is not particularly applicable except invery rare cases and only at the highest level which is easy to back-fill as needed. The second is also challenging, as idioms change between languages and most generic programming tasks are easily delegated to libraries. The crucial skill for programming is the ability to learn new languages and solve problems in the context of existing systems, and developing a curriculum to build those skills is hard.

The topics that I’d like to write about include:

• Queue behavior, particularly in the context of distributed systems.
• Observability/Monitoring and Logging, particularly for reasonable operations at scale.
• build systems and build automation.
• unit-testing, test automation, and continuous integration.
• interface design for users and other programmers.
• maintaining and improving legacy systems.

These are, of course, primarily focused on the project of making software rather than computer science or computing in the abstract. I’m particularly interested (practically) in figuring out what kinds of experiences and patterns are important for new programmers to learn, regardless of background.³ I hope you all find it interesting as well!

The byproduct of a lot of my work on Evergreen over the past few years has been that I’ve amassed a small collection of reusable components in the form of libraries that address important but not particularly core functionality. While I think the actual features and scale that we’ve achieved for “real” features, the infrastructure that we built has been particularly exciting.

It turns out that I’ve written about a number of these components already here, even. Though I think, my initial posts were about these components in their more proof-of-concept stage, now (finally!) we’re using them all in production so their a bit more hardened.

The first grip is a logging framework. Initially, I thought a high-level logging framework with plug-able backends was going to be really compelling. While configurable back-ends has been good for using grip as the primary toolkit for writing messaging and user-facing alerting, the most compelling feature has been structured logging.

Most of the logging that we do, now, (thanks to grip,) has been to pass structures (e.g. maps) to the logger with key/value data. In combination with log aggregation services/tools (like ELK, splunk, or sumologic,) we can basically take care of nearly all of our application observablity (monitoring) use cases in one stop. It includes easy to use system and golang runtime metrics collection, all using an easy push-based collection, and can also power alert escalation. After having maintained an application using this kind of event driven structured logging system, I have a hard time thinking about running applications without it.

Next we have amboy which is a queue-system. Like grip, all of the components are plug-able, so it support in-memory (ephemeral) queues, distributed queues, dependency graph systems and priority queue implementations as well as a number of different execution models. The most powerful thing that amboy affords us is a single and clear abstraction for defining “background” execution and workloads.

In go it’s easy to spin up a go routine to do some work in the background, it’s super easy to implement worker pools to parallelize the processing of simple tasks. The problem is that as systems grow, it becomes pretty hard to track this complexity in your own code, and we discovered that our application was essentially bifurcated between offline (e.g. background) and online (e.g. request-driven) work. To address all of this problem, we defined all of the background work as small, independent units of work, which can be easily tested, and as a result there is essentially no-adhoc concurrency in the application except what runs in the queues.

The end result of having a unified way to characterize background work is that scaling the application because much less complicated. We can build new queue implementations, without needing to think about the business logic of the background work itself, and we add capacity by increasing the resources of worker machines without needing to think about the architecture of the system. Delightfully, the queue metaphor is independent of external services, so we can run the queue in memory backed by a heap or hash map with executors running in dedicated go-routines if we want, and also scale it out to use databases or dedicated queue services with additional process-based workers, as needed.

The last component, gimlet, addresses building HTTP interfaces, and provides tools for registering routes, writing responses, managing middleware and authentication, an defining routes in a way that’s easy to test. Gimlet is just a wrapper around some established tools like negroni, gorilla/mux, all built on established standard-library foundations. Gimlet has allowed us to unify a bunch of different approaches to these problems, and has lowered the barrier to entry for most of our interfaces.

There are other infrastructural problems still on the table: tools for building inter-system communication and RPC when you can’t communicate via a queue or a shared database (I’ve been thinking a lot about gRPC and protocol buffers for this,) and also about object-mapping and database access patterns, which I don’t really have an answer for.¹

Nevertheless, with the observability, background tasks, and HTTP interface problems well understood at supported, it definitely frees developers to spend more of their time focused core problems of importance to users and the goals of the project. Which is a great place to be.

I’ve had a blog¹ for more than 15 years, and I’ve found this experience to be generally quite rewarding. I’ve learned a lot about writing, and enjoyed the opportunity to explore a number of different topics² in great detail. While I haven’t blogged as much in recent years, I’ve been thinking in the past few weeks about getting back into writing more regularly, which has lead me to reflect on my writing in the past and my goals for this in the future.

First, the blog as a genre has changed fundamentally in the last 17 or 18 years. In 2000 or 2001, blogs were independent things that grew out of communities (e.g. MetaFilter, or web-diary/journaling) and were maintained by independent writers or small groups. Then the tooling got better, the community got better and eventually started to segment based on topic, and finally the press³ gained competence in the form.

Publishing a blog today, is a vastly different proposition today even in the recent past.

True to my form, this leaves me with a collection of divergent thoughts:

• maybe the “write everything in one blog even though the topics are not really of interest to any specific group” approach that I’ve always taken. More distinct blogs means more writing (maybe a good thing,)
• having a writing practice is good for focusing thoughts, but also for sharing and distributing understating, and I think that sharing understanding is a really important part of learning and growing, and I miss having a structure for these kinds of notes.
• perhaps, it would make sense to outsource/hire a freelancer to take care of some editing and marketing-adjacent work, which is more required if you want to engage with users more consistently but that I find distracting and outside of my ability to focus on properly. The problem then is figuring out how to fund that work in a longer-term/sustainable way.
• In the past RSS has been a (the?) leading way to distribute content to serious readers, but that isn’t true now and likly hasn’t been true for years. So while I feel able to write a lot of things, I don’t know what the best way to engage with regular readers is
• I used to think that I wanted to organize group blogs, and while I think that blog-discussions are fun, and I think there is merit to combined efforts, I’m less interested in doing the organizing myself.
• There was a period where I wasn’t blogging much because my day job was very writing focused, and I needed side projects that didn’t involve the English language, and I spent a long time focusing on learning programming, which took a lot of time. Now that work mostly involves programming, and only a little writing, and I’ve had some time to recover as a writer, it feels like I have some space.
• I’m not really sure how to host a blog in 2018. The old set up and server I have is more than functional, but there are a lot of services, tools, and patterns that I’m not familiar with and I have some learning to do, even though I probably mostly just want to write things.

Almost two years ago, I switched teams at work to join the team behind evergreen which is a homegrown continuous integration tool that we use organization wide to host and support development efforts and operational automation. It’s been great.

From the high level, Evergreen takes changes that developers make to source code repositories and runs a set of tasks for each of those changes on a wide variety of systems, and is a key part of the system that allows us to verify that the software we write works on computers other than the ones that we interact with directly. There are a number of CI systems in the world, but Evergreen has a number of interesting features:

• it runs tasks in parallel, fanning out tasks to a large pool of machines to shorten the “wall clock” time for task execution.
• tasks execute on ephemeral systems managed by Evergreen in response to demands of incoming work.
• the service maintains a queue of work and handles task dispatching and results collection.

This focus on larger scale task parallelism and managing host pools gives Evergreen the ability to address larger scale continuous integration workflows with a lower maintenance overhead. This is totally my jam: we get to both affect the development workflow and engineering policies for basically everyone and improving operational efficiency is a leading goal.

My previous gig was more operational, on a sibling team, so it’s been really powerful to be able to address problems relating to application scale and drive the architecture from the other side. I wrote a blog post for a work-adjacent outlet about the features and improvements, but this is my blog, and I think it’d be fun to have some space to explore “what I’ve been working on,” rather than focusing on Evergren as a product.

My first order of business, after becoming familiar with the code base, was to work on logging. When I started learning Go, I wrote a logging library (I even bloged about it), and using this library has allowed us to “get serious about logging.” While it was a long play, we now have highly structured logging which has allowed the entire logging system to become a centerpiece in our observably story, and we’ve been able to use centralized log aggregation services (and even shop around!) As our deployment grows, centralized logging is the thing that has kept everything together.

Recently, I’ve been focusing on how the application handles “offline” or background work. Historically the application has had a number of loosely coupled “cron-job” like operations that all happened on single machine at a regular interval. I’m focusing on how to move these systems into more tightly coupled, event-driven operations that can be distributed to a larger cluster of machines. Amboy is a big part of this, but there have been other changes related to this project.

On the horizon, I’m also starting to think about how to reduce the cost of exposing data and operations to clients and users in a way that’s lightweight and flexible, and relatively inexpensive for developer time. Right now there’s a lot of technical debt, a myriad of different ways to describe interfaces, and inconsistent client coverage. Nothing insurmountable, but definitely the next frontier of growing pains.

The theme here is “how do we take an application that works and does something really cool,” and turn it into a robust piece of software that can both scale as needs grown, but also provide a platform for developing new features with increasing levels of confidence, stability, and speed.

The conventional wisdom is that it’s easy to build features fast-and-loose without a bunch of infrastructure, and that as you scale the speed of feature development slows down. I’m pretty convinced that this is untrue and am excited to explore the ways that improved common infrastructure can reduce the impact of this ossification and lead to more nimble and expansive feature development.

We’ll see how it goes! And I hope to be able to find the time to write about it more here.