Running Emacs

OK, this is a weird post, but after reading a post about running emacs with systemd, [1] I've realized that the my take on how I run and manage processes is a bit unique, and worth enumerating and describing. The short version is that I regularly run multiple instances of emacs, under systemd, in daemon mode, and it's pretty swell. Here's the rundown:

  • On Linux systems, I use a build of emacs with the lucid toolkit, rather than GTK, because of this bug with GTK, which I should write more about at some point. Basically, if the X session crashes with GTK emacs, even if you don't have windows open, the daemon will crash, even if the dameon isn't started from a GUI session. The lucid toolkit doesn't have this problem.

  • I run the process under my user's systemd instance, rather than under the PID 1 systemd instance. I like keeping things separate. Run the following command to ensure that your user's systemd will start at boot rather than at login:

    sudo loginctl enable-ligner $(whoami)
  • I have a systemd service file named emacs@.service in my ~/.config/systemd/user/ directory that looks like this:

    Description=Emacs-tychoish: the extensible, self-documenting text editor.
    ExecStart=/usr/bin/emacs --daemon=%i --chdir %h
    ExecStop=/usr/bin/emacsclient --server-file=hud --eval "(progn (setq kill-emacs-hook 'nil) (kill-emacs))"

    I then start emacs dameons:

    systemctl --user start emacs@work
    systemctl --user start emacs@personal
    systemctl --user start emacs@chat

    To enable them so that they start following boot:

      systemctl --user enable emacs@work
      systemctl --user enable emacs@personal
      systemctl --user enable emacs@chat
    Though to be honest, I use different names for daemons.
  • I have some amount of daemon specific code, which might be useful:

    (setq server-use-tcp t)
    (if (equal (daemonp) nil)
        (setq tychoish-emacs-identifier "solo")
      (setq tychoish-emacs-identifier (daemonp)))
    ;; this makes erc configs work less nosily. There's probably no harm in
    ;; turning down the logging
    (if (equal (daemonp) "chat")
        (setq gnutls-log-level 0)
      (setq gnutls-log-level 1))
    (let ((csname (if (eq (daemonp) nil)
      (setq recentf-save-file (concat user-emacs-directory system-name "-" csname "-recentf"))
      (setq session-save-file (concat user-emacs-directory system-name "-" csname "-session"))
      (setq bookmark-default-file (concat user-emacs-directory system-name "-" csname "-bookmarks"))
      (setq helm-c-adaptive-history-file (concat user-emacs-directory system-name "-" csname "--helm-c-adaptive-history"))
      (setq desktop-base-file-name (concat system-name "-" csname "-desktop-file"))
      (setq desktop-base-lock-name (concat system-name "-" csname "-desktop-lock")))

    Basically this just sets up some session-specific information to be saved to different files, to avoid colliding per-instance.

  • Additionally, I use the tychoish-emacs-identifier from above to provide some contextual information as to what emacs daemon/window I'm currently in:

    (setq frame-title-format '(:eval (concat tychoish-emacs-identifier ":" (buffer-name))))
    (spaceline-emacs-theme 'daemon 'word-count)
    (spaceline-define-segment daemon tychoish-emacs-identifier)

    Also, on the topic of configuration, I do have a switch statement that loads different mu4e configurations in different daemons.

  • To start emacs sessions, I use operations in the following forms:

    # create a new emacs frame. Filename optional.
    emacsclient --server-file=<name> --create-frame --no-wait <filename>
    # open a file in an existing (last-focus) frame/window. Filename required.
    emacsclient --server-file=<name> --no-wait <filename>
    # open a terminal emacs mode. Filename optional .
    emacsclient --server-file=<name> --tty --no-wait <filename>

    That's a lot to type, so I use aliases in my shell profile:

    alias e='emacsclient --server-file=personal --no-wait'
    alias ew='emacsclient --server-file=personal --create-frame --no-wait'
    alias et='emacsclient --server-file=personal --tty'

    I create a set of aliases for each daemon prefixing e/ew/et with the first letter of the daemon name.

And that's about it. When I've used OS X, I've managed something similar using launchd but the configuration files are a bit less elegant. On OS X, I tend to install emacs with cocoa toolkit, using homebrew.

Using multiple daemons is cool, though not required, for a number of reasons:

  • you can have good separation between personal things and professional/work things, which is always nice, but particularly gratifying during the pandemic when it's easy to work forever.
  • Managing multiple separation of email. While mu4e has profiles and contexts, and that's great, I like a firmer boundary, and being able maintain separate email databases.
  • Running emacs lisp applications that do a lot of networking, or do other blocking operations. The main case where this matters in my experience is running big erc instance, or something else that isn't easily broken into async/subprocesses.
[1]And commenting!

Emacs and LSP Mode

I've been using emacs for a long time, and it's mostly pretty unexceptional. A few months ago, I started using LSP mode, and it's been pretty great... mostly. This post reflects on some of the dustier corners, and some of the broader implications.

History and Conflict

Language Server is a cool idea, that (as far as I know) grew out of VS Code, and provides a way to give a text editor "advanced, IDE-like" features without having to implement those features per-language in the text editor. Instead, editor developers implement generic integration for the protocol, and language developers produce a single process that can process source code and supports the editor features. Sounds great!

There are challenges:

  • The LSP protocol uses JSON for encoding data between the server and emacs. This is a reasonable choice, and makes development easy, but it's not a perfect fit. Primarily, in emacs, until version 27, which hasn't reached an official release yet (but is quite usable on Linux), parsed JSON in lisp rather than C, which made everything rather slow, but emacs 27 helps a lot. Second,

  • There is some overhead for the lsp, particularly as you attempt to scale very large numbers of files or very large files. Emacs is really good with large files, as an artifact of having had to support editing larger files on computers 20 or 30 years ago, and lsp feels sluggish in these situations.

    I think some of this overhead is probably depends on the implementation of the language server's themselves, and some percentage is probably protocol itself: http's statelessness is great in big distributed web services, and for the local use cases where responsiveness matters, it's less good. Similarly JSON's transparency and ubiquity is great, but for streaming streaming-type use cases, perhaps less so.

I've not poked around the protocol, or any implementations, extensively, so I'm very prepared to be wrong about these points.

There's also some tension about moving features, like support for adding editor support for a language, out of lisp and into an external process. While I'm broadly sympathetic, and think there are a bunch of high-quality language integrations in emacs lisp, and for basic things, the emacs lisp ones might be faster. Having said that, there's a lot of interface inconsistency between different support for languages in emacs, and LSP support feels spiffy my comparisons. Also at least for python (jedi,) go (gocode), and C/C++ (irony,) lisp (slime,) the non-lsp services were already using external processes for some portion of their functionality.

It's also the case that if people spend their emacs-lisp-writing time doing things other than writing support for various development environments it's all for the better anyway.

Suggestions / Notes

  • things will break or get wedged, call lsp-restart-workspace to restart the server (I think.) It helps.
  • use lsp-file-watch-ignored to avoid having the language servers process emacs process files from vendored code, intermediate/generated code, or similar, in cases where you might have a large number of files that aren't actually material to your development work
  • I believe some (many? all?) servers are sourced from the PATH (e.g. clang/llvm, gopls), which means you are responsible for updating them and their dependencies.
  • I've found it useful to update lsp-mode itself more often than I typically update other emacs packages. I also try and keep the language servers as up to date as possible in coordination with lsp-mode updates.
  • If Emacs 27 can work on your system, it's noticeably faster. Do that.
  • If you have an existing setup for a language environment, the LSP features end up layering over existing functionality, and that can have unexpected results.

My Configuration

None of this is particularly exciting, but if you use use-package, then the following might be an interesting starting point. I stole a lot of this from other places, so shrug.

I must say that I don't really use dap or treemacs, because I tend to keep things pretty barebones.

(use-package helm-lsp
  :ensure t
  :after (lsp-mode)
  :commands (helm-lsp-workspace-symbol)
  :init (define-key lsp-mode-map [remap xref-find-apropos] #'helm-lsp-workspace-symbol))

(use-package lsp-mode
  :diminish (lsp-mode . "lsp")
  :bind (:map lsp-mode-map
    ("C-c C-d" . lsp-describe-thing-at-point))
  :hook ((python-mode . #'lsp-deferred)
    (js-mode . #'lsp-deferred)
    (go-mode-hook . #'lsp-deferred))
  (setq lsp-auto-guess-root t       ; Detect project root
   lsp-log-io nil
   lsp-enable-indentation t
   lsp-enable-imenu t
   lsp-keymap-prefix "C-l"
   lsp-file-watch-threshold 500
   lsp-prefer-flymake nil)      ; Use lsp-ui and flycheck

  (defun lsp-on-save-operation ()
    (when (or (boundp 'lsp-mode)
         (bound-p 'lsp-deferred))

(use-package lsp-clients
  :ensure nil
  :after (lsp-mode)
  :init (setq lsp-clients-python-library-directories '("/usr/local/" "/usr/")))

(use-package lsp-ui
  :ensure t
  :after (lsp-mode)
  :commands lsp-ui-doc-hide
  :bind (:map lsp-ui-mode-map
         ([remap xref-find-definitions] . lsp-ui-peek-find-definitions)
         ([remap xref-find-references] . lsp-ui-peek-find-references)
         ("C-c u" . lsp-ui-imenu))
  :init (setq lsp-ui-doc-enable t
         lsp-ui-doc-use-webkit nil
         lsp-ui-doc-header nil
         lsp-ui-doc-delay 0.2
         lsp-ui-doc-include-signature t
         lsp-ui-doc-alignment 'at-point
         lsp-ui-doc-use-childframe nil
         lsp-ui-doc-border (face-foreground 'default)
         lsp-ui-peek-enable t
         lsp-ui-peek-show-directory t
         lsp-ui-sideline-update-mode 'line
         lsp-ui-sideline-enable t
         lsp-ui-sideline-show-code-actions t
         lsp-ui-sideline-show-hover nil
         lsp-ui-sideline-ignore-duplicate t)
  (add-to-list 'lsp-ui-doc-frame-parameters '(right-fringe . 8))

  ;; `C-g'to close doc
  (advice-add #'keyboard-quit :before #'lsp-ui-doc-hide)

  ;; Reset `lsp-ui-doc-background' after loading theme
  (add-hook 'after-load-theme-hook
       (lambda ()
         (setq lsp-ui-doc-border (face-foreground 'default))
         (set-face-background 'lsp-ui-doc-background
                              (face-background 'tooltip))))

  ;; WORKAROUND Hide mode-line of the lsp-ui-imenu buffer
  ;; @see
  (defadvice lsp-ui-imenu (after hide-lsp-ui-imenu-mode-line activate)
    (setq mode-line-format nil)))

;; Debug
(use-package dap-mode
  :diminish dap-mode
  :ensure t
  :after (lsp-mode)
  :functions dap-hydra/nil
  :bind (:map lsp-mode-map
         ("<f5>" . dap-debug)
         ("M-<f5>" . dap-hydra))
  :hook ((dap-mode . dap-ui-mode)
    (dap-session-created . (lambda (&_rest) (dap-hydra)))
    (dap-terminated . (lambda (&_rest) (dap-hydra/nil)))))

(use-package lsp-treemacs
  :after (lsp-mode treemacs)
  :ensure t
  :commands lsp-treemacs-errors-list
  :bind (:map lsp-mode-map
         ("M-9" . lsp-treemacs-errors-list)))

(use-package treemacs
  :ensure t
  :commands (treemacs)
  :after (lsp-mode))

Common Lisp Grip, Project Updates, and Progress

Last week, I did a release, I guess, of cl-grip which is a logging library that I wrote after reflecting on common lisp logging earlier. I wanted to write up some notes about it that aren't covered in the read me, and also talk a little bit4 about what else I'm working on.


This is a really fun and useful project and it was really the right size for a project for me to really get into, and practice a bunch of different areas (packages! threads! testing!) and I think it's useful to boot. The read me is pretty comprehensive, but I thought I'd collect some additional color here:

Really at it's core cl-grip isn't a logging library, it's just a collection of interfaces that make it easy to write logging and messaging tools, which is a really cool basis for an idea, (I've been working on and with a similar system in Go for years.)

As result, there's interfaces and plumbing for doing most logging related things, but no actual implementations. I was very excited to leave out the "log rotation handling feature," which feels like an anachronism at this point, though it'd be easy enough to add that kind of handler in if needed. Although I'm going to let it stew for a little while, I'm excited to expand upon it in the future:

  • additional message types, including capturing stack frames for debugging, or system information for monitoring.
  • being able to connect and send messages directly to likely targets, including systemd's journal and splunk collectors.
  • a collection of more absurd output targets to cover "alerting" type workloads, like desktop notifications, SUMP, and Slack targets.

I'm also excited to see if other people are interested in using it. I've submitted it to Quicklisp and Ultralisp, so give it a whirl!

See the cl-grip repo on github.


At the behest of a friend I've been working on an "egg equilibrium" solver, the idea being to provide a tool that can given a bunch of recipes that use partial eggs (yolks and whites) can provide optimal solutions that use a fixed set of eggs.

So far I've implemented some prototypes that given a number of egg parts, attempt collects recipes until there are no partial eggs in use, so that there are no leftovers. I've also implemented the "if I have these partial eggs, what can I make to use them all." I've also implemented a rudimentary CLI interface (that was a trip!) and a really simple interface to recipe data (both parsing from a CSV format and an in memory format that makes solving the equilibrium problem easier.)

I'm using it as an opportunity to learn different things, and find a way to learn more about things I've not yet touched in lisp (or anywhere really,) so I'm thinking about:

  • building a web-based interface using some combination of caveman, parenscript, and related tools. This could include features like "user submitted databases," as well as links to the sources the recpies, in addition to the basic "web forms, APIs, and table rendering."
  • storing the data in a database (probably SQLite, mostly) both to support persistence and other more advanced features, but also because I've not done database things from Lisp at all.

See the eggquilibrium repo on github it's still pretty rough, but perhaps it'll be interesting!'

Other Projects

  • Writing more! I'm trying to be less obsessive about blogging, as I think it's useful (and perhaps interesting for you all too.) I've been writing a bunch and not posting very much of it. My goal is to mix sort of grandiose musing on technology and engineering, with discussions of Lisp, Emacs, and programming projects.
  • Working on producing texinfo output from cl-docutils! I've been toying around with the idea of writing a publication system targeted at producing books--long-form non-fiction, collections of essays, and fiction--rather than the blogs or technical resources that most such tools are focused on. This is sort of part 0 of this process.
  • Hacking on some Common Lisp projects, I'm particularly interested in the Nyxt and StumpWM.

Common Lisp and Logging

I've made the decision to make all of personal project code that I write to do in Common Lisp. See this post for some of the background for this decision.

It didn't take me long to say "I think I need a logging package," and I quickly found this wonderful comparsion of CL logging libraries, and only a little longer to be somewhat disappointed.

In general, my requirements for a logger are:

  • straightforward API for logging.
  • levels for filtering messages by importance
  • library in common use and commonly available.
  • easy to configure output targets (e.g. system's journal, external services, etc).
  • support for structured logging.

I think my rationale is pretty clear: loggers should be easy to use because the more information that can flow through the logger, the better. Assigning a level to all log messages is great for filtering practically, and it's ubiquitous enough that it's really part of having a good API. While I'm not opposed to writing my own logging system, [1] but I think I'd rather not in this case: there's too much that's gained by using the conventional choice.

Configurable outputs and structured logging are stretch goals, but frankly are the most powerful features you can add to a logger. Lots of logging work is spent crafting well formed logging strings, when really, you just want some kind of arbitrary map and makes it easier to make use of logging at scale, which is to say, when you're running higher workloads and multiple coppies of an application.


I've dug in a bit to a couple of loggers, sort of using the framework above to evaluate the state of the existing tools. Here are some notes:


My analysis of the CL logging packages is basically that log4cl is the most mature and actively maintained tool, but beyond the basic fundamentals, it's "selling" features are... interesting. [2] The big selling features:

  • integration with the developers IDE (slime,) which makes it possible to use the logging system like a debugger, almost. This is actually a phenomenal feature, particularly for development and debugging. The downside is that it wouldn't be totally unreasonable to use it production, and that's sort of terrifying.
  • it attempts to capture a lot of information about logging call sites so you can better map back from log messages to the state of the system when the call was made. Again, this makes it a debugging tool, and that's awesome, but it's overhead, and frankly I've never found it difficult to grep through code.
  • lots of attention to log rotation and log file management. There's not a lot of utility in writing log data to files directly. In most cases you want to write to standard out: the program is being used interactively, and users may want to see the log of what happens. In cases where you're running in a daemon mode, any more you're not, systemd or similar just captures your output. Even then, you're probably in a situation where you want to send the log output to some other process (e.g. an external service, or some kind of local socket for fluentd/etc.)
  • hierarchical organization of log messages is just less useful than annotation with metadata, in practice, and using hierarchical methods to filter logs into different streams or reduce logging volumes just obscures things and makes it harder to understand what's happening in a system.

Having said that, the API surface area is big, and it's a bit confusing to just start using the logger.


The acl package is pretty straightforward, and has a lot of features that I think are consistent with my interests and desires:

  • support for JSON output,
  • internal support for additional output formats (e.g. logstash,)
  • more simple API

It comes with a couple of pretty strong draw backs:

  • there are limited testing.
  • it's SBCL only, because it relies on SBCL fundamentals in collecting extra context about log messages. There's a pending pull request to add ECL compile support, but it looks like it wouldn't be quite that simple.
  • the overhead of collecting contextual information comes at an overhead, and I think logging should err on the side of higher performance, and making expensive things optional, just because it's hard to opt into/out of logging later.


So where does that leave me?

I'm not really sure.

I created a scratch project to write a simple logging project, but I'm definitely not prioritizing working on that over other projects. In the mean time I'll probably end up just not logging very much, or maybe giving log4cl a spin.


[1]When I started writing Go I did this, I wrote a logging tool, for a bunch of reasons. While I think it was the right decision at the time, I'm not sure that it holds up. Using novel infrastructure in projects makes integration a bit more complicated and creates overhead for would be contributors.
[2]To be honest, I think that log4cl is a fine package, and really a product of an earlier era, and it makes the standard assumptions about the way that logs were used, that makes sense given a very different view of how services should be operated.

A Common Failure

I've been intermittently working on a common lisp library to produce a binary encoding of arbitrary objects, and I think I'm going to be abandoning the project. This is an explanation of that decision and an reflection on my experience.

Why Common Lisp?

First, some background. I've always thought that Common Lisp is a language with a bunch of cool features and selling points, but unsurprisingly, I've never really had the experience of writing more than some one-off bits of code in CL, which isn't surprising. This project was a good experience for really digging into writing and managing a conceptually larger project which was a good kick in the pants to learn more.

The things I like:

  • the implementations of the core runtime are really robust and high quality, and make it possible to imagine running your code in a bunch of different contexts. Even though it's a language with relatively few users, it feels robust in a way. The most common implementations also have ways of producing fully self contained static binaries (like Go, say), which makes the thought of distributing software seem reasonable.
  • quicklisp, a package/library management tool is new (in the last decade or so,) has really raised the level of the ecosystem. It's not as complete as I'd expect in many ways, but quicklisp changed CL from something quaint to something that you could actually imagine using.
  • the object system is really nice. There isn't quite compile time-type checking on the values of slots (attributes) of objects, though you can opt in. My general feeling is that I can pretty easily get the feeling of writing statically typed code with all of the freedom of writing dynamic code.
  • multiple dispatch, and the conceptual approach to genericism, is amazing and really simplifies flow control in a lot of cases. You implement the methods you need, for the appropriate types/objects and then just write the logic you need, and the function call machinery just does the right thing. There's surprisingly little conditional logic, as a result.

Things I don't like:

  • there are all sorts of things that don't quite have existing libraries, and so I find myself wanting to do things that require more effort than necessary. This project to write a binary encoding tool would have been a component in service of a much larger project. It'd be nice if you could skip some of the lower level components, or didn't have your design choices so constrained by gaps in infrastructure.
  • at the same time, the library ecosystem is pretty fractured and there are common tools around which there aren't really consensus. Why are there so many half-finished YAML and JSON libraries? There are a bunch of HTTP server (!) implementations, but really you need 2 and not 5?
  • looping/iteration isn't intuitive and difficult to get common patterns to work. The answer, in most cases is to use (map) with lambdas rather than loops, in most cases, but there's this pitfall where you try and use a (loop) and really, that's rarely the right answer.
  • implicit returns seem like an over sight, hilariously, Rust also makes this error. Implicit returns also make knowing what type a function or method returns quite hard to reason about.

Writing an Encoder

So the project I wrote was an attempt to write really object oriented code as a way to writing an object encoder to a JSON-like format. Simple enough, I had a good mental model of the format, and my general approach to doing any kind of binary format processing is to just write a crap ton of unit tests and work somewhat iteratively.

I had a lot of fun with the project, and it gave me a bunch of key experiences which make me feel comfortable saying that I'm able to write common lisp even if it's not a language that I feel maximally comfortable in (yet?). The experiences that really helped included:

  • producing enough code to really have to think about how packaging and code organization worked. I'd written a function here and there, before, but never something where I needed to really understand and use the library/module/packaging (e.g. systems and libraries.) infrastructure.
  • writing and running lots of tests. I don't always follow test-driven development closely, but writing lots of tests is part of my process, and being able to watch the layers of this code come together was a lot of fun and very instructive.
  • this project for me, was mostly about API design and it was nice to have a project that didn't require much design in terms of the actual functionality, as object encoding is pretty straight forward.

From an educational perspective all of my goals were achieved.

Failure Mode

The problem is that it didn't work out, in the final analysis. While the library I constructed was able to encode and decode objects and was internally correct, I never got it to produce encoding that other implementations of the same specification could reliably read, and the ability to read data encoded by other libraries only worked in trivial cases. In the end:

  • this was mostly a project designed to help me gain competence in a programming language I don't really know, and in that I was successful.
  • adding this encoding format isn't particularly useful to any project I'm thinking of working on in the short term, and doesn't directly enable me to do anything in particular.
  • the architecture of the library would not be particularly performant in practice, as the encoding process didn't deploy a buffer pool of any kind, and it would have been harder than not to back fill that in, and I wasn't particularly interested in that.
  • it didn't work, and the effort to debug the issue would be more substantive than I'm really interested in doing at this point, particularly given the limited utility.

So here we are. Onto a different project!

There's No Full Stack

Software engineers use terms like "backend" and "frontend" to describe areas of expertice and focus, and the thought is that these terms map roughly onto "underlying systems and business logic" and "user interfaces." The thought, is that these are different kinds of work and no person can really specialize on "everything."

But it's all about perspective. Software is build in layers: there are frontends and backends at almost every level, so the classification easily breaks down if you look at it too hard. It's also the case that that logical features, from the perspective of the product and user, require the efforts of both disciplines. Often development organizations struggle to hand projects off between groups of front-end and back-end teams. [1]

[1]In truth this problem of coordination between frontend and backend teams is really that it forces a waterfall-like coordination between teams, which is always awkward. The problem isn't that backend engineers can't write frontend code, but that having different teams requires a handoff that is difficult to manage correctly, and around that handoff processes and management happens.

Backend/Frontend is also a poor way to organize work, as often it forces a needless boundary between people and teams wokring on related projects. Backend work (ususally) has to be completed first, and if that slips (or estimation is off) then the front end work has to happen in a crunch. Even if timing goes well, it's difficult to maintain engineering continuity through the handoff and context is often lost in the process.

In response to splitting projects and teams into front and backend, engineers have developed this idea of "full stack" engineering. This typically means "integrated front end and backend development." A noble approach: keep the same engineer on the project from start to finish, and avoid an awkward handoff or resetting context halfway through a project. Historic concerns about "front end and backend being in different languages" are reduced both by the advent of back-end javascript, and a realization that programmers often work in multiple languages.

While full stack sounds great, it's a total lie. First, engineers by and large cannot maintain context on all aspects of a system, so boundaries end up appearing in different places. A full stack engineer might end up writing front end and the APIs on the backed that the front end depends on, but not the application logic that supports the feature. Or an engineer might focus only a very specific set of features, but not be able to branch out very broadly. Second, specialization is important for allowing engineers to focus and be productive, and while context switching projects between engineers, having engineers that must context switch regularly between different disciplines is bad for those engineers. In short you can't just declare that engineers will be able to do it all.'

Some, particularly larger, teams and prodcuts can get around the issue entirely by dividing ownership and specialization along functional boundaries rather than by engineering discipline, but there can be real technical limitations, and getting a team to move to this kind of ownership model is super difficult. Therefore, I'd propose a different organization or a way of dividing projects and engineering that avoids both "frontend/backend" as well as the idea of "full stack":

  • feature or product engineers, that focus on core functionality delivered to users. This includes UI, supporting backend APIs, and core functionality. The users of these teams are the users of the product. These jobs have the best parts of "full stack" type orientation, but draw an effective "lower" boundary of responsibility and allow feature-based specialization.
  • infrastructure or product platform engineers, that focus on deployment, operations and supporting internal APIs. These teams and engineers should see their users as feature and product engineers. These engineers should fall somewhere between "backend engineers," and the "devops" and "sre" -type roles of the last decade, and cover the area "above" systems (e.g. not inclusive of machine management and access provisioning,) and below features.

This framework helps teams scale up as needs and requirements change: Feature teams can be divided and parallelized and focus in functionality slices, while, infrastructure teams divide easily into specialties (e.g. networking, storage, databases, internal libraries, queues, etc.) and along service boundaries. Teams are in a better position to handle continuity of projects, and engineers can maintain context and operate using more agile methods. I suspect that, if we look carefully, many organizations and teams have this kind of de facto organization, even if they use different kind of terminology.


What is it That You Do?

The longer that I have this job, the more difficult it is to explain what I do. I say, "I'm a programmer," and you'd think that I write code all day, but that doesn't map onto what my days look like, and the longer it seems the less code I actually end up writing. I think the complexity of this seemingly simple question grows from the fact that building software involves a lot more than writing code, particularly as projects become more complex.

I'd venture to say that most code is written and maintained by one person, and typically used by a very small number of pepole (often on behalf of many more people,) though this is difficult to quantify. Single maintainer software is still software, and there are lots of interesting problems, but as much as anything else I'm interested in the problems adjacent to multi-author code-bases and multi-operator software development. [1]

Fundamentally, I'm interested in the following questions:

  • How can (sometimes larger) groups of people collaborate to build something that's bigger than the scope of any of their work?
  • How can we build software in a way that lets individual developers focus most of the time on the features and concerns that are the most important to them and their users. [2]

The software development process, regardless of the scope of the problem, has a number of different aspects:

  • Operations: How does is this software execute and how do we know that its successful when it runs?
  • Behavior: What does it do, and how do we ensure it has the correct behavior?
  • Interface: How will users interact with the process, and how do we ensure a consistent experience across versions and users' environment?
  • Product: Who are the users? What features do they want? Which features are the most important?

Sometimes we can address these questions by writing code, but often there's a lot of talking to users, other developers, and other people who work in software development organizations (e.g. product managers, support, etc.) not to mention writing a lot of English (documentation, specs, and the like.)

I still don't think that I've successfully answered the framing question, except to paint a large picture of what kinds of work goes into making software, and described some of my specific domain interests. This ends up boiling down to:

  • I write a lot of documents describing new features and improvements to our software. [product]
  • I think a lot about how our product works as it grows (scaling), and what kinds of changes we can make now to make that process more smooth. [operations]
  • How can I help the more junior members of my team focus on the aspects of their jobs that they enjoy the most, and help illustrate broader contexts to them. [mentoring]
  • How can we take the problems we're solving today and build the solution that balances the immediate requirements with longer term maintainability and reuse. [operations/infrastructure]

The actual "what" I'm spending my time boils down to reading a bunch of code, meeting with my teamates, meeting with users (who are also coworkers.) And sometimes writing code. If I'm lucky.

[1]I think the single-author and/or single-operator class is super interesting and valuable, particularly because it includes a lot of software outside of the conventional disciplinary boundaries of software and includes things like macros, spreadsheets, small scale database, and IT/operations ("scripting") work.
[2]It's very easy to spend most of your time as a developer writing infrastructure code of some sort, to address either internal concerns (logging, data management and modeling, integrating with services) or project/process automation (build, test, operations) concerns. Infrastructure isn't bad, but it isn't the same as working on product features.

Non-Trad Software Engineer

It happened gradually, and it wasn't entirely an intentional thing, but at some point I became a software engineer. While a lot of people become software engineers, many of them have formal backgrounds in engineering, or have taken classes or done programs to support this retooling (e.g. bootcamps or programming institutes.)

I skipped that part.

I wrote scripts from time to time for myself, because there were things I wanted to automate. Then I was working as a technical writer and had to read code that other people had written for my job. Somewhere in there I was responsible for managing the publication workflow, and write a couple of build systems.

And then it happened.

I don't think it's the kind of thing that is right for everyone, but I was your typical, nerdy/bookish kid who wasn't great in math class, and I suspect that making software is the kind of thing that a lot of people could do. I don't think that my experience is particularly replicable, but I have learned a number of useful (and important) things, and I realize as I've started writing more about what I'm working on now, I realize that I've missed some of the fundamentals [1]

Formal education in programming, from what I've been able to gather strikes me as really weird: there are sort of two main ways of teaching people about software and computer science: Option one is that you start with a very theoretical background that focuses on data structures, the performance of algorithms, or the internals of how core technologies function (operating systems, compilers, databases, etc.) Option two, is that you spend a lot of time learning about (a) programming language and about how to solve problems using programming.

The first is difficult, because the theory [2] is not particularly applicable except invery rare cases and only at the highest level which is easy to back-fill as needed. The second is also challenging, as idioms change between languages and most generic programming tasks are easily delegated to libraries. The crucial skill for programming is the ability to learn new languages and solve problems in the context of existing systems, and developing a curriculum to build those skills is hard.

The topics that I'd like to write about include:

  • Queue behavior, particularly in the context of distributed systems.
  • Observability/Monitoring and Logging, particularly for reasonable operations at scale.
  • build systems and build automation.
  • unit-testing, test automation, and continuous integration.
  • interface design for users and other programmers.
  • maintaining and improving legacy systems.

These are, of course, primarily focused on the project of making software rather than computer science or computing in the abstract. I'm particularly interested (practically) in figuring out what kinds of experiences and patterns are important for new programmers to learn, regardless of background. [3] I hope you all find it interesting as well!

[1]This is, at least in part, because I mostly didn't blog very much during this process. Time being finite and all.
[2]In practice, theoretical insights come up pretty infrequently and are mostly useful for providing shorthand for characterizing a problem in more abstract terms. Most of the time, you're better off intuiting things anyway because programming is predominantly a pragmatic exercise. For the exceptions, there are a lot of nerds around (both at most companies and on the internet) who can figure out what the proper name is for a phenomena and then you can look on wikipedia.
[3]A significant portion of my day-to-day work recently has involved mentoring new programmers. Some have traditional backgrounds or formal technical education and many don't. While everyone has something to learn, I often find that because my own background is so atypical it can be hard for me to outline the things that I think are important, and to identify the high level concepts that are important from more specific sets of experiences.