New Beginnings: Deciduous Platform

I left my job at MongoDB (8.5 years!) at the beginning of the summer, and started a new job at the beginning of the month. I'll be writing and posting more about my new gig, career paths in general, reflections on what I accomplished on my old team, the process of interviewing as a software engineer, as well as the profession and industry over time. For now, though, I want to write about one of the things I've been working on this summer: making a bunch of the open source libraries that I worked on more generally useable. I've been calling this the deciduous platform, [2] which now has its own github organization! So it must be real.

The main modification in these forks, aside from adding a few features that had been on my list for a while, has been to update the buildsystem to use go modules [3] and rewrite the history of the repository to remove all of the old vendoring. I expect to continue development on some aspects of these over time, though the truth is that these libraries were quite stable and were nearly in maintenance mode anyway.

Background

The team was responsible for a big monolith (or so) application: development had begun in 2013, which was early for Go, and while everything worked, it was a bit weird. My efforts when I joined in 2015 focused mostly on stabilization, architecture, and reliability. While the application worked, mostly, it was clear that it suffered from a few problem, which I believe were the result of originating early in the history of Go: First, because no one had tried to write big applications yet, the patterns weren't well established, and so the team ended up writing code that worked but that was difficult to maintain, and ended up with bespoke solutions to a number of generic problems like running workloads in the background or managing Apia. Second, Go's standard library tends to be really solid, but also tends towards being a little low level for most day-to-day tasks, so things like logging and process management end up requiring more code [4] than is reasonable.

I taught myself to write Go by working on a logging library, and worked on a distributed queue library. One of the things that I realized early, was that breaking the application into "microservices," would have been both difficult and offered minimal benefit, [5] so I went with the approach of creating a well factored monolith, which included a lot of application specific work, but also building a small collection of libraries and internal services to provide useful abstractions and separations for application developers and projects.

This allowed for a certain level of focus, both for the team creating the infrastructure, but also for the application itself: the developers working on the application mostly focused on the kind of high level core business logic that you'd expect, while the infrastructure/platform team really focused on these libraries and various integration problems. The focus wasn't just organizational: the codebases became easier to maintain and features became easier to develop.

This experience has lead me to think that architecture decisions may not be well captured by the monolith/microservice dichotomy, but rather there's' this third option that centers on internal architecture, platforms, and the possibility for developer focus and velocity.

Platform Overview

While there are 13 or so repositories in the platform, really there are 4 major libraries: grip, a logging library; jasper, a process management framework; amboy, a (possibly distributed) worker queue; and gimlet, a collection of tools for building HTTP/REST services.

The tools all work pretty well together, and combine to provide an environment where you can focus on writing the business logic for your HTTP services and background tasks, with minimal boilerplate to get it all running. It's pretty swell, and makes it possible to spin up (or spin out) well factored services with similar internal architectures, and robust internal infrastructure.

I wanted to write a bit about each of the major components, addressing why I think these libraries are compelling and the kinds of features that I'm excited to add in the future.

Grip

Grip is a structured-logging friendly library, and is broadly similar to other third-party logging systems. There are two main underlying interfaces, representing logging targets (Sender) and messages, as well as a higher level "journal" interface for use during programming. It's pretty easy to write new message or bakcends, which means you can use grip to capture all kinds of arbitrary messages in consistent manners, and also send those messages wherever they're needed.

Internally, it's quite nice to be able to just send messages to specific log targets, using configuration within an application rather than needing to operationally manage log output. Operations folks shouldn't be stuck dealing with just managing logs, after all, and it's quite nice to just send data directly to Splunk or Sumologic. We also used the same grip fundamentals to send notifications and alerts to Slack channels, email lists, or even to create Jira Issues, minimizing the amount of clunky integration code.

There are some pretty cool projects in and around grip:

  • support for additional logging targets. The decudous version of grip adds twitter as an output format as well as creating desktop notifications (e.g. growl/libnotify,) but I think it would also be interesting to add fluent/logstash connections that don't have to transit via standard error.'
  • While structured logging is great, I noticed that we ended up logging messages automatically in the background as a method of metrics collection. It would be cool to be able to add some kind of "intercepting sender" that handled some of these structured metrics, and was able to expose this data in a format that the conventional tools these days (prometheus, others,) can handle. Some of this code would clearly need to be in Grip, and other aspects clearly fall into other tools/libraries.

Amboy

Amboy is an interface for doing things with queues. The interfaces are simple, and you have:

  • a queue that has some way of storing and dispatching jobs.
  • implementations of jobs which are responsible for executing your business logic, and with a base implemention that you can easily compose, into your job types, all you need to implement, really is a Run() method.
  • a queue "group" which provides a higher level abstraction on top of queues to support segregating workflows/queues in a single system to improve quality of service. Group queues function like other queues but can be automatically managed by the processes.
  • a runner/pool implementation that provides the actual thread pool.

There's a type registry for job implementations and versioning in the schema for jobs so that you can safely round-trip a job between machines and update the implementation safely without ensuring the queue is empty.

This turns out to be incredibly powerful for managing background and asynchronous work in applications. The package includes a number of in-memory queues for managing workloads in ephemeral utilities, as well as a distributed MongoDB backed-queue for running multiple copies of an application with a shared queue(s). There's also a layer of management tools for introspecting, managing, the state of jobs.

While Amboy is quite stable, there is a small collection of work that I'm interested in:

  • a queue implementation that store jobs to a local Badger database on-disk to provide single-system restartabilty for jobs.
  • a queue implementation that stores jobs in a PostgreSQL, mirroring the MongoDB job functionality, to be able to meet job backends.
  • queue implementations that use messaging systems (Kafka, AMPQ) for backends. There exists an SQS implementation, but all of these systems have less strict semantics for process restarts than the database options, and database can easily handle on the order of a hundred of thousand of jobs an hour.
  • changes to the queue API to remove a few legacy methods that return channels instead of iterators.
  • improve the semantics for closing a queue.

While Amboy has provisions for building architectures with workers running on multiple processes, rather than having queues running multiple threads within the same process, it would be interesting to develop more fully-fledged examples of this.

Jasper

Jasper provides a high level set of tools for managing subprocesses in Go, adding a highly ergonomic API (in Go,) as well as exposing process management as a service to facilitate running processes on remote machines. Jasper also manages/tracks the state of running processes, and can reduce pressures on calling code to track the state of processes.

The package currently exposes Jasper services over REST, gRPC, and MongoDB's wire protocol, and there is also code to support using SSH as a transport so that you don't need to expose remote these services publically.

Jasper is, perhaps, the most stable of the libraries, but I am interested in thinking about a couple of extensions:

  • using jasper as PID 1 within a container to be able to orchestrate workloads running on containers, and contain (some) support for lower level container orchestration.
  • write configuration file-based tools for using jasper to orchestrate buildsystems and distributed test orchestration.

I'm also interested in cleaning up some of the MongoDB-specific code (i.e. the code that downloads MongoDB versions for use in test harnesses,) and perhaps reinvisioning that as client code that uses Jasper rather than as a part of Jasper.

Gimlet

I've written about gimlet here before when I started the project, and it remains a pretty useful and ergonomic way to define and regester HTTP APIs, in the past few years, its grown to add more authentication features, as well as a new "framework" for defining routes. This makes it possible to define routes by implementing an interface that:

  • makes it very easy to produce paginated routes, and provides some helpers for managing content
  • separates the parsing of inputs from executing the results, which can make route definitions easy to test without integration tests.
  • rehome functionality on top of chi router. The current implementation uses Negroni and gorilla mux (but neither are exposed in the interface), but I think it'd be nice to have this be optional, and chi looks pretty nice.

Other Great Tools

The following libraries are defiantly smaller, but I think they're really cool:

  • birch is a builder for programatically building BSON documents, and MongoDB's extended JSON format. It's built upon an earlier version of the BSON library. While it's unlikely to be as fast at scale, for many operations (like finding a key in a document), the interface is great for constructing payloads.
  • ftdc provides a way to generate (and read,) MongoDB's diagnostic data format, which is a highly compressed timeseries data format. While this implementation could drift from the internal implementation over time, the format and tool remain useful for arbitrary timeseries data.
  • certdepot provides a way to manage a certificate authority with the certificates stored in a centralized store. I'd like to add other storage backends over time.

And more...

Notes

[1]Though, given my usual publication lag, I'm writing this a couple days before starting.
[2]My old team built a continuous integration tool called evergreen which is itself a pun (using "green" to indicate passing builds, most CI systems are not ever-green.) Many of the tools and libraries that we built had got names with tree puns, and somehow "deciduous" seems like the right plan.
[3]For an arcane reason, all of these tools had to build with an old version of Go (1.10) that didn't support modules, so we had an arcane and annoying vendoring solution that wasn't compatible with modules.
[4]Go tends to be a pretty verbose language, and I think most of the time this creates clarity; however, for common tasks it has the feeling of offering a poor abstraction, or forcing you to write duplicated code. While I don't believe that more-terse-code is better, I think there's a point where the extra verbosity for route operations just creates the possibility for more errors.
[5]The team was small, and as an internal tools team, unlikely to grow to the size where microservices offered any kind of engineering efficiency (at some cost,) and there weren't significant technical gains that we could take advantage of: the services of the application didn't need to be globally distributed and the boundaries between components didn't need to scale independently.

Tycho Emacs Config Kit

So I made a thing, or at least, I took a thing I've built and made it presentable for other people. I'm talking, of course, about my Emacs configuration.

Check it out at github.com/tychoish/.emacs.d! The README is pretty complete, and giving it a whirl is simple, just do something like:

mv ~/.emacs.d/ ~/emacs.d-archive
git clone --recurse-submodules git@github.com:tychoish/.emacs.d.git ~/.emacs.d/

If you're using Emacs 27, you might also be able to clone it into ~/.config/emacs, for similar effect. It doesn't matter much to me. Let me know what you think!


The tl;dr of "what's special about this config," is that, it has:

  • a good set of defaults out of the box.
  • a lot of great wiz-bang features enabled and configured.
  • very quick start times, thanks to lazy-loading of libraries.
  • great support for running as a daemon, and even running multiple daemons at once.

I've sporadically produced copies of my emacs config for other folks to use, but those were always ad hoc, and difficult to reproduce and maintain, and I've been working on and off to put more polish on things, so making it usable for other people seemed like a natural step at this point.

I hope other people find it useful, but also I think it's a good exercise for me in keeping things well maintained, and it doesn't bother me much one way or another.

Discussion

I think a lot about developer experience: I've spent my career, thus far, working on infrastructure products (databases, CI systems, build tools, release engineering,) that help improve developer experience and productivity, and even my day-to-day work tends to focus mostly on problems that impact the way that my teams write software: system architecture, service construction, observability, test infrastructure, and similar.

No matter how you slice it, editor experience is a huge factor in people's experience, and can have a big impact on productivity, and there are so many different editors with wildly different strengths. Editors experience is also really hard: unlike other kinds of developer infrastructure (like buildsystems, or even programming languages) the field conceptualizes editors as personal: your choice in editor is seen to reflect on you (it probably doesn't), and your ability to use your editor well is seen to be a reflection of your skills as a developer (it isn't). It's also the case that because editors are so personal, it's very difficult to produce an editor with broad appeal, and the most successful editors tend to be very configurable and can easily lack defaults, which means that even great editors are terrible out of the box, which mostly affects would-be and new developers.

Nevertheless, time being able to have an editor that you're comfortable with and that you can use effectively without friction does make it easier to build software, so even if folks often conceptualize of editors in the wrong way, improving the editing experience is important. I think that there are two areas for improvement:

  • editor configurations should--to some extent--be maintained at the project (or metaproject) level, rather than on the level individual engineer. The hard part here is how to balance individual preference with providing a consistent experience, particularly for new developers. [1]
  • there should be more "starter kits" (like this one! or the many other starter kits, but also for (neo)vim, vscode, and others.) that help bootstrap the experience, while also figuring out ways to allow layering other project-based extensions on top of a base configuration.

Also, I want to give chemacs a shout out, for folks who want to try out other base configurations.

[1]There are two kinds of new developers with different experiences but some overlap: folks who have development experience in general but no experience with a specific project, and folks who are new to all development.

Pickle Quarantine

What a year, am I right?

I've avoided writing a more "journal"-type post because there's both too much to say and somehow not enough all at the same time. This is a tough time to be a human, and I've been incredibly lucky in many regards: I can stay busy, I have a living situation that makes quarantining easy and tolerable, and while I wish the produce at my local grocery had better produce, it's nice to live in a very walkable and accessible neighborhood. Even though "nowtimes" are fundamentally different from the "beforetimes," the speed at which things become routine--if not normal--is almost disturbing. Delightfully, though, I've felt like I'm still getting things done. In no particular order:

  • As the title of the post suggests I have firmly entered the home pickling phase of quarantine, and have begun fermenting rather a lot of cauliflower, with additional vegetables to follow. Frankly I don't know why I didn't think of this earlier: I love pickles and it's really hard to get compelling pickles for things that aren't cucumbers. My first batch is cauliflower, and I'm also excited to do things like: baby okra, cabbage, greenbeans, and maybe some of my favorite root vegetables (shallots, raddishes, horseradish, jerusalem artichokes, etc.) Details forcoming, perhaps.
  • I've begun cooking more often. My go to, is to make a pot of chicken stock, and then figure out things to use the stock to cook. I've done some nice proto-beef stew things, and have made some pretty great lentil dishes. Nothing fancy or complex. I'm also delightfully, getting to a point where I can produce a meal that consists of more than one flavor profile, which is nice.
  • I've written enough common lisp that writing little bits of software in CL seem doable. It's still a bit harder to get into bigger projects, and I have a lot to learn still, but I've crossed a line, and it's nice to have an additional skill.
  • I replaced the video card in my home desktop, and I feel like I have a totally new computer! It's amazing. I got this computer in 2013, and have mostly been using laptops for the past few years. Before quarantine, I was even planning to de-accession it, but I've been using it a lot more and the new video card really changes things. Since I don't do a lot of graphics things and my computer use is not particularly performance intensive, I forget how much having an underpowered GPU can really impact perceptions: I was suffering through a lot of rendering-hangs and glitches that just don't happen any more.
  • I've been writing blog posts more, and am working on a plan to have something that resembles a publication schedule. I wrote professionally for so long, that writing for fun never seemed like a viable way to spend time, but as time goes on, it's clear that I really want to be able to experiment with ideas and reflect on things in a more regular way, and continuing to write posts here seems like a good way to do that. My lisp-related posts are now appearing on Planet Lisp, and I'm interested in expanding my reach... slightly.
  • I completed a sort of long term project to modernize the way that I use and configure emacs: I got all of the hardcoded weirdness undercontrol, I started using use-package, and I got lsp-mode to work, and I fixed a number of hilarious little bugs in how my config worked. The end result, is I get to use the editor I want in the way that I want, and it all starts up on my (old) computers in well under 2 seconds, not that I start up all that often. I expect I'll write a bit more on this soon.
  • Since quarantine started I've gone through an entire complete box of Chemex filters (plus the tail end of a box, and I'm a few pots into the next box.) I started drinking coffee during the summer of 2015, and have primarily consumed coffee produced by other people (i.e. at work) in that time. While I've definitely gone through a few boxes of Chemex filters over the years, my new rate is noticeable. But quarantine means making your own coffee, and apparently I've made a lot of coffee. It feels like a milestone.

Lazy Lentils

Alternate title: yolo lentils

I've been cooking more in the last few months that I have, basically ever. I'm not a bad cook, and my instincts and raw skills are pretty decent, but I'm not super inspired, I guess. Left to my own devices I eat the same few things again and again. So the exercise of mostly cooking for myself has been a lot about figuring out things to cook that taste good (and different!) and that I can reproduce reliably. Sometimes I'm more successful than other times, and frankly a lot of what I've come up with isn't super inspiring, but I recently stumbled upon something that was pretty great:

Ingredients:

  • 1 Cup of Black Lentils
  • 2 Cups of Chicken Stock
  • 1 Onion chopped reasonably finely.
  • 3 cloves of garlic sliced.
  • 1 inch piece of fresh ginger microplaned or minced.
  • (some) Olive Oil
  • Spices: Sweet Curry Powder, Garam Masala, Tumeric
  • Salt and Pepper

Procedure:

  • Saute garlic onion, and ginger in olive oil until soft and becoming translucent, adding spices near the end. (5 minutes.)
  • Add stock and lentils, and cook until lentils soften. I did the whole thing in an instant pot for 8-10 minutes and let the pressure release naturally (without keep warm). I'm sure other kinds of pots and cooking methods work well too.

That's it.

Modifications and notes:

  • I put the spices in at the end of the sauteing phase, and I might have put them in slightly earlier, but it's fine.
  • I like that black lentils retain their form after cooking, but if you like more mushy texture other lentils would be fine.
  • If you don't use a pressure cooker, it seems reasonable that more liquid would be useful.
  • I used chicken stock because it's delicious and I had just made a batch of right before I made the lentils so it seemed reasonable. The stock flavor is secondary to the spices.
  • I didn't measure the spices (which were not particularly fresh, I must confess,) but I listed them roughly in terms of amount.

Enjoy!

Running Emacs

OK, this is a weird post, but after reading a post about running emacs with systemd, [1] I've realized that the my take on how I run and manage processes is a bit unique, and worth enumerating and describing. The short version is that I regularly run multiple instances of emacs, under systemd, in daemon mode, and it's pretty swell. Here's the rundown:

  • On Linux systems, I use a build of emacs with the lucid toolkit, rather than GTK, because of this bug with GTK, which I should write more about at some point. Basically, if the X session crashes with GTK emacs, even if you don't have windows open, the daemon will crash, even if the dameon isn't started from a GUI session. The lucid toolkit doesn't have this problem.

  • I run the process under my user's systemd instance, rather than under the PID 1 systemd instance. I like keeping things separate. Run the following command to ensure that your user's systemd will start at boot rather than at login:

    sudo loginctl enable-ligner $(whoami)
    
  • I have a systemd service file named emacs@.service in my ~/.config/systemd/user/ directory that looks like this:

    [Unit]
    Description=Emacs-tychoish: the extensible, self-documenting text editor.
    
    [Service]
    Type=forking
    ExecStart=/usr/bin/emacs --daemon=%i --chdir %h
    ExecStop=/usr/bin/emacsclient --server-file=hud --eval "(progn (setq kill-emacs-hook 'nil) (kill-emacs))"
    Restart=always
    TimeoutStartSec=0
    
    [Install]
    WantedBy=default.target
    

    I then start emacs dameons:

    systemctl --user start emacs@work
    systemctl --user start emacs@personal
    systemctl --user start emacs@chat
    

    To enable them so that they start following boot:

      systemctl --user enable emacs@work
      systemctl --user enable emacs@personal
      systemctl --user enable emacs@chat
    
    Though to be honest, I use different names for daemons.
    
  • I have some amount of daemon specific code, which might be useful:

    (setq server-use-tcp t)
    
    (if (equal (daemonp) nil)
        (setq tychoish-emacs-identifier "solo")
      (setq tychoish-emacs-identifier (daemonp)))
    
    ;; this makes erc configs work less nosily. There's probably no harm in
    ;; turning down the logging
    (if (equal (daemonp) "chat")
        (setq gnutls-log-level 0)
      (setq gnutls-log-level 1))
    
    (let ((csname (if (eq (daemonp) nil)
                 "generic"
                 (daemonp))))
      (setq recentf-save-file (concat user-emacs-directory system-name "-" csname "-recentf"))
      (setq session-save-file (concat user-emacs-directory system-name "-" csname "-session"))
      (setq bookmark-default-file (concat user-emacs-directory system-name "-" csname "-bookmarks"))
      (setq helm-c-adaptive-history-file (concat user-emacs-directory system-name "-" csname "--helm-c-adaptive-history"))
      (setq desktop-base-file-name (concat system-name "-" csname "-desktop-file"))
      (setq desktop-base-lock-name (concat system-name "-" csname "-desktop-lock")))
    

    Basically this just sets up some session-specific information to be saved to different files, to avoid colliding per-instance.

  • Additionally, I use the tychoish-emacs-identifier from above to provide some contextual information as to what emacs daemon/window I'm currently in:

    (setq frame-title-format '(:eval (concat tychoish-emacs-identifier ":" (buffer-name))))
    
    (spaceline-emacs-theme 'daemon 'word-count)
    (spaceline-define-segment daemon tychoish-emacs-identifier)
    

    Also, on the topic of configuration, I do have a switch statement that loads different mu4e configurations in different daemons.

  • To start emacs sessions, I use operations in the following forms:

    # create a new emacs frame. Filename optional.
    emacsclient --server-file=<name> --create-frame --no-wait <filename>
    
    # open a file in an existing (last-focus) frame/window. Filename required.
    emacsclient --server-file=<name> --no-wait <filename>
    
    # open a terminal emacs mode. Filename optional .
    emacsclient --server-file=<name> --tty --no-wait <filename>
    

    That's a lot to type, so I use aliases in my shell profile:

    alias e='emacsclient --server-file=personal --no-wait'
    alias ew='emacsclient --server-file=personal --create-frame --no-wait'
    alias et='emacsclient --server-file=personal --tty'
    

    I create a set of aliases for each daemon prefixing e/ew/et with the first letter of the daemon name.

And that's about it. When I've used OS X, I've managed something similar using launchd but the configuration files are a bit less elegant. On OS X, I tend to install emacs with cocoa toolkit, using homebrew.

Using multiple daemons is cool, though not required, for a number of reasons:

  • you can have good separation between personal things and professional/work things, which is always nice, but particularly gratifying during the pandemic when it's easy to work forever.
  • Managing multiple separation of email. While mu4e has profiles and contexts, and that's great, I like a firmer boundary, and being able maintain separate email databases.
  • Running emacs lisp applications that do a lot of networking, or do other blocking operations. The main case where this matters in my experience is running big erc instance, or something else that isn't easily broken into async/subprocesses.
[1]And commenting!

Emacs and LSP Mode

I've been using emacs for a long time, and it's mostly pretty unexceptional. A few months ago, I started using LSP mode, and it's been pretty great... mostly. This post reflects on some of the dustier corners, and some of the broader implications.

History and Conflict

Language Server is a cool idea, that (as far as I know) grew out of VS Code, and provides a way to give a text editor "advanced, IDE-like" features without having to implement those features per-language in the text editor. Instead, editor developers implement generic integration for the protocol, and language developers produce a single process that can process source code and supports the editor features. Sounds great!

There are challenges:

  • The LSP protocol uses JSON for encoding data between the server and emacs. This is a reasonable choice, and makes development easy, but it's not a perfect fit. Primarily, in emacs, until version 27, which hasn't reached an official release yet (but is quite usable on Linux), parsed JSON in lisp rather than C, which made everything rather slow, but emacs 27 helps a lot. Second,

  • There is some overhead for the lsp, particularly as you attempt to scale very large numbers of files or very large files. Emacs is really good with large files, as an artifact of having had to support editing larger files on computers 20 or 30 years ago, and lsp feels sluggish in these situations.

    I think some of this overhead is probably depends on the implementation of the language server's themselves, and some percentage is probably protocol itself: http's statelessness is great in big distributed web services, and for the local use cases where responsiveness matters, it's less good. Similarly JSON's transparency and ubiquity is great, but for streaming streaming-type use cases, perhaps less so.

I've not poked around the protocol, or any implementations, extensively, so I'm very prepared to be wrong about these points.

There's also some tension about moving features, like support for adding editor support for a language, out of lisp and into an external process. While I'm broadly sympathetic, and think there are a bunch of high-quality language integrations in emacs lisp, and for basic things, the emacs lisp ones might be faster. Having said that, there's a lot of interface inconsistency between different support for languages in emacs, and LSP support feels spiffy my comparisons. Also at least for python (jedi,) go (gocode), and C/C++ (irony,) lisp (slime,) the non-lsp services were already using external processes for some portion of their functionality.

It's also the case that if people spend their emacs-lisp-writing time doing things other than writing support for various development environments it's all for the better anyway.

Suggestions / Notes

  • things will break or get wedged, call lsp-restart-workspace to restart the server (I think.) It helps.
  • use lsp-file-watch-ignored to avoid having the language servers process emacs process files from vendored code, intermediate/generated code, or similar, in cases where you might have a large number of files that aren't actually material to your development work
  • I believe some (many? all?) servers are sourced from the PATH (e.g. clang/llvm, gopls), which means you are responsible for updating them and their dependencies.
  • I've found it useful to update lsp-mode itself more often than I typically update other emacs packages. I also try and keep the language servers as up to date as possible in coordination with lsp-mode updates.
  • If Emacs 27 can work on your system, it's noticeably faster. Do that.
  • If you have an existing setup for a language environment, the LSP features end up layering over existing functionality, and that can have unexpected results.

My Configuration

None of this is particularly exciting, but if you use use-package, then the following might be an interesting starting point. I stole a lot of this from other places, so shrug.

I must say that I don't really use dap or treemacs, because I tend to keep things pretty barebones.

(use-package helm-lsp
  :ensure t
  :after (lsp-mode)
  :commands (helm-lsp-workspace-symbol)
  :init (define-key lsp-mode-map [remap xref-find-apropos] #'helm-lsp-workspace-symbol))

(use-package lsp-mode
  :diminish (lsp-mode . "lsp")
  :bind (:map lsp-mode-map
    ("C-c C-d" . lsp-describe-thing-at-point))
  :hook ((python-mode . #'lsp-deferred)
    (js-mode . #'lsp-deferred)
    (go-mode-hook . #'lsp-deferred))
  :init
  (setq lsp-auto-guess-root t       ; Detect project root
   lsp-log-io nil
   lsp-enable-indentation t
   lsp-enable-imenu t
   lsp-keymap-prefix "C-l"
   lsp-file-watch-threshold 500
   lsp-prefer-flymake nil)      ; Use lsp-ui and flycheck

  (defun lsp-on-save-operation ()
    (when (or (boundp 'lsp-mode)
         (bound-p 'lsp-deferred))
      (lsp-organize-imports)
      (lsp-format-buffer))))

(use-package lsp-clients
  :ensure nil
  :after (lsp-mode)
  :init (setq lsp-clients-python-library-directories '("/usr/local/" "/usr/")))

(use-package lsp-ui
  :ensure t
  :after (lsp-mode)
  :commands lsp-ui-doc-hide
  :bind (:map lsp-ui-mode-map
         ([remap xref-find-definitions] . lsp-ui-peek-find-definitions)
         ([remap xref-find-references] . lsp-ui-peek-find-references)
         ("C-c u" . lsp-ui-imenu))
  :init (setq lsp-ui-doc-enable t
         lsp-ui-doc-use-webkit nil
         lsp-ui-doc-header nil
         lsp-ui-doc-delay 0.2
         lsp-ui-doc-include-signature t
         lsp-ui-doc-alignment 'at-point
         lsp-ui-doc-use-childframe nil
         lsp-ui-doc-border (face-foreground 'default)
         lsp-ui-peek-enable t
         lsp-ui-peek-show-directory t
         lsp-ui-sideline-update-mode 'line
         lsp-ui-sideline-enable t
         lsp-ui-sideline-show-code-actions t
         lsp-ui-sideline-show-hover nil
         lsp-ui-sideline-ignore-duplicate t)
  :config
  (add-to-list 'lsp-ui-doc-frame-parameters '(right-fringe . 8))

  ;; `C-g'to close doc
  (advice-add #'keyboard-quit :before #'lsp-ui-doc-hide)

  ;; Reset `lsp-ui-doc-background' after loading theme
  (add-hook 'after-load-theme-hook
       (lambda ()
         (setq lsp-ui-doc-border (face-foreground 'default))
         (set-face-background 'lsp-ui-doc-background
                              (face-background 'tooltip))))

  ;; WORKAROUND Hide mode-line of the lsp-ui-imenu buffer
  ;; @see https://github.com/emacs-lsp/lsp-ui/issues/243
  (defadvice lsp-ui-imenu (after hide-lsp-ui-imenu-mode-line activate)
    (setq mode-line-format nil)))

;; Debug
(use-package dap-mode
  :diminish dap-mode
  :ensure t
  :after (lsp-mode)
  :functions dap-hydra/nil
  :bind (:map lsp-mode-map
         ("<f5>" . dap-debug)
         ("M-<f5>" . dap-hydra))
  :hook ((dap-mode . dap-ui-mode)
    (dap-session-created . (lambda (&_rest) (dap-hydra)))
    (dap-terminated . (lambda (&_rest) (dap-hydra/nil)))))

(use-package lsp-treemacs
  :after (lsp-mode treemacs)
  :ensure t
  :commands lsp-treemacs-errors-list
  :bind (:map lsp-mode-map
         ("M-9" . lsp-treemacs-errors-list)))

(use-package treemacs
  :ensure t
  :commands (treemacs)
  :after (lsp-mode))

Common Lisp Grip, Project Updates, and Progress

Last week, I did a release, I guess, of cl-grip which is a logging library that I wrote after reflecting on common lisp logging earlier. I wanted to write up some notes about it that aren't covered in the read me, and also talk a little bit4 about what else I'm working on.

cl-grip

This is a really fun and useful project and it was really the right size for a project for me to really get into, and practice a bunch of different areas (packages! threads! testing!) and I think it's useful to boot. The read me is pretty comprehensive, but I thought I'd collect some additional color here:

Really at it's core cl-grip isn't a logging library, it's just a collection of interfaces that make it easy to write logging and messaging tools, which is a really cool basis for an idea, (I've been working on and with a similar system in Go for years.)

As result, there's interfaces and plumbing for doing most logging related things, but no actual implementations. I was very excited to leave out the "log rotation handling feature," which feels like an anachronism at this point, though it'd be easy enough to add that kind of handler in if needed. Although I'm going to let it stew for a little while, I'm excited to expand upon it in the future:

  • additional message types, including capturing stack frames for debugging, or system information for monitoring.
  • being able to connect and send messages directly to likely targets, including systemd's journal and splunk collectors.
  • a collection of more absurd output targets to cover "alerting" type workloads, like desktop notifications, SUMP, and Slack targets.

I'm also excited to see if other people are interested in using it. I've submitted it to Quicklisp and Ultralisp, so give it a whirl!

See the cl-grip repo on github.

Eggqulibrium

At the behest of a friend I've been working on an "egg equilibrium" solver, the idea being to provide a tool that can given a bunch of recipes that use partial eggs (yolks and whites) can provide optimal solutions that use a fixed set of eggs.

So far I've implemented some prototypes that given a number of egg parts, attempt collects recipes until there are no partial eggs in use, so that there are no leftovers. I've also implemented the "if I have these partial eggs, what can I make to use them all." I've also implemented a rudimentary CLI interface (that was a trip!) and a really simple interface to recipe data (both parsing from a CSV format and an in memory format that makes solving the equilibrium problem easier.)

I'm using it as an opportunity to learn different things, and find a way to learn more about things I've not yet touched in lisp (or anywhere really,) so I'm thinking about:

  • building a web-based interface using some combination of caveman, parenscript, and related tools. This could include features like "user submitted databases," as well as links to the sources the recpies, in addition to the basic "web forms, APIs, and table rendering."
  • storing the data in a database (probably SQLite, mostly) both to support persistence and other more advanced features, but also because I've not done database things from Lisp at all.

See the eggquilibrium repo on github it's still pretty rough, but perhaps it'll be interesting!'

Other Projects

  • Writing more! I'm trying to be less obsessive about blogging, as I think it's useful (and perhaps interesting for you all too.) I've been writing a bunch and not posting very much of it. My goal is to mix sort of grandiose musing on technology and engineering, with discussions of Lisp, Emacs, and programming projects.
  • Working on producing texinfo output from cl-docutils! I've been toying around with the idea of writing a publication system targeted at producing books--long-form non-fiction, collections of essays, and fiction--rather than the blogs or technical resources that most such tools are focused on. This is sort of part 0 of this process.
  • Hacking on some Common Lisp projects, I'm particularly interested in the Nyxt and StumpWM.

Common Lisp and Logging

I've made the decision to make all of personal project code that I write to do in Common Lisp. See this post for some of the background for this decision.

It didn't take me long to say "I think I need a logging package," and I quickly found this wonderful comparsion of CL logging libraries, and only a little longer to be somewhat disappointed.

In general, my requirements for a logger are:

  • straightforward API for logging.
  • levels for filtering messages by importance
  • library in common use and commonly available.
  • easy to configure output targets (e.g. system's journal, external services, etc).
  • support for structured logging.

I think my rationale is pretty clear: loggers should be easy to use because the more information that can flow through the logger, the better. Assigning a level to all log messages is great for filtering practically, and it's ubiquitous enough that it's really part of having a good API. While I'm not opposed to writing my own logging system, [1] but I think I'd rather not in this case: there's too much that's gained by using the conventional choice.

Configurable outputs and structured logging are stretch goals, but frankly are the most powerful features you can add to a logger. Lots of logging work is spent crafting well formed logging strings, when really, you just want some kind of arbitrary map and makes it easier to make use of logging at scale, which is to say, when you're running higher workloads and multiple coppies of an application.

Ecosystem

I've dug in a bit to a couple of loggers, sort of using the framework above to evaluate the state of the existing tools. Here are some notes:

log4cl

My analysis of the CL logging packages is basically that log4cl is the most mature and actively maintained tool, but beyond the basic fundamentals, it's "selling" features are... interesting. [2] The big selling features:

  • integration with the developers IDE (slime,) which makes it possible to use the logging system like a debugger, almost. This is actually a phenomenal feature, particularly for development and debugging. The downside is that it wouldn't be totally unreasonable to use it production, and that's sort of terrifying.
  • it attempts to capture a lot of information about logging call sites so you can better map back from log messages to the state of the system when the call was made. Again, this makes it a debugging tool, and that's awesome, but it's overhead, and frankly I've never found it difficult to grep through code.
  • lots of attention to log rotation and log file management. There's not a lot of utility in writing log data to files directly. In most cases you want to write to standard out: the program is being used interactively, and users may want to see the log of what happens. In cases where you're running in a daemon mode, any more you're not, systemd or similar just captures your output. Even then, you're probably in a situation where you want to send the log output to some other process (e.g. an external service, or some kind of local socket for fluentd/etc.)
  • hierarchical organization of log messages is just less useful than annotation with metadata, in practice, and using hierarchical methods to filter logs into different streams or reduce logging volumes just obscures things and makes it harder to understand what's happening in a system.

Having said that, the API surface area is big, and it's a bit confusing to just start using the logger.

a-cl-logger

The acl package is pretty straightforward, and has a lot of features that I think are consistent with my interests and desires:

  • support for JSON output,
  • internal support for additional output formats (e.g. logstash,)
  • more simple API

It comes with a couple of pretty strong draw backs:

  • there are limited testing.
  • it's SBCL only, because it relies on SBCL fundamentals in collecting extra context about log messages. There's a pending pull request to add ECL compile support, but it looks like it wouldn't be quite that simple.
  • the overhead of collecting contextual information comes at an overhead, and I think logging should err on the side of higher performance, and making expensive things optional, just because it's hard to opt into/out of logging later.

Conclusion

So where does that leave me?

I'm not really sure.

I created a scratch project to write a simple logging project, but I'm definitely not prioritizing working on that over other projects. In the mean time I'll probably end up just not logging very much, or maybe giving log4cl a spin.

Notes

[1]When I started writing Go I did this, I wrote a logging tool, for a bunch of reasons. While I think it was the right decision at the time, I'm not sure that it holds up. Using novel infrastructure in projects makes integration a bit more complicated and creates overhead for would be contributors.
[2]To be honest, I think that log4cl is a fine package, and really a product of an earlier era, and it makes the standard assumptions about the way that logs were used, that makes sense given a very different view of how services should be operated.