Discussion

I think a lot about developer experience: I've spent my career, thus far, working on infrastructure products (databases, CI systems, build tools, release engineering,) that help improve developer experience and productivity, and even my day-to-day work tends to focus mostly on problems that impact the way that my teams write software: system architecture, service construction, observability, test infrastructure, and similar.

No matter how you slice it, editor experience is a huge factor in people's experience, and can have a big impact on productivity, and there are so many different editors with wildly different strengths. Editors experience is also really hard: unlike other kinds of developer infrastructure (like buildsystems, or even programming languages) the field conceptualizes editors as personal: your choice in editor is seen to reflect on you (it probably doesn't), and your ability to use your editor well is seen to be a reflection of your skills as a developer (it isn't). It's also the case that because editors are so personal, it's very difficult to produce an editor with broad appeal, and the most successful editors tend to be very configurable and can easily lack defaults, which means that even great editors are terrible out of the box, which mostly affects would-be and new developers.

Nevertheless, time being able to have an editor that you're comfortable with and that you can use effectively without friction does make it easier to build software, so even if folks often conceptualize of editors in the wrong way, improving the editing experience is important. I think that there are two areas for improvement:

• editor configurations should--to some extent--be maintained at the project (or metaproject) level, rather than on the level individual engineer. The hard part here is how to balance individual preference with providing a consistent experience, particularly for new developers. [1]
• there should be more "starter kits" (like this one! or the many other starter kits, but also for (neo)vim, vscode, and others.) that help bootstrap the experience, while also figuring out ways to allow layering other project-based extensions on top of a base configuration.

Also, I want to give chemacs a shout out, for folks who want to try out other base configurations.

History and Conflict

Language Server is a cool idea, that (as far as I know) grew out of VS Code, and provides a way to give a text editor "advanced, IDE-like" features without having to implement those features per-language in the text editor. Instead, editor developers implement generic integration for the protocol, and language developers produce a single process that can process source code and supports the editor features. Sounds great!

There are challenges:

• The LSP protocol uses JSON for encoding data between the server and emacs. This is a reasonable choice, and makes development easy, but it's not a perfect fit. Primarily, in emacs, until version 27, which hasn't reached an official release yet (but is quite usable on Linux), parsed JSON in lisp rather than C, which made everything rather slow, but emacs 27 helps a lot. Second,

• There is some overhead for the lsp, particularly as you attempt to scale very large numbers of files or very large files. Emacs is really good with large files, as an artifact of having had to support editing larger files on computers 20 or 30 years ago, and lsp feels sluggish in these situations.

  I think some of this overhead is probably depends on the implementation of the language server's themselves, and some percentage is probably protocol itself: http's statelessness is great in big distributed web services, and for the local use cases where responsiveness matters, it's less good. Similarly JSON's transparency and ubiquity is great, but for streaming streaming-type use cases, perhaps less so.

I've not poked around the protocol, or any implementations, extensively, so I'm very prepared to be wrong about these points.

There's also some tension about moving features, like support for adding editor support for a language, out of lisp and into an external process. While I'm broadly sympathetic, and think there are a bunch of high-quality language integrations in emacs lisp, and for basic things, the emacs lisp ones might be faster. Having said that, there's a lot of interface inconsistency between different support for languages in emacs, and LSP support feels spiffy my comparisons. Also at least for python (jedi,) go (gocode), and C/C++ (irony,) lisp (slime,) the non-lsp services were already using external processes for some portion of their functionality.

It's also the case that if people spend their emacs-lisp-writing time doing things other than writing support for various development environments it's all for the better anyway.

Suggestions / Notes

• things will break or get wedged, call lsp-restart-workspace to restart the server (I think.) It helps.
• use lsp-file-watch-ignored to avoid having the language servers process emacs process files from vendored code, intermediate/generated code, or similar, in cases where you might have a large number of files that aren't actually material to your development work
• I believe some (many? all?) servers are sourced from the PATH (e.g. clang/llvm, gopls), which means you are responsible for updating them and their dependencies.
• I've found it useful to update lsp-mode itself more often than I typically update other emacs packages. I also try and keep the language servers as up to date as possible in coordination with lsp-mode updates.
• If Emacs 27 can work on your system, it's noticeably faster. Do that.
• If you have an existing setup for a language environment, the LSP features end up layering over existing functionality, and that can have unexpected results.

My Configuration

None of this is particularly exciting, but if you use use-package, then the following might be an interesting starting point. I stole a lot of this from other places, so shrug.

I must say that I don't really use dap or treemacs, because I tend to keep things pretty barebones.

(use-package helm-lsp
  :ensure t
  :after (lsp-mode)
  :commands (helm-lsp-workspace-symbol)
  :init (define-key lsp-mode-map [remap xref-find-apropos] #'helm-lsp-workspace-symbol))

(use-package lsp-mode
  :diminish (lsp-mode . "lsp")
  :bind (:map lsp-mode-map
    ("C-c C-d" . lsp-describe-thing-at-point))
  :hook ((python-mode . #'lsp-deferred)
    (js-mode . #'lsp-deferred)
    (go-mode-hook . #'lsp-deferred))
  :init
  (setq lsp-auto-guess-root t       ; Detect project root
   lsp-log-io nil
   lsp-enable-indentation t
   lsp-enable-imenu t
   lsp-keymap-prefix "C-l"
   lsp-file-watch-threshold 500
   lsp-prefer-flymake nil)      ; Use lsp-ui and flycheck

  (defun lsp-on-save-operation ()
    (when (or (boundp 'lsp-mode)
         (bound-p 'lsp-deferred))
      (lsp-organize-imports)
      (lsp-format-buffer))))

(use-package lsp-clients
  :ensure nil
  :after (lsp-mode)
  :init (setq lsp-clients-python-library-directories '("/usr/local/" "/usr/")))

(use-package lsp-ui
  :ensure t
  :after (lsp-mode)
  :commands lsp-ui-doc-hide
  :bind (:map lsp-ui-mode-map
         ([remap xref-find-definitions] . lsp-ui-peek-find-definitions)
         ([remap xref-find-references] . lsp-ui-peek-find-references)
         ("C-c u" . lsp-ui-imenu))
  :init (setq lsp-ui-doc-enable t
         lsp-ui-doc-use-webkit nil
         lsp-ui-doc-header nil
         lsp-ui-doc-delay 0.2
         lsp-ui-doc-include-signature t
         lsp-ui-doc-alignment 'at-point
         lsp-ui-doc-use-childframe nil
         lsp-ui-doc-border (face-foreground 'default)
         lsp-ui-peek-enable t
         lsp-ui-peek-show-directory t
         lsp-ui-sideline-update-mode 'line
         lsp-ui-sideline-enable t
         lsp-ui-sideline-show-code-actions t
         lsp-ui-sideline-show-hover nil
         lsp-ui-sideline-ignore-duplicate t)
  :config
  (add-to-list 'lsp-ui-doc-frame-parameters '(right-fringe . 8))

  ;; `C-g'to close doc
  (advice-add #'keyboard-quit :before #'lsp-ui-doc-hide)

  ;; Reset `lsp-ui-doc-background' after loading theme
  (add-hook 'after-load-theme-hook
       (lambda ()
         (setq lsp-ui-doc-border (face-foreground 'default))
         (set-face-background 'lsp-ui-doc-background
                              (face-background 'tooltip))))

  ;; WORKAROUND Hide mode-line of the lsp-ui-imenu buffer
  ;; @see https://github.com/emacs-lsp/lsp-ui/issues/243
  (defadvice lsp-ui-imenu (after hide-lsp-ui-imenu-mode-line activate)
    (setq mode-line-format nil)))

;; Debug
(use-package dap-mode
  :diminish dap-mode
  :ensure t
  :after (lsp-mode)
  :functions dap-hydra/nil
  :bind (:map lsp-mode-map
         ("<f5>" . dap-debug)
         ("M-<f5>" . dap-hydra))
  :hook ((dap-mode . dap-ui-mode)
    (dap-session-created . (lambda (&_rest) (dap-hydra)))
    (dap-terminated . (lambda (&_rest) (dap-hydra/nil)))))

(use-package lsp-treemacs
  :after (lsp-mode treemacs)
  :ensure t
  :commands lsp-treemacs-errors-list
  :bind (:map lsp-mode-map
         ("M-9" . lsp-treemacs-errors-list)))

(use-package treemacs
  :ensure t
  :commands (treemacs)
  :after (lsp-mode))

cl-grip

This is a really fun and useful project and it was really the right size for a project for me to really get into, and practice a bunch of different areas (packages! threads! testing!) and I think it's useful to boot. The read me is pretty comprehensive, but I thought I'd collect some additional color here:

Really at it's core cl-grip isn't a logging library, it's just a collection of interfaces that make it easy to write logging and messaging tools, which is a really cool basis for an idea, (I've been working on and with a similar system in Go for years.)

As result, there's interfaces and plumbing for doing most logging related things, but no actual implementations. I was very excited to leave out the "log rotation handling feature," which feels like an anachronism at this point, though it'd be easy enough to add that kind of handler in if needed. Although I'm going to let it stew for a little while, I'm excited to expand upon it in the future:

• additional message types, including capturing stack frames for debugging, or system information for monitoring.
• being able to connect and send messages directly to likely targets, including systemd's journal and splunk collectors.
• a collection of more absurd output targets to cover "alerting" type workloads, like desktop notifications, SUMP, and Slack targets.

I'm also excited to see if other people are interested in using it. I've submitted it to Quicklisp and Ultralisp, so give it a whirl!

See the cl-grip repo on github.

Eggqulibrium

At the behest of a friend I've been working on an "egg equilibrium" solver, the idea being to provide a tool that can given a bunch of recipes that use partial eggs (yolks and whites) can provide optimal solutions that use a fixed set of eggs.

So far I've implemented some prototypes that given a number of egg parts, attempt collects recipes until there are no partial eggs in use, so that there are no leftovers. I've also implemented the "if I have these partial eggs, what can I make to use them all." I've also implemented a rudimentary CLI interface (that was a trip!) and a really simple interface to recipe data (both parsing from a CSV format and an in memory format that makes solving the equilibrium problem easier.)

I'm using it as an opportunity to learn different things, and find a way to learn more about things I've not yet touched in lisp (or anywhere really,) so I'm thinking about:

• building a web-based interface using some combination of caveman, parenscript, and related tools. This could include features like "user submitted databases," as well as links to the sources the recpies, in addition to the basic "web forms, APIs, and table rendering."
• storing the data in a database (probably SQLite, mostly) both to support persistence and other more advanced features, but also because I've not done database things from Lisp at all.

See the eggquilibrium repo on github it's still pretty rough, but perhaps it'll be interesting!'

Other Projects

• Writing more! I'm trying to be less obsessive about blogging, as I think it's useful (and perhaps interesting for you all too.) I've been writing a bunch and not posting very much of it. My goal is to mix sort of grandiose musing on technology and engineering, with discussions of Lisp, Emacs, and programming projects.
• Working on producing texinfo output from cl-docutils! I've been toying around with the idea of writing a publication system targeted at producing books--long-form non-fiction, collections of essays, and fiction--rather than the blogs or technical resources that most such tools are focused on. This is sort of part 0 of this process.
• Hacking on some Common Lisp projects, I'm particularly interested in the Nyxt and StumpWM.

Ecosystem

I've dug in a bit to a couple of loggers, sort of using the framework above to evaluate the state of the existing tools. Here are some notes:

Conclusion

So where does that leave me?

I'm not really sure.

I created a scratch project to write a simple logging project, but I'm definitely not prioritizing working on that over other projects. In the mean time I'll probably end up just not logging very much, or maybe giving log4cl a spin.

Notes

Why Common Lisp?

First, some background. I've always thought that Common Lisp is a language with a bunch of cool features and selling points, but unsurprisingly, I've never really had the experience of writing more than some one-off bits of code in CL, which isn't surprising. This project was a good experience for really digging into writing and managing a conceptually larger project which was a good kick in the pants to learn more.

The things I like:

• the implementations of the core runtime are really robust and high quality, and make it possible to imagine running your code in a bunch of different contexts. Even though it's a language with relatively few users, it feels robust in a way. The most common implementations also have ways of producing fully self contained static binaries (like Go, say), which makes the thought of distributing software seem reasonable.
• quicklisp, a package/library management tool is new (in the last decade or so,) has really raised the level of the ecosystem. It's not as complete as I'd expect in many ways, but quicklisp changed CL from something quaint to something that you could actually imagine using.
• the object system is really nice. There isn't quite compile time-type checking on the values of slots (attributes) of objects, though you can opt in. My general feeling is that I can pretty easily get the feeling of writing statically typed code with all of the freedom of writing dynamic code.
• multiple dispatch, and the conceptual approach to genericism, is amazing and really simplifies flow control in a lot of cases. You implement the methods you need, for the appropriate types/objects and then just write the logic you need, and the function call machinery just does the right thing. There's surprisingly little conditional logic, as a result.

Things I don't like:

• there are all sorts of things that don't quite have existing libraries, and so I find myself wanting to do things that require more effort than necessary. This project to write a binary encoding tool would have been a component in service of a much larger project. It'd be nice if you could skip some of the lower level components, or didn't have your design choices so constrained by gaps in infrastructure.
• at the same time, the library ecosystem is pretty fractured and there are common tools around which there aren't really consensus. Why are there so many half-finished YAML and JSON libraries? There are a bunch of HTTP server (!) implementations, but really you need 2 and not 5?
• looping/iteration isn't intuitive and difficult to get common patterns to work. The answer, in most cases is to use (map) with lambdas rather than loops, in most cases, but there's this pitfall where you try and use a (loop) and really, that's rarely the right answer.
• implicit returns seem like an over sight, hilariously, Rust also makes this error. Implicit returns also make knowing what type a function or method returns quite hard to reason about.

Writing an Encoder

So the project I wrote was an attempt to write really object oriented code as a way to writing an object encoder to a JSON-like format. Simple enough, I had a good mental model of the format, and my general approach to doing any kind of binary format processing is to just write a crap ton of unit tests and work somewhat iteratively.

I had a lot of fun with the project, and it gave me a bunch of key experiences which make me feel comfortable saying that I'm able to write common lisp even if it's not a language that I feel maximally comfortable in (yet?). The experiences that really helped included:

• producing enough code to really have to think about how packaging and code organization worked. I'd written a function here and there, before, but never something where I needed to really understand and use the library/module/packaging (e.g. systems and libraries.) infrastructure.
• writing and running lots of tests. I don't always follow test-driven development closely, but writing lots of tests is part of my process, and being able to watch the layers of this code come together was a lot of fun and very instructive.
• this project for me, was mostly about API design and it was nice to have a project that didn't require much design in terms of the actual functionality, as object encoding is pretty straight forward.

From an educational perspective all of my goals were achieved.

Failure Mode

The problem is that it didn't work out, in the final analysis. While the library I constructed was able to encode and decode objects and was internally correct, I never got it to produce encoding that other implementations of the same specification could reliably read, and the ability to read data encoded by other libraries only worked in trivial cases. In the end:

• this was mostly a project designed to help me gain competence in a programming language I don't really know, and in that I was successful.
• adding this encoding format isn't particularly useful to any project I'm thinking of working on in the short term, and doesn't directly enable me to do anything in particular.
• the architecture of the library would not be particularly performant in practice, as the encoding process didn't deploy a buffer pool of any kind, and it would have been harder than not to back fill that in, and I wasn't particularly interested in that.
• it didn't work, and the effort to debug the issue would be more substantive than I'm really interested in doing at this point, particularly given the limited utility.

So here we are. Onto a different project!