I've made the decision to make all of personal project code that I write to do in Common Lisp. See this post for some of the background for this decision.

It didn't take me long to say "I think I need a logging package," and I quickly found this wonderful comparsion of CL logging libraries, and only a little longer to be somewhat disappointed.

In general, my requirements for a logger are:

  • straightforward API for logging.
  • levels for filtering messages by importance
  • library in common use and commonly available.
  • easy to configure output targets (e.g. system's journal, external services, etc).
  • support for structured logging.

I think my rationale is pretty clear: loggers should be easy to use because the more information that can flow through the logger, the better. Assigning a level to all log messages is great for filtering practically, and it's ubiquitous enough that it's really part of having a good API. While I'm not opposed to writing my own logging system, [1] but I think I'd rather not in this case: there's too much that's gained by using the conventional choice.

Configurable outputs and structured logging are stretch goals, but frankly are the most powerful features you can add to a logger. Lots of logging work is spent crafting well formed logging strings, when really, you just want some kind of arbitrary map and makes it easier to make use of logging at scale, which is to say, when you're running higher workloads and multiple coppies of an application.

Ecosystem

I've dug in a bit to a couple of loggers, sort of using the framework above to evaluate the state of the existing tools. Here are some notes:

log4cl

My analysis of the CL logging packages is basically that log4cl is the most mature and actively maintained tool, but beyond the basic fundamentals, it's "selling" features are... interesting. [2] The big selling features:

  • integration with the developers IDE (slime,) which makes it possible to use the logging system like a debugger, almost. This is actually a phenomenal feature, particularly for development and debugging. The downside is that it wouldn't be totally unreasonable to use it production, and that's sort of terrifying.
  • it attempts to capture a lot of information about logging call sites so you can better map back from log messages to the state of the system when the call was made. Again, this makes it a debugging tool, and that's awesome, but it's overhead, and frankly I've never found it difficult to grep through code.
  • lots of attention to log rotation and log file management. There's not a lot of utility in writing log data to files directly. In most cases you want to write to standard out: the program is being used interactively, and users may want to see the log of what happens. In cases where you're running in a daemon mode, any more you're not, systemd or similar just captures your output. Even then, you're probably in a situation where you want to send the log output to some other process (e.g. an external service, or some kind of local socket for fluentd/etc.)
  • hierarchical organization of log messages is just less useful than annotation with metadata, in practice, and using hierarchical methods to filter logs into different streams or reduce logging volumes just obscures things and makes it harder to understand what's happening in a system.

Having said that, the API surface area is big, and it's a bit confusing to just start using the logger.

a-cl-logger

The acl package is pretty straightforward, and has a lot of features that I think are consistent with my interests and desires:

  • support for JSON output,
  • internal support for additional output formats (e.g. logstash,)
  • more simple API

It comes with a couple of pretty strong draw backs:

  • there are limited testing.
  • it's SBCL only, because it relies on SBCL fundamentals in collecting extra context about log messages. There's a pending pull request to add ECL compile support, but it looks like it wouldn't be quite that simple.
  • the overhead of collecting contextual information comes at an overhead, and I think logging should err on the side of higher performance, and making expensive things optional, just because it's hard to opt into/out of logging later.

Conclusion

So where does that leave me?

I'm not really sure.

I created a scratch project to write a simple logging project, but I'm definitely not prioritizing working on that over other projects. In the mean time I'll probably end up just not logging very much, or maybe giving log4cl a spin.

Notes

[1]When I started writing Go I did this, I wrote a logging tool, for a bunch of reasons. While I think it was the right decision at the time, I'm not sure that it holds up. Using novel infrastructure in projects makes integration a bit more complicated and creates overhead for would be contributors.
[2]To be honest, I think that log4cl is a fine package, and really a product of an earlier era, and it makes the standard assumptions about the way that logs were used, that makes sense given a very different view of how services should be operated.