Pave the On and Off Ramps

I participated in a great conversation in the #commonlisp channel on libera (IRC) the other day, during which I found a formulation of a familar argument that felt more clear and more concrete.

The question--which comes up pretty often, realistically--centered on adoption of Common Lisp. CL has some great tools, and a bunch of great libraries (particularly these days,) why don’t we see greater adoption? Its a good question, and maybe 5 year ago I would have said “the libraries and ecosystem are a bit fragmented,” and this was true. It’s less true now--for good reasons!--Quicklisp is just great and there’s a lot of coverage for doing common things.

I think it has to do with the connectivity and support at the edges of a project, an as I think about it, this is probably true of any kind of project.

When you decide to use a new tool or technology you ask yourself three basic questions:

1. “is this tool (e.g. language) capable of fulfilling my current needs” (for programming languages, this is very often yes,)
2. “are there tools (libraries) to support my use so I can focus on my core business objectives,” so that you’re not spending the entire time writing serialization libraries and HTTP servers, which is also often the case.
3. “will I be able to integrate what I’m building now with other tools I use and things I have built in the past.” This isn’t so hard, but it’s a thing that CL (and lots of other projects) struggle with.

In short, you want to be able to build a thing with the confidence that it’s possible to finish, that you’ll be able to focus on the core parts of the product and not get distracted by what should be core library functionality, and finally that the thing you build can play nicely with all the other things you’ve written or already have. Without this third piece, writing a piece of software with such a tool is a bit of a trap.

We can imagine tools that expose data only via quasi-opaque APIs that require special clients or encoding schemes, or that lack drivers for common databases, or integration with other common tools (metrics! RPC!) or runtime environments. This is all very reasonable. For CL this might look like:

• great support for gRPC

  There’s a grpc library that exists, is being maintained, and has basically all the features you’d want except support for TLS (a moderately big deal for operational reasons,) and async method support (not really a big deal.) It does depend on CFFI, which makes for a potentially awkward compilation story, but that’s a minor quibble.

  The point is not gRPC qua gRPC, the point is that gRPC is really prevalent globally and it makes sense to be able to meet developers who have existing gRPC services (or might like to imagine that they would,) and be able to give them confidence that whatever they build (in say CL) will be useable in the future.

• compilation that targets WASM

  Somewhat unexpectedly (to me, given that I don’t do a lot of web programming,) WebAssembly seems to be the way deploy portable machine code into environments that you don’t have full control over,¹ and while I don’t 100% understand all of it, I think it’s generally a good thing to make it easier to build software that can run in lots of situation.

• unequivocally excellent support for JSON (ex)

  I remember working on a small project where I thought “ah yes, I’ll just write a little API server in CL that will just output JSON,” and I completely got mired in various comparisons between JSON libraries and interfaces to JSON data. While this is a well understood problem it’s not a very cut and dry problem.

  The thing I wanted was to be able to take input in JSON and be able to handle it in CL in a reasonable way: given a stream (or a string, or equivalent) can I turn it into an object in CL (CLOS object? hashmap?)? I’m willing to implement special methods to support it given basic interfaces, but the type conversion between CL types and JSON isn’t always as straight forward as it is in other languages. Similarly with outputting data: is there a good method that will take my object and convert it to a JSON stream or string? There’s always a gulf between what’s possible and what’s easy and ergonomic.

I present these not as a complaint, or even as a call to action to address the specific issues that I raise (though I certianly wouldn’t complain if it were taken as such,) but more as an illustration of technical decision making and the things that make it possible for a team or a project to say yes to a specific technology.

There are lots of examples of technologies succeeding from a large competitive feild mostly on the basis of having great interoperability with existing solutions and tools, even if the core technology was less exciting or innovative. Technology wins on the basis of interoperability and user’s trust, not (exactly) on the basis of features.