Topic: Software

This is a post I wrote a long time ago and never posted, but I’ve started getting back into doing some work in Common Lisp and thought it’d be good to send this one off.

On my recent “(re)learn Common Lisp” journey, I’ve happened across a few things that I’ve found frustrating or confusing: this post is a collection of them, in hopes that other people don’t struggle with them:

• Implementing an existing generic function for a class of your own, and have other callers specialize use your method implementation you must import the generic function, otherwise other callers will (might?) fall back to another method. This makes sense in retrospect, but definitely wasn’t clear on the first go.
• As a related follow on, you don’t have to define a generic function in order to write or use a method, and I’ve found that using methods is actually quite nice for doing some type checking, at the same time, it can get you into a pickle if you later add the generic function and it’s not exported/imported as you want.
• Property lists seem cool for a small light weight mapping, but they’re annoying to handle as part of public APIs, mostly because they’re indistinguishable from regular lists, association lists are preferable, and maybe with make-hash even hash-tables.
• Declaring data structures inline is particularly gawky. I sometimes want to build a list or a hash map in-line an alist, and it’s difficult to do that in a terse way that doesn’t involve building the structure programatically. I’ve been writing (list (cons "a" t) (cons "b" nil)) sort of things, which I don’t love.
• If you have a variadic macro (i.e. that takes &rest args), or even I suppose any kind of macro, and you have it’s arguments in a list, there’s no a way, outside of eval to call the macro, which is super annoying, and makes macros significantly less appealing as part of public APIs. My current conclusion is that macros are great when you want to add syntax to make the code you’re writing clearer or to introduce a new paradigm, but for things that could also be a function, or are thin wrappers on for function, just use a function.

I talk to lots of people about software and programming: people who are trying to make a technical decision for a new project or who are interested in learning something new, and some form of the question “what programming language should I learn?” or “what’s the best language for this new project?” comes up a lot.

These are awful questions, because there is no singular right answer, and in some senses all answers are wrong. This post will be an exploration of some decent answers to this question, and some useful ways to think about the differences between programming languages.

• If you already build and maintain software in one programming language, build new components in the same language you already use. Adding new tools and technologies increases maintenance burden for all engineers, and software tends to stick around for a long time, so this cost can stick around for a long time.

• Sometimes the software you want to write must target a specific runtime or environment, there’s really only one reasonable choice. The prototypical examples of these are things like: iOS apps (Swift,) Android apps (Kotlin), or things that run in the browser (JavaScript,) although:

• Given things like React Native and Electron, it’s reasonable to just write JavaScript for all GUI code, although often this might actually mean TypeScript in practice. While it used to be the case that it made sense to write GUI code in various native tool kits, at this point it seems like it makes sense to just figure out ways of doing it all in JS.

• If you already know how to program in one language, and want to learn something new, but don’t have a specific project in mind attempt to learn something that’s quite different from you already know: if you’re comfortable in something like Python, try and learn something like Go or Rust. If you’re primarily a Java programmer, something like JavaScript or Python might be an interesting change of pace.

  The same basic ideas applies to selecting languages that will be used by teams: choose a tool that’s complementary to what you’re already doing, and that could provide value.

• If you’re more familiar with a few programming languages or don’t feel you need to learn a new language for professional reasons pick something fun and off the wall: Ocaml! Common Lisp! Rust! Haskell! Scheme! Elixir! It doesn’t matter and in these cases you probably can probably learn new languages when you need, the point is to learn something that’s radically different and to help you think about computers and programming in radically different ways.

• Choose the language that people working on similar projects are already using. For instance, if you’re doing a lot of data science, using Python makes a lot of sense; if you’re writing tools that you expect banks (say) to use, something that runs on the JVM is a good bet. The idea here is you may be able to find more well developed tools and resources relevant to the kinds of problems you encounter.

• When starting a new project and there isn’t a lot of prior art in the area that you’re working, or you want to avoid recapitulating some flaw in the existing tools, you end up having a lot of freedom. In general:

  • Think about concurrency and workload characteristics. Is the workload CPU, Network, or IO bound? Is the application multithreaded, or could take advantage of parallelism within processes? There are different kinds of concurrency, and different execution models, so this isn’t always super cut-and-dry: theoretically languages that have “real threads” (C/C++, Java, Rust, Common Lisp, etc.) or a close enough approximation (Go,) are better, but for workloads that are network bound, event-driven systems (e.g. Python’s Tornado and Node.J’s) work admirably.
  • How will you distribute and run the application? There are some languages that can provide static binaries that include all of their dependencies for distribution, which can simplify some aspects of distribution and execution process, but for software that you control the runtime (e.g. services deployed on some kind of container based-platform,) it might matter less.
  • Are there strong real-time requirements? If so, and you’re considering a garbage collected language, make sure that the GC pauses aren’t going to be a problem. It’s also the case that all GCsare not the same, so having a clear idea of what the tolerances
  • Is this software going to be maintained by a team, and if so, what kind of tools will they need in order to succeed and be productive. Would static typing help? What’s the developer tooling and experience like? Are there libraries that you’d expect to need that are conspicuously missing?

Have fun! Build cool things!

This is a post in my alignment series. See the introductory post Finding Alignment for more context.

I think, in practice, most of what managers do--and indeed all leadership--is about building alignment. The core concept, of alignment, having a shared understanding of the problem space and its context combined with relevant goals and objectives, and grasp of how the context contexts to these objectives. Alignment isn’t just “agreement” or “understanding the solution,” and really centers on this connection between context and goals. Alignment shows up in many different situations and interactions:

• a small working group (2-4 people) who are working on building or developing something. The thing can be any kind of work product: a piece of software, documentation, a business process, a marketing campaign, a sales deal. When you have more than one person working on something, if they’re not aligned, each person may be able to work on a piece of work as delegated or assigned, but lacks the ability to (reliably) continue to work on the next piece of work after finishing a narrow task, or be able to assess if a line of work is still germane to the goals as things develop. If we view people’s roles in projects as machines, and they perform assigned tasks well, then alignment isn’t super critical, but if you need people to make decisions and act upon them, then they have to be aligned, as a group otherwise the project runs a huge risk of stalling out as each contributor pulls in an opposite direction.
• one person aligning with the rest of their team to understand how their background and personal goals contribute to and interact with the team’s context and goals. Individuals all bring unique skills and interests and (hopefully) useful to teams, and teams (e.g. their leaders) need to be able to understand how to effectively use those skills and interests to support the team’s goals. This happens over conversation and in the context of someones participation in a team over time, and doesn’t need to take a lot of time on a regular basis, but cannot be entirely abandoned.
• managers need to align their teams with the company’s objectives. This takes the form of making sure that the projects that the team is working on (and will work on in the future,) support the organization and company’s larger goals.
• across all level each team needs to align with its peer teams and the organization that it belongs in. This is true in organizations with 30 people and 3-4 teams, and in organizations of 2000 people and dozens of teams.

Alignment is hierarchical, and largely the responsibility of leaders to monitor alignment above and below them, and understand if their teams or specific contributors are falling out of alignment. This doesn’t necessarily mean that it’s not participatory and discursive: individuals can impact the direction, goals, or alignment of their teams, but there must be well formed goals of the organization (that they can understand!) and they must be supported by their team in order to actualize in this dimension. Despite being hierarchical, and individuals and teams must align up building and maintaining alignment in all directions is actually the responsibility of leadership at all levels.

It’s easy to frame this as “you must align with the goals sent from above,” this couldn’t be further from the truth. Some organizations function like this, but it’s probably not healthy for anyone, because the kinds of alignment that it builds are fleeting and tactical. Teams and contributors do need to align with broader goals (up), but their job is not building alignment, it’s building whatever their specialty is: attending to the organizational health and alignment is the concern of leadership whose work must center on building alignment. At almost every level, the alignment goes both ways: you work with leaders above you to align your own work and team, and work with the people you collaborate and mentor to build alignment.

When it works, and even though it takes a while, it helps teams and organizations work really well.

I’ve been thinking more recently about that the way that we organize software development projects, and have been using this model of “hard mode” vs “easy mode” a bit recently, and thought it might be useful to expound upon it.

Often users or business interests come to software developers and make a feature request. “I want it to be possible to do thing faster or in combination with another operation, or avoid this class of errors.” Sometimes (often!) these are reasonable requests, and sometimes they’re even easy to do, but sometimes a seemingly innocuous feature or improvement are really hard sometimes, engineering work requires hard work. This isn’t really a problem, and hard work can be quite interesting. It is perhaps, an indication of an architectural flaw when many or most easy requests require disproportionately hard work.

It’s also the case that it’s possible to frame the problem in ways that make the work of developing software easier or harder. Breaking problems into smaller constituent problems make them easier to deal with. Improving the quality of the abstractions and testing infrastructure around a problematic area of code makes it easier to make changes later to an area of the code.

I’ve definitely been on projects where the only way to develop features and make improvement is to have a large amount of experience with the problem domain and the codebase, and those engineers have to spend a lot of consentrated time building features and fighting against the state of the code and its context. This is writing software in “hard mode,” and not only is the work harder than it needs to be, features take longer to develop than users would like. This mode of development makes it very hard to find and retain engineers because of the large ramping period and consistently frustrating nature of the work. Frustration that’s often compounded by the expectation or assumption that easy requests are easy to produce.

In some ways the theme of my engineering career has been work on taking “hard mode projects” reducing the barriers to entry in code bases and project so that they become more “easy mode projects”: changing the organization of the code, adding abstractions that make it easier to develop meaningful features without rippling effects in other parts of the code, improving operational observability to facilitate debugging, restructuring project infrastructure to reduce development friction. In general, I think of the hallmarks of “easy mode” projects as:

• abstractions and library functions exist for common tasks. For most pieces of “internet infrastructure” (network attached services,) developer’s should be able to add behavior without needing to deal with the nitty gritty of thread pools or socket abstractions (say.) If you’re adding a new REST request, you should be able to just write business logic and not need to think about the applications threading model (say). If something happens often (say, retrying failed requests against upstream API,) you should be able to rely on an existing tool to orchestrate retries.
• APIs and tools are safe ergonomic. Developers writing code in your project should be able to call into existing APIs and trust that they behave reasonably and handle errors reasonably. This means, methods should do what they say, and exported/public interfaces should be difficult to use improperly, and (e.g. expected exception handling/safety, as well as thread safety and nil semantics ad appropriate.) While it’s useful to interact with external APIs defensively, you can reduce the amount of effort by being less defensive for internal/proximal APIs.
• Well supported code and operational infrastructure. It should be easy to deploy and test changes to the software, the tests should run quickly, and when there’s a problem there should be a limited number of places that you could look to figure out what’s happening. Making tests more reliable, improving error reporting and tracing, exposing more information to metrics systems, to make the behavior of the system easier to understand in the long term.
• Changes are scoped to support incremental development. While there are lots of core technical and code infrastructure work that support making projects more “easy mode” a lot of this is about the way that development teams decide to structure projects. This isn’t technical, ususally, but has more to do with planning cadences, release cadences, and scoping practices. There are easier and harder ways of making changes, and it’s often worthwhile to ask yourself “could we make this easier.” The answer, I’ve found, is often “yes”.

Moving a project from hard to easy mode is often in large part about investing in managing technical debt, but it’s also a choice: we can prioritize things to make our projects easier, we can make small changes to the way we approach specific projects that all move projects toward being easier. The first step is always that choice.

Before I started actually working as a software engineer full time, writing code was this fun thing I was always trying to figure out on my own, and it was fun, and I could hardly sit down at my computer without learning something. These days, I do very little of this kind of work. I learn more about computers by doing my job and frankly, the kind of software I write for work is way more satisfying than any of the software I would end up writing for myself.

I think this is because the projects that a team of engineers can work on are necessarily larger and more impactful. When you build software with a team, most of the time the product either finds users (or your end up without a job.) When you build software with other people and for other people, the things that make software good (more rigorous design, good test discipline, scale,) are more likely to be prevalent. Those are the things that make writing software fun.

Wait, you ask “this is a lisp post?” and “where is the lisp content?” Wait for it…

In Pave the On/Off Ramps¹ I started exploring this idea that technical adoption is less a function of basic capabilities or numbers of features in general, but rather about the specific features that support and further adoption and create confidence in maintenance and interoperability. A huge part of the decision process is finding good answers to “can I use these tools as part of the larger system of tools that I’m using?” and “can I use this tool a bit without needing to commit to using it for everything?”

Technologies which are and demand ideological compliance are very difficult to move into with confidence. A lot of technologies and tools demand ideological compliance, and their adoption depends on once-in-a-generation sea changes or significant risks.² The alternate method, to integrate into people’s existing workflows and systems, and provide great tools that work for some usecases and to prove their capability is much more reliable: if somewhat less exciting.

The great thing about Common Lisp is that it always leans towards the pragmatic rather than the ideological. Common Lisp has a bunch of tools--both in the langauge and in the ecosystem--which are great to use but also not required. You don’t have to use CLOS (but it’s really cool), you don’t have to use ASDF, there isn’t one paradigm of developing or designing software that you have to be constrained to. Do what works.

I think there are a lot of questions that sort of follow on from this, particularly about lisp and the adoption of new technologies. So let’s go through the ones I can think of, FAQ style:

• What kind of applications would a “pave the exits” support?

  It almost doesn’t matter, but the answer is probably a fairly boring set of industrial applications: services that transform and analyze data, data migration tools, command-line (build, deployment) tools for developers and operators, platform orchestration tools, and the like. This is all boring (on the one hand,) but most software is boring, and it’s rarely the case that programming langauge actually matters much.

  In addition, CL has a pretty mature set of tools for integrating with C libaries and might be a decent alternative to other langauges with more complex distribution stories. You could see CL being a good langauge for writing extensions on top of existing tools (for both Java with ABCL and C/C++ with ECL and CLASP), depending.

• How does industrial adoption of Common Lisp benefit the Common Lisp community?

  First, more people writing common lisp for their jobs, which (assuming they have a good experience,) could proliferate into more projects. A larger community, maybe means a larger volume of participation in existing projects (and more projects in general.) Additionally, more industrial applications means more jobs for people who are interested in writing CL, and that seems pretty cool.

• How can CL compete with more established languages like Java, Go, and Rust?

  I’m not sure competition is really the right model for thinking about this: there’s so much software to write that “my langauge vs your langauge” is just a poor model for thinking about this: there’s enough work to be done that everyone can be successful.

  At the same time, I haven’t heard about people who are deeply excited about writing Java, and Go folks (which I count myself among) tend to be pretty pragmatic as well. I see lots of people who are excited about Rust, and it’s definitely a cool langauge though it shines best at lower level problems than CL and has a reasonable FFI so it might be the case that there’s some exciting room for using CL for higher level tasks on top of rust fundamentals.

This is a post in my alignment series. See the introductory post Finding Alignment for more context.

I really want to dig into some topics related to building alignment and figuring out when you’re aligned as a contributor, or when the people you’re working with are falling out of alignment with you and/or your team or organization, but I think it’s worth it to start slow and chew on a big question: What it feels like when you and your team are well aligned, and why that’s a good thing.

To my mind, when you have a foundation of alignment, and an understanding of what the business goals are for your organization, then it becomes really easy to work independently, because you know what’s important, you know what needs to happen next and the people your working for/with can be confident that you’ll be moving in the right direction, and don’t need to do as much monitoring. Every so often, teams find this, and can really grind on it and deliver great features and products on the basis of this. It takes a long time (months!) for a team to gel like this, and sometimes teams don’t quite get there.

This isn’t to say that needing more guidance and wokring less independently means that you’re unaligned just that you (or the people you’re working with/for) are newer to the team, or there’s been a change recently and everyone needs more touch points to build alignment. One of the risks of hiring people and growing teams that are really well aligned is that the change in team dynamic can throw off alignment, and I think this is one of the reasons that teams sometimes struggle to grow. In any case, while alignment is great and it doesn’t happen for free, and it’s fine for it to be a thing you’re working on.

Alignment also reduces a lot of potentially contentious conversations and interactions: when you have alignment within a team or between teams you have a framework for prioritizing decisions: the most possible things that have the largest positive impact on the goals that you have are more important than… everything else. It all ends up being pretty simple. Sometimes you have to spend a bit of time on something that’s locally lower priority if another team depends on it, or if you’re helping someone learn something, but for the most part alignment helps you move toward the right direction.

When teams (and contributors) lack alignment, it’s easy for low priority work to get done, or projects that don’t end up supporting the business goals and so fail to find use (projects fail for other reasons, some of which are expected, so failed projects don’t necessarily indicate miss-alignment). An unaligned team can end up competing with peer teams and internal collaborators. If some parts of a team or organization are well aligned and other’s aren’t, resentment and frustration can brew between teams. Basically, without alignment you can--if you’re lucky--skate by with a little wasted effort, but often alignment deficits are a blight that can threaten a team’s ability to be productive and make it really hard to retain great team members.

Not everything is an alignment problem: teams and projects fail for technical or logistical reasons. Sometimes conflicts emerge between collaborators who are well aligned but working on disconnected projects, or hold different concerns within a project. Alignment is a framework for understanding how organizations can move together and be productive particularly as they grow, and in this I hope that this has been helpful!

I posted something about how I organize my own work, and I touched on “multi-tasking,” and I realized immediately that I had touched something that required a bit more explanation.

I feel like a bit of an outlier to suggest that people spend time learning how to multitask better, particularly when the prevaling conventional wisdom is just “increase focus,” “decrease multitasking,” reduce “context switches,” between different tasks. It’s as if there’s this mythical word where you can just “focus more” taking advantage of longer blocks of time, with fewer distractions, and suddenly be able to get more done.

This has certainly never been true of my experience.

I was, perhaps unsurprisingly, a bit disorganized as a kid. Couldn’t sit still, forgot deadlines, focused inconsistently on things: sometimes I was unstoppable, and sometimes nothing stuck. As an adult, I’ve learned more about myself and I know how to provide the kind of structure I need to get things done, even for work that I find less intrinsically fascinating. Also I drink a lot more caffeine. I’m also aware that with a slightly different brain or a slightly different set of coping strategies, I would struggle a lot more than I do.

There are a lot of reasons why it can be difficult to focus, but I don’t think the why matters much here: thinking pragmatically about how to make the most of the moments we do have, the focus that’s available. Working on multiple things just is, and I think to some extent its a skill that we can cultivate or at least approximate. Perhaps some of the things I do would be useful to you:

• fit your tasks to the attention you have. I often write test code later in the day or during my afternoon slump between 2-3:30, and do more complicated work with my morning coffee between 9:30 and 11:30, and do more writing later in the day. There are different of tasks, and knowing what kinds of work makes sense for which part of the day can be a great help.
• break tasks apart as small as you can do, even if it’s just for yourself. It’s easy to get a little thing done, and bigger tasks can be intimidating. If the units of work that you focus on are the right size it’s possible to give yourself enough time to do the work that you need to do and intersperse tasks from a few related projects.
• plan what you do before you do it, and leave yourself notes about your plan. As I write code I often write a little todo list that contains the requirements for a function. This makes it easy to pick something up if you get interrupted. My writing process also involves leaving little outlines of paragraphs that I want to write or narrative elements that I want to pass.
• leave projects, when possible, at a stopping point. Make it easy for yourself to pick it back up when you’re ready. Maybe this means making sure that you finish writing a test or some code, rather than leaving a function half written. When writing prose, I sometimes finish a paragraph and write the first half of the next sentence, to make it easier to pick up.
• exercise control what and when you do things. There are always interruptions, or incoming mesages and alerts that could require our attention. There are rarely alerts that must cause us to drop what we’re currently working on. While there are “drop everything” tasks sometimes, most things are fine to come back to in a little while, and most emails are safe to ignore for a couple of hours. It’s fine to quickly add something to a list to come back to later. It’s also fine to be disrupted, but having some control over that is often helpful.
• find non-intrusive ways to feel connected. While it should be possible to do some level of multitasking as you work, there are some kind of interruptions that take a lot of attention. When you’re focusing on work, checking your email can be a distraction (say), but it can be hard to totally turn off email while you’re working. Rather than switch to look at my email on some cadence throughout the day, I (effectively,) check my phone far more regularly just to make sure that there’s nothing critical, and can go much longer between looking at my email. The notifications I see are limited, and may messages never trigger alerts. I feel like I know what’s going on, and I don’t get stuck replying to email all day.¹

This is, more or less, what works for me, and I (hope) that there’s something generalizable here, even if we do different kinds of work!

I participated in a great conversation in the #commonlisp channel on libera (IRC) the other day, during which I found a formulation of a familar argument that felt more clear and more concrete.

The question--which comes up pretty often, realistically--centered on adoption of Common Lisp. CL has some great tools, and a bunch of great libraries (particularly these days,) why don’t we see greater adoption? Its a good question, and maybe 5 year ago I would have said “the libraries and ecosystem are a bit fragmented,” and this was true. It’s less true now--for good reasons!--Quicklisp is just great and there’s a lot of coverage for doing common things.

I think it has to do with the connectivity and support at the edges of a project, an as I think about it, this is probably true of any kind of project.

When you decide to use a new tool or technology you ask yourself three basic questions:

1. “is this tool (e.g. language) capable of fulfilling my current needs” (for programming languages, this is very often yes,)
2. “are there tools (libraries) to support my use so I can focus on my core business objectives,” so that you’re not spending the entire time writing serialization libraries and HTTP servers, which is also often the case.
3. “will I be able to integrate what I’m building now with other tools I use and things I have built in the past.” This isn’t so hard, but it’s a thing that CL (and lots of other projects) struggle with.

In short, you want to be able to build a thing with the confidence that it’s possible to finish, that you’ll be able to focus on the core parts of the product and not get distracted by what should be core library functionality, and finally that the thing you build can play nicely with all the other things you’ve written or already have. Without this third piece, writing a piece of software with such a tool is a bit of a trap.

We can imagine tools that expose data only via quasi-opaque APIs that require special clients or encoding schemes, or that lack drivers for common databases, or integration with other common tools (metrics! RPC!) or runtime environments. This is all very reasonable. For CL this might look like:

• great support for gRPC

  There’s a grpc library that exists, is being maintained, and has basically all the features you’d want except support for TLS (a moderately big deal for operational reasons,) and async method support (not really a big deal.) It does depend on CFFI, which makes for a potentially awkward compilation story, but that’s a minor quibble.

  The point is not gRPC qua gRPC, the point is that gRPC is really prevalent globally and it makes sense to be able to meet developers who have existing gRPC services (or might like to imagine that they would,) and be able to give them confidence that whatever they build (in say CL) will be useable in the future.

• compilation that targets WASM

  Somewhat unexpectedly (to me, given that I don’t do a lot of web programming,) WebAssembly seems to be the way deploy portable machine code into environments that you don’t have full control over,¹ and while I don’t 100% understand all of it, I think it’s generally a good thing to make it easier to build software that can run in lots of situation.

• unequivocally excellent support for JSON (ex)

  I remember working on a small project where I thought “ah yes, I’ll just write a little API server in CL that will just output JSON,” and I completely got mired in various comparisons between JSON libraries and interfaces to JSON data. While this is a well understood problem it’s not a very cut and dry problem.

  The thing I wanted was to be able to take input in JSON and be able to handle it in CL in a reasonable way: given a stream (or a string, or equivalent) can I turn it into an object in CL (CLOS object? hashmap?)? I’m willing to implement special methods to support it given basic interfaces, but the type conversion between CL types and JSON isn’t always as straight forward as it is in other languages. Similarly with outputting data: is there a good method that will take my object and convert it to a JSON stream or string? There’s always a gulf between what’s possible and what’s easy and ergonomic.

I present these not as a complaint, or even as a call to action to address the specific issues that I raise (though I certianly wouldn’t complain if it were taken as such,) but more as an illustration of technical decision making and the things that make it possible for a team or a project to say yes to a specific technology.

There are lots of examples of technologies succeeding from a large competitive feild mostly on the basis of having great interoperability with existing solutions and tools, even if the core technology was less exciting or innovative. Technology wins on the basis of interoperability and user’s trust, not (exactly) on the basis of features.