This spring and summer I worked on a new Go library called Amboy that provides a few useful (to my mind!) abstractions and tools around worker pools and job queues, and as its stabilized a bit, I wanted to take a chance to write more about it.

Amboy is really just a side effect of my main project over this time period. I was working on a project to automate the way that we generate Linux package repositories for our software. The procedure itself is pretty straightforward, the challenges are about the number and size of the files to download, the fact that the duration of the operation can take many minutes, and figuring out the best way to handle task overhead and parallelism of these tasks.

Historically these operations have run on a single shared service, to help cut the overhead and make it possible to reduce potential concurrency bugs. While the project made it possible to break the shared service into a bunch of different concurrent/stateless processes, we delayed decisions about the process until quite late in the process, leaving open the idea that we might want to use a shared service.

Enter Amboy.

I wanted to be able to develop and test a system that ran as part of a single local process: something with a command line interface that I could run by hand as needed. But I also wanted to be able to run the same code as a service. If things came to that.

The thing about writing services, is I think it's important to do it right. There had been a service but that's really stretching it: it was a script in a cron job, and you could only run one copy of the script at a time or bad things happened. I've also known lots of "real" applications that don't quite support multiple concurrent instances. As a result I tend to think that applications have to be stateless in order to be "real" and "deplorably."

Amboy is centered around a the idea of "offline" job queues, and provides a collection of tools that address processing, storing, groups of operations. If you define your work unit so that it implements the amboy.Job interface, then you can execute it locally, or you can submit it to a queue that many workers could service. And there's a rest interface and client, or the tools to build the same, as needed.

The best, and perhaps most rewarding is seeing how in subsequent projects there's been an Amboy-shaped aspect that appears pretty naturally, and it's helped other projects, like greenbay a system tool and an idea for a monitoring tool based on greenbay.

I want to write more about the system and some aspects that I hope to develop as needed, but that feels like another issue for another day.