Amboy: Golang Queue Introduction
Amboy is centered around a collection of interfaces: queues to describe "collections of offline work," jobs for "units of work", runners for "task executors." There's also a notion of a "remote queue" which has some additional abstraction around non-local persistence of work.
Then, if you can define your applications core work (e.g. "business logic") in terms of discrete units of work, then you can use these workers and the infrastructure around them to manage most of the work of your application. And the infrastructure around them is pretty useful: you can read some background on this project in an earlier post.
Amboy is two things: first a collection of interfaces that define a simple worker/queue system, and second a collection of simple straightforward example implementations tailored to a few basic use cases.
One: Buildsystems a la Make
A collection of features make it pretty possible to implement a simple Make-like engine using Amboy for all the task processing. The dependency.Manager interface which is a property of every Job, makes it possible to define relationships between jobs, as well as define arbitrary checks to determine if a job should run.
The LocalOrderedQueue uses this dependency manager information, to dispatch jobs to workers ordered by dependencies and prerequisites.
The limitation of this queue is that all tasks must be defined before starting the queue. It's also a "local" queuue, and only stores queues locally, and cannot persist workers. Nevertheless, I'd be interested in developing implementations that would provide the same kind of ordering benefits without these requirements.
Amboy is not a make replacement but it has all the tools needed to build a custom build system.
Two: job.Base for Simple Job Construction
The job.Base struct implements all requireed methods of the Job interface other than the Run method. This means that by including this structure you only need to implement a constructor a Run() method to have a job.
There are cases, when it makes sense to implement custom handlers, particularly (e.g. constraining default SetDependency/Dependency getters/setters,) but for the most part tasks should be easier to define.
As a general rule, I'd treat Run methods like the main methods of your program, mostly to make it easier to test and organize.
Three: Job Centric Application
Many applications are reasonably job-oriented, or have significant components that are offline require asynchronous job processing: downloading large numbers of files, validating resources, generating content from templates, running tests.
I end up writing a fair number of command line tools, which are generally "read a config file and command line operations and then complete a number of tasks, and the return." Often the "number of tasks" in the middle of this translate easily in to a Job, in the Amboy sense.
Four: Queue as Embedded Tool
In a number of cases, it makes sense to keep the worker-queue and job aspects of your application as an implementation detail. I did this a bunch in the bond package, which provides a functional interface, around something (downloading release archives) that's implemented using an amboy queue. It's also core to the way the S3 file syncing tool works inside of curator.
Five: One Queue Many Workers
The remote queue and the queue driver package make it possible to have multiple queue front ends acting on a single queue back end. When the queue back end (e.g. the persistence layer,) is in-memory, this isn't super exciting; however, when that persistence layer is a database, say, the possibilities are more exciting.
The job, then, becomes a way that the application can persist data (though need not be the only unit of persistence,) and indeed jobs themselves might want to access a persistence layer outside of their own internal state, and a way that application instances can attempt to achieve statelessness.
I'm interested in writing additional queue front ends that use the Driver interface: something that manages dependencies, perhaps. I'm also interested in thinking about the locking model, and seeing if there are ways to simplify matters.'
Six: the REST Interface
Initially I thought that applications would have their own rest interface, and clients would make requests that would trigger job creation and execution the server.
That's a nifty idea, but amboy already has this registry and interchange facility to make it possible to serialize jobs, and since jobs are really just structs, and the operations around submitting a job.
So there's a Service (built with gimlet) and a Client which you can use in your own applications for something that might look a lot like RPC.
Jobs, as part of their Type() method have information that s helpful to prevent incompatible clients and services. Having said that, I think these aspects of amboy could use a lot more development, including, better support for SSL/TLS, an implementation of Runner that is implemented on top of one or more remote REST interfaces.