Have a Gimlet: A Go JSON/HTTP API Toolkit

Look folks, I made a thing!

It’s called gimlet, and it’s a Go(lang) tool for making JSON/HTTP APIs (i.e. REST with JSON). Give it a whirl!

It’s actually even less a tool, and more of a toolkit or just “a place to put all of the annoying infrastructure that you’ll inevitably need when you want to build an JSON/HTTP interface, but that have nothing to do what whatever your API/application does: routing, serializing and serializing JSON.

Nothing hard, nothing terribly interesting, and certainly not anything you couldn’t do another way, but, it’s almost certainly true that this layer of application infrastructure is totally orthogonal to whatever you application is actually doing, so you should focus on that, and probaly use something like Gimliet.

Background

I’m using the term HTTP/JSON APIs for services where you send and recive JSON data over HTTP. Sometimes people call these REST APIs, and that’s not inaccurate, but I think REST is a bit more complicated, and not exactly the core paradigm that I’m pursuing with Gimlet.

Sending and reviving JSON over HTTP makes a lot of sense: there are great tools for parsing JSON and HTTP is a decent high level protocol for interprocess communication between simple data applications. Look up “microservices” at your leisure.

Go is a great language for this it has a lot of tooling that anticipates these kinds of applications, and the deployment model is really friendly to operations teams and systems. Also the static

typing and reasonable separation of private and public interfaces is particularly lovely.

So it should be no surprise that there are a lot tools for building stweb applications, frameworks even. They’re great, things like gorilla and negroni are great and provide a very useful set of tools for building Go web apps. Indeed even Gimlet uses components of each of these tools.

The issue, and reason for Gimlet, is that all of these tools assume that you’re building a web application, with web pages, static resources, form handling, session state handling, and other things that are totally irrelevant to writing JSON/HTTP interfaces.

So then, Gimlet is a tool to build these kinds of APIs: simple, uses Negroni and Gorilla’s mux, and does pretty much everything you need except actually write your code.

Example

Set up the app with some basic configuration: :

import "github.com/tychoish/gimlet"

app := gimlet.NewApp()
app.SetPort(9001)
app.SetDefaultVersion(1)

This sets which port the HTTP server is going to listen for requests and configures the default version of the API. You do want all of your endpoints prefixed with “/v<number>” right? The default version of the API is also avalible without the prefix, or if the version of the route is 0. If you don’t set it to 0.

Then register some routes: :

app.AddRoute("/<path>").Version(<int>).Get().Handler(http.HandlerFunc)
app.AddRoute("/<path>").Version(<int>).Post().Handler(http.HandlerFunc)

app.AddRoute returns an API route object with a set of chainable methods for defining the routes. If you add multiple HTTP methods (GET POST and the like,) then Gimlet automatically defines multiple routes with the same handler for each method.

For handlers, I typically just write functions that take arguments from the top level context (database connections, application configuration, etc) and returnhttp.HandlerFunc objects. For example: :

func helloWorld(config *Configuration) http.HandlerFunc {
     return func(w http.ResponseWriter, r *http.Request) {
          input := make(map[string]interface{})
          response := make(map[string]interface{})            

          err := gimlet.GetJSON(input)

          // do stuff here

          gimlet.WriteJSON(w, response)
     }
}

Gimlet has the following functions that parse JSON out of the body of a request, or add JSON output to the body of a response, they are:

  • WriteJSONResponse(w http.ResponseWrite, code int, data interface{})
  • GetJSON(r *http.Request, data interface)

Which read or write data into the interface{} object (typically a struct.) The following three provide consistent response writers for common exit codes:

  • WriteJSON(w http.ResponseWriter, data interface{}) // 200
  • WriteErrorJSON(w http.ResponseWriter, data interface{}) // 400
  • WriteInternalErrorJSON(w http.ResponseWriter, data interface{}) // 500

Finally, when you’ve written your app, kick it all off, with the following: :

err := app.Run()
if err != nil {
   fmt.Println(err)
   os.Exit(1)
}

And that’s it. Enjoy, tell me in the comments or on the issues feed if you find something broken or confusing. Contribution welcome, of course.

A New Era

I’m writing the draft of this from an airplane bound for Ireland for a week of singing. I travel often: taking weekend jaunts to go to folk festivals, singing conventions, Morris dancing tours, and so forth, I don’t really vacation often. I find travel and managing the logistics of being in unfamiliar places stressful, and my idea of a good time has a lot to do with sipping a cup of coffee1 writing something, and reading a book.

Even though I love spending time at home, it’s still important to (sometimes) leave, try new things, and exist somewhere differently for a little while to reset and reflect a bit. With luck, this vacation thing will become something I feel comfortable doing, at least occasionally.

While it certainly wasn’t part of the initial plan, it turns out that this trip is pretty well timed both as denotes a relatively significant change in my life, and I think is a fitting celebration of a period of large changes in my life.


I am no longer a technical writer.

When I return to work, I’m joining the core engineering team to work on build infrastructure and systems projects. I’ll be working on automating our release process, maintaining continuous integration systems, along with an eclectic set of other projects (some of which may involve some of technical writing. There are somethings that you can never escape.)

The truth though, is that this change has been a long time coming: it feels pretty natural. I’ve been working on the documentation build system for a while, and that’s increasingly been the most fun part of my job, so it’ll be good to spend more time doing that kind of work and learn from folks who know more about this kind of thing. Also, build infrastructure and packaging is incredibly important to how people use software and how engineers work, which have been consistent interests of mine for years.

Also, through the last release process, I’ve also found that I’m burning out on writing documentation. I can do it, and I’m not bad at it. After writing, editing and shepherding, more than a million words of documentation (over several thousand pages,) I sometimes feel like I’ve seen it all. I’m interested in seeing the new ideas and perspectives that will prosper in my absence. I’m also eager to see how the foundation I’ve built stands up without me around. It was time.


The decision to change jobs happened rather suddenly. While I’ve built a narrative (see above), in reality, something clicked and I realized it was time. Ten days later there was a plan. I said to myself, “Shit I thought I was done with major life changes for a while.

In the last year, I’ve bought an apartment and moved to Brooklyn and reorganized my local family grouping:2 The good news is that I find my self in a good state, and ’there’s nothing left to change.3

It’s been a rocky year, nothing is going to change that. Even if in retrospect I find myself satisfied with my actions and decisions, and even if I come out the other end better for my struggle, this is a year I wouldn’t care to repeat.

And even so, I’m excited about the future, about continuing to do interesting work professionally, about enjoying my city and local geography, to surround myself with top notch humans, and to make cool things.

These are early days, and there’s work to do.


In the mean time, I’ll be over here, enjoying something different for a little while.


  1. Astute readers of the blog will note that I am historically a tea drinker. I changed to coffee in late June 2014: I discovered that I didn’t mind the taste as much as I thought, and I like a slightly more potent caffeine delivery system. ↩︎

  2. Break ups suck, each in its own special fucked up way. ↩︎

  3. I’ve started singing more tenor, I guess, so maybe there’s more to change after all. ↩︎

Style Chameleon

When I started my current job there were three major problems with the documentation:

  • There was too much duplicated content, so it was difficult to know where to point people.
  • Given that there were always multiple versions of the product in use, it was hard to figure out which paragraph refereed to which version, particularly as the product changed.
  • Each page felt like it was written by someone else (it was!) and the reading experience could be quite jarring.

The first two were huge tasks, but the solutions were pretty straight forward: build system for documentation that was structured such that it could theoretically hold all the information (replace lots of information repositories with a single information repository) and then use maintenance branches in a version control system to snapshot and fork off old branches as they’re released.

Done.

The last problem is harder. Much harder.

I did a pretty good job, at first, of just writing everything myself, which meant that the first drafts all sounded like they were written by one person: because they were. This doesn’t scale. The next step was to edit the hell out of all contributions that weren’t by me.

This also doesn’t scale.

There are some canonical solutions: write a long style guide and try to get people to comply with it; use templates and standard formats for documents so that everything uses common structure and forms. It’s still hard to enforce, but it’s something.

I put a lot of time into content reuse systems that had additional structure. It helps, and reduces some editorial overhead, but has the same weak points as the conventional solution.

At some point, you need actual humans to edit and make sure things are clear and consistent across the entire corpus. If you don’t have the resources for good editors, then either writers have to spend a significant amount of time editing, which is a huge time suck (and slows progress,) or you start to get drift.

I’m not sure that we have a good answer yet. In the mean time

Appendices

Style in Group Processes

  1. As you innovate and improve it’s really hard to resist the impulse to go back to older pieces to revise them. You should make passes through all your content on some sort of schedule, but you have to give yourself permission and allowance to go back and fix style later.
  2. Common style is less about “being right,” and more about figuring out the kind of communication that’s appropriate for the target audience(s) and using effective structure to support them. In short: compromise.
  3. Style is about the entire reading expenses, not just the syntax, or the typesetting: it is both of these things as well as others. For some kinds of texts, the trick is to often to write as few words as possible, and to make it so that people can scan through documents as quickly as possible while only reading the sections that are relevant to them.

Personal Observations

It’s actually interesting to write blogs again, as I’m (re?)discovering a style of writing that I have been very comfortable with in the past but haven’t really exercised recently. It’s also interesting to see how my own writing and writing process has changed as a result of writing so much technical material.

Knitting and Programming

For 2013 and some of 2012 and 2014, I mostly stopped knitting and spent the time teaching myself how to really make software. It wasn’t exactly that intentional in the moment, but time is finite and both knitting and coding require similar kinds of attention and thought. In retrospect this is what happened.

I’ve started knitting again, not as much or in the same ways that I used to, but there’s a bit more space in my brain for additional projects. I’ve thought often about writing about similarities between knitting and programming.

A sweater ontology

A photo posted by Sam Kleinman (@tychoish) on

'

The two activities have much in common. Code and knitting are both highly structured activities, with a reasonably finite syntax of possible component operations. Most programming languages have a few dozen keywords, fundamental types, and built in operations that give rise to all programs (more or less,) and knitting similarly provides a small number of operations (knit, purl, increase, decrease, slip) that you can combine and alter to produce pretty much any kind of fabric.

Knitting patterns are procedural (like programs, generally) and have a standard syntax that shares a number of common elements with contemporary programming languages, including loops, conditionals, map (functional) operations, as well as some non-trivial concurrency.

While each knitting stitch depends on the stitches that happen before and after it, they are all reasonably atomic within the knitted object. At the same time, many knitting projects have lots other objects that happen at the same time, along different dimensions: shaping, stitch patterns, short rows, sleeves/gloves as well as parallel objects like socks. Knitters routinely track multiple things happening at the same time, with some success: a task that many programmers struggle with.

The connection between knitting and programming isn’t terribly surprising or novel: indeed there’s a long history between the development of textile production and technology more broadly, but hand knitting is sort of a technological oddity, so the connections don’t have the same history that you’d have in weaving (say.)


I’ve struggled to write this post for a while and have always had a hard time figuring out what the conclusion is or should be. I think I’ve felt like there should be some potential for synthesis, some way to take the things I’ve learned about software and use that to write programs to support knitting or design work, or vice versa. Increasingly, I think that this is the anti-conclusion.

Sure you can represent knitting patterns using formal languages (such as programming languages,) and it wouldn’t be that hard to write software that converted a pattern, or validated the math, but the math is really the easy part of knitting.

The hard part about knitting things is not figuring out how many stitches you need to knit an object that’s the right size, but rather figuring out what the right size is. It’s easy to know how many stitches you need, and hard to decide how big or small a thing should be. It’s also true that actually knitting things takes the largest amount of time; figuring out the math and making the design decisions are comparatively quick for most projects.

Computers wouldn’t be able to tell how long we should knit the cuffs of our socks or the sleeves of our sweaters, no matter how much code we write.

In any case, I’ll be over here, knitting away at my sweater.

Brunch Plans

There’s been a lot of change in my life in the last six months or a year, and I’ve mostly avoided reflecting about it in writing. I’ve tried writing about it and tried fictionalizing it: I’ve concluded that I’m just not comfortable with my own interpretation of either the last six months or my current state in a way that I could write from. Nevertheless, given the recent calendar roll over, there’s a lot of reflection in the air.

The longer I don’t write, the larger the burden of not writing becomes. Surprisingly, I’ve not been alarmed by the feeling that I am not trustworthy observer of my own life: I’m confident that I am rapidly reaching a place where the present and the future will be easy to write from, particularly on the smaller scale. There’s just this hole in the larger narrative of my life, that I feel obligated to feel.

But maybe it’s alright if I don’t. Or if I don’t just yet.

It also feels fine to start writing about little things that have changed, things that I’m enjoying in my life that are new and different. This will be the first post of this kind, but hopefully not the last.


Today I had four people at my house for brunch.

I made collard greens cooked with a whole onion, an entire head of garlic, and one beef sausage for flavor. I’d never made collards before (it was great). For next time, I definitely need a larger pan, and need more onion and garlic, perhaps adding mushrooms instead of the sausage, which was nice but not noteworthy. Will make again.

I also made oatmeal, of the steal cut variety cooked using the overnight method where you boil the oats in milk and let it simmer for two minutes, and then let it cool and sit over night, and then boil again in the morning and simmer for 10-12 minutes. My modification was to put a couple of tea spoons of olive oil in with the oats at night. I served with blackberry jam and greek yogurt.

And, of course, several pots of coffee. (Chemex, for the win.)

Simple, not a lot of work. Excessively delicious. Nice to have people in my living room, chatting, eating, and enjoying being with people, and a good way to spend time on a weekend.

I used to think of weekend mornings as an opportunity to write or work on my own projects that required a lot of attention and focus. I’m not sure that I really like this way of ritualizing creativity. I also now live in the same neighborhood as a large number of my friends with half a dozen brunch places (or more!) within a mile. Brunch became a comfortable baseline ritual, and something that I’m much more interested in carving out space for.

While I certainly enjoy going somewhere for brunch, there’s something great about having people over and feeling awesome and comfortable about having friends over.


I don’t think I’m a different person now than I was before. I don’t think its fair or correct to narrate the changes in my life as a progress narrative. Lives change, settings change, people change: things are different, and perhaps that’s enough.

Deep Dark Piles

For the last 9 years or so, I’ve kept a copy of a novel that I wrote during my junior year of high school. It’s so crappy: I think I have a Word 95 file of it somewhere, but it mainly exists in this binder, which turns out to be just the right height to use as a foot rest, to keep my feet planted on the floor while I sit up at my desk.

It’s so bad folks. So bad. A few years back I reread the prologue in order to rip off a few things for a different, only slightly less terrible story. The plotting isn’t particularly inventive, the characters are juvenile, and the settings are bland.

I realized the other day that it’s like all of those shell scripts that I wrote when I was learning to program that are hundreds of lines long, that intermingle argument parsing with actual logic, with horrible unabstracted data structures and access patterns, and implementations that are neither clear nor efficient. Those same scripts that I run a few dozen times a day because they just work, and if I don’t think about it too hard, it doesn’t matter much.

There’s also the two hundred thousand words of total crap that I wrote for my last job. And another couple of un-salvageable fiction projects that I didn’t even bother to print. Even with my current job, I sometimes come across awful paragraphs that I know that I must have written a few years ago, but completely fail to understand what I was thinking.

We all, probably have our own examples of the detritus of our past efforts. I spent a long time feeling like these concrete examples of my ineptitude were a burden, a body of work that I had to revise before I could start working on new projects.

Untrue.

I learned a lot from writing these crappy pieces of code, the failed stories, and terrible pieces of work product. Furthermore all of the work I’ve done in these areas since producing the terrible dark piles, has been better, and continues to get better.

Embedded in the idea that these dark piles are a burden is the notion that the production of text, of code is difficult, challenging, and the majority of the work in creation. But this isn’t true. The work is in figuring out the interfaces, the implementation, figuring out the structure and flow of ideas through a system of functions or paragraphs. Typing it all out is the easy and fun part. It is, of course, important to revise and polish work, but sometimes it’s enough to learn from your mistakes, to remind ourselves of our progress, and move on.

Every time I sit down to write, I put my feet on this binder and remember how far I’ve come.

More than anything, I am thankful.

As You Mean to Go On

I’ve not posted for a long time. A lot of things have changed since I last wrote, and this post is probably not the best place to recount all of them. Indeed many things haven’t changed, but the highlights..

I’m still living in New York, but I bought a coop in Brooklyn and moved, which has been great. I’m surprised at how quickly I have felt at home and rooted. The sequence of changes in my life that brought me there are simple, really, but I’ve struggled to make sense of things even so. The fact that I am aware of development is both a great comfort, but it has been hard to write about my life with confidence.

I replaced my tea habit with a coffee habit. I attend yoga classes regularly. I sing Sacred Harp (and sometimes other shaped notes). I Morris dance with the Bowery Boys, the Men’s team in New York City, and continue to dance with Braintrust Morris, an kind of butch Morris team with a spiritual center in the Midwest.

I’m still working on the same kinds of developer documentation, build systems, and software development projects for the same company as before. I still think open source software is important. I am really interested in helping people develop technological literacy and understanding. I want to work on improving infrastructure for developers and development. I use Linux extensively, write a lot of Python, tinker with Go and Common Lisp, and live and breathe in Emacs.

Life is good. Life is difficult. Life is.

I look forward to writing about it here.


So the blog looks different. I changed some things:

I switched from using ikiwiki to using Sphinx and an extension called ablog. I wanted to use a system that I was familiar with and could hack on (that’s Sphinx, which is the core of the tool chain I use at work.) Also, I wanted to use reStructuredText, which I prefer.

Ikiwiki is great software, and I quite enjoy it’s architecture and use. The problem is that it’s not in particularly active development, the code base is in Perl (which I don’t know, and except for ikiwiki, have no reason to learn,) and the project has probably peaked in terms of its adoption curve. To boot, my goals and user story aren’t totally inline it’s goals and user stories.

Sphinx isn’t totally right for a blogging engine either, but it’s solid. I actively develop and maintain tools around Sphinx. Indeed, I was able to make some small changes in ablog that shaved about a third of the time off of the total build time. Not bad.

It took a lot of time to convert all the posts to the new format, but now that everything is in order and the tools are usable it is time to start writing again.


It’s good to be home.

Writing More Again

I haven’t really written very much in the past six months or more, and while I think I’ve done cool things and learned about cool things,

Let me take that back. I haven’t written anything in a sustained sort of way that wasn’t for work. Hell, even what I’ve been doing for work has been smaller and more tactical. Unfortunately you don’t write books, or book-like-objects as small tactical approaches.

So I’ve been trying to figure out how to get back to that. It’s about rhythm more than anything. If you approach time management for log-form rationally, there’s never enough time, so you have to make time, and trick yourself into writing and figure out how to make a little progress on a regular

As someone who definitely tends towards binge writing (and who doesn’t) remembering to write a little bit on a regular basis is hard and nearly counter intuitive. So as I’ve been attempting to restart the writing habit, I’ve been thinking about what I’ve done in the past that’s worked to keep up the momentum and work on projects. I’ve uncovered:

Reading

Even more important, I think, than reestablishing a habig of writing regularly, is restablishing the practice of reading regularly. I often get hung up on the fact that I don’t think I’m a very fast or very through reader. But reading is quite inspiring, and I often find that the more I read, the more I want to write.

So I’ve been reading the Vorkosigan novels, and it’s been great.

Tracking Word Counts

I like having some record of my progress, and I’ve taken different approaches to tracking progress, mostly using word counts, over time.

When you’re writing longer pieces, particularly in editors that don’t reflect word counts or page numbers in the interface, it’s easy to loose context for how much progress you’re making.

Way back when, I used to record the current page and word count for all of my projects in a note boot, and this developed into a rather incurable tick to mash a few keys down every few sentences to check the word count. Then, three years ago, I wrote a script to check the word count of all my projects on a regular interval.

I stopped really using it about a year ago, because the script handled different branches (in git) of the same project really poorly, and there wasn’t a good way to hack that in.

So I recently rewrote this program, and I rather like it. I’ll post more about it soon.

Physical Activity and Care

To do writing one definitely needs to spend a serious amount of time in front of a computer typing. Unless you really like notebooks and pens, there’s no other way to get things written.

But you can’t take this to the extreme: if all you do is sit in front of your computer and stare at cursor waiting for inspiration to strike. As people we need different kinds of focus, and different kinds of experiences, on the small scale, to keep the generative impulses functional.1

Reading voraciously, as above, is a key part of being able to write effectively, but other kinds of activity are also useful. When I’m not writing, my first instinct is to try and find more time to write, but sometimes--often--what I need more is more time for things that aren’t writing: exercise, reading, and so forth.

I’ve started to do yoga, as I’ve written before, a few times a week. I need to also remember to set aside time for other non-writing activities: more exercise, walks, dinners with friends, dates with my email backlog, etc.

Onward and Upward!


  1. I like the idea of generative impulses rather than creative muscles. Writing isn’t always creative, but it is, like any other kind of production, work, and always requires energy and effort. ↩︎

Fabric Complexity

I’ve been a big fan of fabric which is a Python tool for deployment orchestration: deploying a new release of an application, uploading files, deploying new configurations to a group of hosts in a cluster. Before fabric, the options were either to write fragile shell scripts that often didn’t do a good job of handling multiple hosts, or use more heavy weight configuration management tools, which had a lot of overhead and bother.

Fabric is great. Make a python module called “fabfile,” write idiomatic Python, stick anywhere, and then you can call functions in that module using the fab command. And it includes a lot of helpers for interacting with groups of hosts and dealing with remote and local resources that is designed for takes into consideration common contemporary application development.

For a while, I basically thought that fabric, or tools like it would completly obviate the need for shell scripting

I think there are two major problems with using fabric as a framework for general purpose scripting:

  1. Global state can get messy.

    This isn’t only an issue with Fabric, but I think the fab solution isn’t great here. Fabric has this envy object that you can import and then use to track the settings or set custom “global” data values. It’s better than having actual global data, maybe, but you end up having to use it too much, particularly it for communication between different related operations and you end up using tasks and global state to run-time configuration.

  2. There’s no real overriding abstraction to guide code organization.

    In a lot of ways fab scripts are just a way of writing a large number of scripts within a single code base and a command line interface for specifying different entry points. It would be nice to be able to express relationships between tasks, or centralize the registry of tasks in some of non-adhoc.

I actually don’t think that these are problems with fabric, itself, but I think it’s probably the case that we need a few different kinds of tools and patterns to help support the authorship of useful and maintainable shell scripts and build systems.

  • a good, usable, non-procedural way manage argument parsing.

    In python, I like argprase and think it works really well, so I’m not saying we need some set of tools to actually parse the arguments that people input on the command line. But I think when you’re writing a script, argparse is too low level, and pretty soon there are a few hundred lines of declarative code, and you still have to connect this up with the actual code, which leads to maintenance headaches and all sorts of dumb errors.

  • a good framework for expressing multi-stage tasks and their dependencies and relationships.

    This may basically be waf, but I think we can do something similar. I made an attempt at this with buildcloth but I think I ended up with something that’s too much of a (meta) build automation tool, and not enough of a general purpose paradigm for organizing code to perform related tasks in a clean, modular, and well organized sort of way.

These aren’t fabric specific problems, or Python-specific problems, of course. But they are particularly relevant to Python moderatly sized Python code bases in general and most fabric implementations that I’veb seen. Maybe Python and fabric make writing useful code easy enough that we start to see complexity problems earlier than we would otherwise. Not sure.

As you may have been able to guess at this point, I am somewhat burdened by a large and somewhat overgrown fabric-based project, that needs refactoring. Thankfully, matters are not critical, but I do want to figure out the right way to solve this kind of problem before I start hacking.

Best Practice Practices

At work people often ask for more “best practices” guides. In some ways this is sign of success: they’re no longer begging for fundamental reference material and descriptions of basic use. Nevertheless I almost always wince:

1. “Best practices” carries an implicit sense of guarantee along the lines of “if you adhere to the best practices, then you won’t run into problems,” which is sort of difficult to assert with confidence, and is really a product design issue, not a documentation issue.

  1. It is really hard to make one-size-fits-all recommendations.

3. Best practice-guides don’t address actual needs of users, or actual solutions to real problems, because they generalize problems and solutions beyond the point of re-usability.

This isn’t to say that it’s difficult or impossible to give recommendations, and indeed documentation is the best way that the purveyors of software have to to shape practices. However, the documentation should guide users towards better practices everywhere and isolating recommendations into certain types of documents is probably counter productive.

Onward and Upward!