The free software/open source/software development world needs a distributed bug tracking story. Because the current one sucks.

::: {.contents} :::

The State of the Art

There are a number of tools written between 2006 and 2010 or so that provide partial or incomplete solutions to the problem. Almost isn’t quite good enough. The “resources” section of this post, contains an overview of the most important (my judgment,) representatives of the current work in the area with a bit of editorializing.

In general these solutions are good starts, and I think they allow us (or me) a good starting point for thinking about what distributed bug tracking could be like. Someday.

Bug tracking needs are diverse, which creates a signifigant design challenge for any system in this space. There are many existing solutions, that everyone hates, and I suspect most would-be developers and innovators in the space would like to avoid opening this can of worms.

Another factor is that, while most people have come to the conclusion that distributed source control tools are the “serious” contemporary tool for managing source code the benefits of distributed bug tracking hasn’t yet propogated in the same way. Many folks have begun to come to terms with the fact that some amount of tactical centralization is inevitable, required, and even desirable¹ in the context of a issue tracking systems.

Add to this the frequent requirement that non-developer users often need to track and create issues, and the result is that we’ve arrived at something of an impasse.

Requirements

A distributed bug tracking system would need:

• A good way to provide short, unique identifiers for individual issues and comments so that users can discuss issues canonically.

• An interface contained in a single application, script, or binary, that you could distribute with the application.

• A simple/lightweight web-based interface so that users can (at least) review, search, and reference issues from a web browser.

  Write access would also be good, but is less critical. Also, it might be more practical (both from a design and a workflow perspective,) to have users submit bugs on the web into a read-only “staging queue,” that developers/administrators would then formally import into the project. This formalizes a certain type of triage approach that many projects may find useful.

• To be separable from the source code history, either by using a branch, or by using pre-commit hooks to ensure that you never commit changes to code/content and the bugs at the same time.

• To be editable, and to interact with commonly accessible tools that users already use. Email, command line tools, the version control systems, potentially documentation systems, build systems, testing frameworks and so forth.

• Built on reliable tools.²

• To provide an easy way to customize your “views” on bugs for a particular team or project. In other words, each team can freely decide which extra fields get attached to their bugs, along with which fields are visible by default, which are required, and so on--without interfering with other projects.

The Future of the Art

1. We (all) need to work on building new and better tools to help solve the distributed issue tracking problem. This will involve:
   • learning from the existing attempts,
   • continuing to develop and solidify the above requirements,
   • (potentially) test and develop a standard (yaml/json?) based data storage format that is easy to parse, and easily merged that multiple tools can use.
   • Develop some simple prototype tools, potentially as a suite of related utilities (a la early versions of git.) that facilitate interaction with the git database. With an eye towards flexibility and extensible.
2. While there are implications for free software hosting as well as vendor independence and network service autonomy (a la `Franklin Street Statement <http://autonomo.us/2008/07/franklin-street-statement/>`_.) I think the primary reason to pursue distributed bug tracking has more to do with productivity and better engineering practices, and less with the policy. In summary:
   • Bug database systems that run locally and are fast³ and always available.
   • Tools that permit offline interaction with issue database.
   • Tools that allow users to connect issues to branches.
   • Tools that make it possible to component-ize bug databases in parallel with software

Resources

(With commentary,)

• dist-bugs mailing list

  This is the canonical source for discussion around distributed bug tracking.

• Bugs Everywhere

  This is among the most well developed solution speaking holistically. “be” is written in Python, can generate output for the web. It uses its own data format, and has a pretty good command line tool. The HTML output generate is probably not very fast at scale (none are,) but I have not tested it.

• Ditz

  Ditz is a very well developed solution. Ditz: implemented in Ruby, has a web interface, has a command line tool, uses a basic YAML data format, and stores data in branch. Current development is slow, getting it up and running is non-trivial, and my sense is that there isn’t a very active community of contributors. There are reasons for this, likely but they are beyond the scope of this overview.

• pitz

  Pitz is a Python re-implementation of Ditz, and while the developer(s?) have produced a “release,” the “interface” is a Python shell, and to interact with the database you have to, basically write commands in Python syntax. From a data perspective, however, Pitz, like Ditz is quite developed. Pitz while it stores data in-tree, I think it’s important source of ideas/examples/scaffolding.

• Artemis

  This is a really clever solution that uses Maildirs to store issues. As a result you can interact with and integrate Artimis issues with your existing email client. Pull down changes, and see new bugs in your email, without any complicated email and list server setups.

  The huge caveat is that it’s implemented as a plugin for Mercurial, and so can’t be used with git projects. Also, all data resides in the tree.

• git-issues

  In most ways, git-issues is my favorite: it’s two Python files, 1700 lines of code, stores issues outside of the source branch, and has a good command line interface. On the downside, it uses XML (which shouldn’t matter, but I think probably does, at least in terms of attracting developers,) and doesn’t have a web-based interface. It’s also currently un-maintained.

• Prophet/sd

  SD, which is based on a distributed database named Prophet, is a great solution. The primary issue is that it’s currently unmentioned and is not as feature complete as it should be. Also a lot of SD focuses on synchronizing with existing centralized issue trackers, potentially at the expense of developing other tools.

Cron is great, right? For the uninitiated, if there are any of you left, Cron is a task scheduler that makes it possible to run various scripts and programs at specified intervals. This means that you can write programs that “do a thing” in a stateless way, set them to run regularly, without having to consider any logic regarding when to run, or any kind of state tracking. Cron is simple and the right way to do a great deal of routine automation, but there are caveats.

At times I’ve had scads of cron jobs, and while they work, from time to time I find myself going through my list of cron tasks on various systems and removing most of them or finding better ways.

The problems with cron are simple:

• Its often a sledge hammer, and it’s very easy to put something in cron job that needs a little more delicacy.

• While it’s possible to capture the output of cron tasks (typically via email,) the feedback from cronjobs is hard to follow. So it’s hard to detect errors, performance deterioration, inefficiencies, or bugs proactively.

• Its too easy to cron something to run every minute or couple of minutes. A task that seems relatively lightweight when you run it once can end up being expensive in the aggregate when they have to run a thousand times a day.

  This isn’t to say that there aren’t places where using cron isn’t absolutely the right solution, but there are better solutions. For instance:

• Include simple tests and logic for the cron task to determine if it needs to run before actually running.

• Make things very easy to invoke or on demand rather than automatically running them regularly.

  I’ve begun to find little scripts and dmenu, or an easily called emacs-lisp function to be preferable to a cron job for a lot of tasks that I’d otherwise set in a cron job.

• Write real daemons. It’s hard and you have to make sure that they don’t error out or quit unexpectedly--which requires at least primitive monitoring--but a little bit of work here can go a long way.

Onward and Upward!

Most people don’t use git particularly well. It’s a capable piece of software that supports a number of different workflows, but because it doesn’t mandate any particular workflow it’s possible to use git productively for years without ever really touching some features.

And some of the features--in my experience mostly those related to more manual branching, merging, and history manipulation operations--are woefully underutilized. Part of this is because Github, which is responsible for facilitating much of git’s use, promotes a specific workflow that makes it possible to do most of the (minimal required) branch operations on the server side, with the help of a much constrained interface. Github makes git usable by making it possible to get most of the benefit of git without needing to mess with SHA1 hashes, or anything difficult on the command-line.

That’s a good thing. Mostly.

Nevertheless, there are a few operations that remain hard with git: I sometimes encounter situations that I have to try a few times before I get it right, and there are commands that I always have to check the man page to figure out how to specify the references. And even then I’m sometimes still confused. So maybe I (or we?) can spend a little bit of time and figure out what processes remain hard with git and maybe try and see if there is a way to make the process a bit more streamlined.

Here’s my list:

• Reorder all commits since x commit.

  This is basically: find the commit before the earliest one that you want to change, run git rebase -i <commit hash> to reorder the commits even though git sorts the commits in the order that I find most un-intuitive.

• Create local branches to track remote branches or repositories.

  Setup the remotes, if necessary, and then run: git branch --track <local-branch-name> <remote>/<branch-name> and git config branch.{name}.push {local-branch}:master.

• Stash all local changes and switch branches.

  It would also be nice if you could figure out way for git (or a helper) to see any open files in your text editor and save/close them if needed.

• Pull a commit from the history of one branch into another branch without pulling anything else.

  I think this is chery-pick? It might also be nice to pull a series of commits from one branch, rebase them into one commit in the destination branch, and then commit that.

• Pretty much every time I’ve tried to use the merge command to get something other than what I would have expected to happen by using “pull,” it ends tragically.

Reader suggestions:

• Put your process/procedural frustrations with git here.

How about we work on figuring out how to solve these problems in comments?

I’m sure that most “hacker bloggers” have probably done their own “N Git Tricks,” post at this point. But git is one of those programs that has so much functionality and everyone uses it differently that there is a never ending supply of fresh posts on this topic. My use of git changes enough that I could probably write this post annaully and come up with a different 9 things. That said here’s the best list right now.

::: {.contents} :::

See Staged Differences

The git diff command shows you the difference between the last commit and the state of the current working directory. That’s really useful and you might not use it as much as you should. The --cached option shows you just the differences that you’ve staged.

This provides a way to preview your own patch, to make sure everything is in order. Crazy useful. See below for the example:

git diff --cached

Eliminate Merge Commits

In most cases, if two or more people publish commits to a shard repository, and everyone commits to remote repositories more frequently then they publish changes, when they pull, git has to make “meta commits” that make it possible to view a branching (i.e. “tree-like”) commit history in a linear form. This is good for making sure that the tool works, but it’s kind of messy, and you get histories with these artificial events in them that you really ought to remove (but no one does.) The “--rebase” option to “git pull” does this automatically and subtally rewrites your own history in such a way as to remove the need for merge commits. It’s way clever and it works. Use the following command:

git pull --rebase

There are caveats:

• You can’t have uncommitted changes in your working copy when you run this command or else it will refuse to run. Make sure everything’s committed, or use “git stash”
• Sometimes the output isn’t as clear as you’d want it to be, particularly when things don’t go right. If you don’t feel comfortable rescuing yourself in a hairy git rebase, you might want to avoid this one.
• If the merge isn’t clean, there has to be a merge commit anyway I believe.

Amend the Last Commit

This is a recent one for me..

If you commit something, but realized that you forgot to save one file, use the “--amend” switch (as below) and you get to add whatever changes you have staged to the previous commit.

git commit --amend

Note: if you amend a commit that you’ve published, you might have to do a forced update (i.e. git push -f) which can mess with the state of your collaborators and your remote repository.

Stage all of Current State

I’ve been using a versing of this function for years now as part of my download mail scheme. For some reason in my head, it’s called “readd.” In any case, the effect of this is simple:

• If a file is deleted from the working copy of the repository, remove it (git rm) from the next commit.
• Add all changes in the working copy to the next commit.

git-stage-all(){
   if [ "`git ls-files -d | wc -l`" -gt "0" ]; then; git rm --quiet `git ls-files -d`; fi
   git add .
}

So the truth of the matter is that you probably don’t want to be this blasé about commits, but it’s a great time saver if you use the rm/mv/cp commands on a git repo, and want to commit those changes, or a have a lot of small files that you want to process in one way and then snapshot the tree with git.

Editor Integration

The chances are that your text editor has some kind of git integration that makes it possible to interact with git without needing to drop into a shell.

If you use something other than emacs I leave this as an exercise for the reader. If you use emacs, get “magit,” possibly from your distribution’s repository, or from the upstream.

As an aside you probably want to add the following to your .emacs somewhere.

(setq magit-save-some-buffers nil)
(add-hook 'before-save-hook 'delete-trailing-whitespace)

Custom Git Command Aliases

In your user account’s “~/.gitconfig” file or in a per-repository “.git/config” file, it’s possible to define aliases that add bits of functionality to your git command. This is useful defining shortcuts, combinations, and for triggering arbitrary scripts. Consider the following:

[alias]
all-push  = "!git push origin master; git push secondary master"
secondary = "!git push secondary master"

Then from the command line, you can use:

git secondary
git all-push

Git Stash

“git stash” takes all of the staged changes and stores them away somewhere. This is useful if you want to break apart a number of changes into several commits, or have changes that you don’t want to get rid of (i.e. “git reset”) but also don’t want to commit. “git stash” puts staged changes onto the stash and “git stash pop” applies the changes to the current working copy. It operates as a FILO stack (e.g. “First In, Last Out”) stack in the default operation.

To be honest, I’m not a git stash power user. For me it’s just a stack that I put patches on and pull them off later. Apparently it’s possible to pop things off the stash in any order you like, and I’m sure I’m missing other subtlety.

Everyone has room for growth.

Ignore Files

You can add files and directories to a .gitignore file in the top level of your repository, and git will automatically ignore these files. One “ignore pattern” per line, and it’s possible to use shell-style globing.

This is great to avoid accidentally committing temporary files, but I also sometimes put entire sub-directories if I need to nest git repositories within git-repositories. Technically, you ought to use git’s submodule support for this, but this is easier. Here’s the list of temporary files that I use:

.DS_Store
*.swp
*~
\#*#
.#*
\#*
*fasl
*aux
*log

Host Your Own Remotes

I’ve only once accidentally said “git” when I meant “github” (or vice versa) once or twice. With github providing public git-hosting services and a great compliment of additional tooling, it’s easy forget how easy it is to host your own git repositories.

The problem is that, aside from making git dependent on one vendor, this ignores the “distributed” parts of git and all of the independence and flexibility that comes with that. If you’re familiar with how Linux/GNU/Unix works, git hosting is entirely paradigmatic.

Issue the following commands to create a repository:

mkdir -p /srv/git/repo.git
cd /srv/git/repo.git
git init --bare

Edit the .git/config file in your existing repository to include a remote block that resembles the following:

[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = [username]@[hostname]:/srv/git/repo.git

If you already have a remote named origin, change the occurrence of the word remote in the above snippet with the name of your remote. (In multi-remote situations, I prefer to use descriptive identifier like “public” or machine’s hostnames.)

Then issue “git push origin master” on the local machine, and you’re good. You can us a command in the following form to clone this repository at any time.

git clone [username]@[hostname]:/srv/git/repo.git

Does anyone have git tricks that they’d like to share with the group?

A couple of years ago I wrote “Why Arch Linux Rocks” and “Getting the most from Arch Linux.” I’ve made a number of attempts to get more involved in the Arch project and community, but mostly I’ve been too busy working and using Arch to do actual work. Then a few weeks ago when I needed to do something minor with my system--I forget what--and I found myself thinking “this Arch thing is pretty swell, really.”

This post is a collection of the clever little things that make Arch great.

::: {.contents} :::

abs

I’m using abs as a macro for all of the things about the package build system that I enjoy.

Arch packages are easy to build for users: you download a few files read a bash script in the PKGBUILD file and run the makepkg command. Done. Arch packages are also easy to specify for developers: just specify a “build()” function and some variables int eh PKGBUILD file.

Arch may not have as many packages as Debian, but I think it’s clear that you don’t need comprehensive package coverage when making packages is trivially easy.

If you use Arch and you don’t frequent that AUR, or if you ever find yourself doing “./configure; make; make install” then you’re wasting your time or jeopardizing the stability of your server.

yaourt

The default package management tool for Arch Linux, pacman, is a completely sufficient utility. This puts pacman ahead of a number of other similar tools, but to be honest I’m not terribly wild about it. Having said that, I think that yaourt is a great thing. It provides a wrapper around all of pacman’s functionality and adds support for AUR/ABS packages in a completely idiomatic manner. The reduction in cost of installing this software is quite welcome.

It’s not “official” or supported, because it’s theoretically possible to really screw up your system with yaourt but if you’re cautious, you should be good.

yaourt -G

The main yaourt functions that I use regularly are the “-Ss” which provides a search of the AUR, and the -G option. -G just downloads the tarball with the package specification (e.g. the PKGBUILD and associated files) from the AUR and untars the archive into the current directory.

With that accomplished, it’s trivial to build and install the package, but you get to keep a record of the build files for future reference and possible tweaking. So basically, you this is the way to take away the tedium of getting packages from the AUR, while giving you more control and oversight of package installation.

rc.conf

If you’ve installed Arch, then you’re already familiar with the rc.conf file. In case you didn’t catch how it works, rc.conf is bash script that defines certain global configuration values, which in turn controls certain aspects of the boot process and process initialization.

I like that it’s centralized, that you can do all kinds of wild network configuration in the script, and I like that everything is in one place.

netcfg

In point of fact, one of primary reasons I switched to Arch Linux full time, was because of the network configuration tool, netcfg. Like the rc.conf setup, netcfg works by having a network configuration files which define a number of variables which are sourced by netcfg when imitating a network connection.

It’s all in bash, of course, and it works incredibly well. I like having network management easy to configure, and setup in a way that doesn’t require a management daemon.

Init System

Previous points have touched on this, but the “BSD-style” init system is perfect. It works quickly, and boot ups are stunningly fast: even without an SSD I got to a prompt in less than a minute, and probably not much more than 30 seconds. With an SSD: it’s even better great. The points that you should know:

• Daemon control scripts, (i.e. init scripts) are located in /etc/rc.d. There’s a pretty useful “library” of shell functions in /etc/rc.d/function and a good template file in``/etc/rc.d/skel` for use when building your own control scripts. The convention is to have clear and useful output and easy to understand scripts, and with the provided material this is pretty easy.

• In /etc/rc.conf there’s a DAEMON variable that holds an array. Place names, corresponding to the /etc/rc.d file name, of daemons in this array to start them at boot time. Daemons are started synchronously by default (i.e. order of items in this array matters and the control script must exit before running the next script.) However, if a daemon’s name is prefixed by an @ sign, the process is started in the background and the init process moves to the next item in the array without waiting.

  Start-up dependency issues are yours to address, but using order and background start-up this is trivial to implement. Background start ups lead to fast boot times.

I added a few lines to a script that I use to build my task list, and for the first time ever, I opened a file with code in it, added a feature, tested it, and it worked. Here’s the code with enough context so it makes sense (explained later if you don’t want to spend the time parsing it:)

ARG=`echo "$@" | sed -r 's/\s*\-[c|p|s]\s*//g'`
WIKI_DIR="`echo $ARG | cut -d " " -f 1`"
if [ "`echo $ARG | cut -d " " -f 2 | grep -c /`" = 1 ]; then
   TODO_PAGE="`echo $ARG | cut -d " " -f 2`"
elif [ "`echo $ARG | cut -d " " -f 2 | grep -c $EXT`" = 1 ]; then
   TODO_PAGE="$WIKI_DIR/`echo $ARG | cut -d " " -f 2`"
else
   TODO_PAGE="$WIKI_DIR/`echo $ARG | cut -d " " -f 2`.$EXT"
fi

This is from the section of the script that processes the arguments and options on the command line. Previously, commands were issued such that:

ikiwiki-tasklist [-c -p -s] [DIR_TO_CRAWL] [OUTPUT TODO FILE NAME]

My goal with the options was to have something that “felt like” a normal command with option switches and had a lot of flexibility. The two fields that followed: however, I didn’t provide as much initial flexibility. The directory to crawl for tasks (i.e. “[DIR_TO_CRAWL]”) was specified the way it is now, but the output file was 1) assumed to have an extension specified in a variable at the top of the script, 2) automatically placed the output file in the top level of the destination directory.

It worked pretty well, but with the advent of a new job I realized that I needed some compartmentalization. I needed to fully use the tasklist system for personal and professional tasks without getting one set of items mixed in with the other. Being able to have better control of the output is key to having full control over this.

The modification detects if the output file looks like a path rather than a file name. If it’s senses a path, it creates the task list in the path specified, with no added extension. If a file name specifies the extension, then you won’t get “.ext.ext” files. And the original behavior is preserved.

I’m a hacker by inclination: I take code that I find and figure out how to use it. Sometimes I end up writing or writing code, but I’m not really a programmer. My own code, at least until recently has tended to be somewhat haphazard and until now (more or less) I’ve not felt like I could write code from scratch that was worth maintaining and enhancing in any meaningful way.

Apparently some of that’s changed.

I’ve made a few additional changes to the scripts, but most of these feel more trivial and can be described as “I learned how to write slightly tighter shell scripts. so if you’re using it you might want to update: the ikiwiki tasklist page is up to date.

This post is mostly just an overview of Epistle by Matteo Villa, which is--to my mind--the best Android note taking application ever. By the time you read this I will have an Android Tablet, but it’s still in transit while you read this and that’s a topic that dissevers it’s own post.

Epistle is a simple notes application with two features that sealed the deal:

1. It knows markdown, and by default provides a compiled rich text view of notes before providing a simple notes editing interface. While syntax highlighting would be nice, we’ll take what we can get.

2. It’s a nice, simple application. There’s nothing clever or fancy going on. This simplicity means that the interface is clean and it just edits text.

For those on the other side there’s Paragraft that seems similar. While in my heart of hearts I’m probably still holding out for the tablet equivalent¹ of emacs. In the mean time, I think developing a text editing application that provide a number of paradigmatic text editing features and advances for the touch screen would be an incredibly welcome development.

In the end there’s much work to be done, and the tools are good enough to get started.

Like issue tracking systems, documentation publication systems are never quite perfect. There are dozens of options, and most of them are horrible and difficult to use for one reason or another. Rather than outline why these systems are less than ideal, I want to provide a list of basic requirements that I think every documentation publishing system¹ should have.

Requirements

• Tag System. You have to be able to identify and link different pieces of content together in unique and potentially dynamic ways across a number of dimensions. Tagging systems, particularly those that can access and create lists of “other posts with similar tags,” are essential for providing some much needed organization to projects that are probably quite complex. Tagging systems should provide some way of supporting multiple tag namespaces. Also operations affecting tags need to be really efficient, from the users and software’s perspective, or else it won’t work at realistic scales.
• Static Generation. Content needs to be statically generated. There are so many good reasons to have static repositories. It allows you to plan releases (which is good if you need to coordinate documentation releases with software releases) most documentation changes infrequently. The truth is this feature alone isn’t so important, but static generation makes the next several features possible or easier.
• Development Builds. As you work on documentation, it’s important to be able to see what the entire resource will look like when published. This is a mass-preview mode, if you will. The issue here, is that unlike some kinds of web-based publications, documentation often needs to be updated in batches, and it’s useful to be able to see those changes all at once because the can all interact in peculiar ways. These test builds need to be possible locally, so that work isn’t dependent on a network connection, or shared infrastructure.
• Verification and Testing. While building “self-testing” documentation is really quite difficult (see also /technical-writing/dexy,) I think publication systems should be able to do “run tests” against documents and provide reports, even if the tests are just to make sure that new of software versions haven’t been released, or that links still work. It’s probably also a good idea to be able to verify that certain conventions are followed in terms of formatting: trailing white space, optional link formats, tagging conventions, required metadata, and so forth.
• Iteration Support. Documents need to be released and revised on various schedules as new versions and products are developed. Compounding this problem, old documentation (sometimes,) needs to hang around for backwards compatibility and legacy support. Document systems need to have flexible ways to tag documents as out of date, or establish pointers that say “the new version of this document is located here.” It’s possible to build this off of the tag system, but it’s probably better for it to be a separate piece of data.
• Version Control. These systems are great for storing content, facilitating easy collaboration, and supporting parallel work. Diffs are a great way to provide feedback for writers, and having history is useful for being able to recreate and trace your past thinking when you have to revisit a document or decision weeks and months later.
• Lightweight Markup. It’s dumb to make people write pure XML in pretty much every case. With rst, markdown, and pandoc the like there’s no reason to write XML. Ever. of story End.
• Renaming and Reorganization. As document repositories grow and develop, it seems inevitable that the initial sketch of the organization for the body of work change. Documents will need to be moved, URLs will need to be redirected or rewritten, and links will need to be updated. The software needs to support this directly.
• Workflow Support. Documentation systems need to be able to facilitate editorial workflows and reviews. This should grow out of some combination of a private tag name space and a reporting feature for contributions, which can generate lists of pages to help groups distribute labor and effort.

This might just be a quirk of my approach, but I tend approach documentation, terms of process and tooling, as if it were programming and writing software. They aren’t identical tasks, of course, but there are a lot of functional similarities And definitely enough to take advantage of the tooling and advances (i.e. make, git, etc.) that programmers have been able to build for themselves. Am I missing something or totally off base?

Sorry for the lame title. I was thinking the other day, about how awesome SSH is, and how it’s probably one of the most crucial pieces of technology that I use every single day. Here’s a list of 10 things that I think are particularly awesome and perhaps a bit off the beaten path.

Update: (2011-09-19) There are some user-submitted ssh-tricks on the wiki now! Please feel free to add your favorites. Also the hacker news thread might be helpful for some.

SSH Config

I used SSH regularly for years before I learned about the config file, that you can create at ~/.ssh/config to tell how you want ssh to behave.

Consider the following configuration example:

Host example.com *.example.net
User root
Host dev.example.net dev.example.net
User shared
Port 220
Host test.example.com
User root
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
Host t
HostName test.example.org
Host *
Compression yes
CompressionLevel 7
Cipher blowfish
ServerAliveInterval 600
ControlMaster auto
ControlPath /tmp/ssh-%r@%h:%p

I’ll cover some of the settings in the “Host *” block, which apply to all outgoing ssh connections, in other items in this post, but basically you can use this to create shortcuts with the ssh command, to control what username is used to connect to a given host, what port number, if you need to connect to an ssh daemon running on a non-standard port. See “man ssh_config” for more information.

Control Master/Control Path

This is probably the coolest thing that I know about in SSH. Set the “ControlMaster” and “ControlPath” as above in the ssh configuration. Anytime you try to connect to a host that matches that configuration a “master session” is created. Then, subsequent connections to the same host will reuse the same master connection rather than attempt to renegotiate and create a separate connection. The result is greater speed less overhead.

This can cause problems if you’ want to do port forwarding, as this must be configured on the original connection, otherwise it won’t work.

SSH Keys

While ControlMaster/ControlPath is the coolest thing you can do with SSH, key-based authentication is probably my favorite. Basically, rather than force users to authenticate with passwords, you can use a secure cryptographic method to gain (and grant) access to a system. Deposit a public key on servers far and wide, while keeping a “private” key secure on your local machine. And it just works.

You can generate multiple keys, to make it more difficult for an intruder to gain access to multiple machines by breaching a specific key, or machine. You can specify specific keys and key files to be used when connected to specific hosts in the ssh config file (see above.) Keys can also be (optionally) encrypted locally with a pass-code, for additional security. Once I understood how secure the system is (or can be), I found my self thinking “I wish you could use this for more than just SSH.”

SSH Agent

Most people start using SSH keys because they’re easier and it means that you don’t have to enter a password every time that you want to connect to a host. But the truth is that in most cases you want to have unencrypted private keys that have meaningful access to systems because once someone has access to a copy of the private key the have full access to the system. That’s not good.

But the truth is that typing in passwords is a pain, so there’s a solution: the ssh-agent. Basically one authenticates to the ssh-agent locally, which decrypts the key and does some magic, so that then whenever the key is needed for the connecting to a host you don’t have to enter your password. ssh-agent manages the local encryption on your key for the current session.

SSH Reagent

I’m not sure where I found this amazing little function but it’s great. Typically, ssh-agents are attached to the current session, like the window manager, so that when the window manager dies, the ssh-agent loses the decrypted bits from your ssh key. That’s nice, but it also means that if you have some processes that exist outside of your window manager’s session (e.g. Screen sessions) they loose the ssh-agent and get trapped without access to an ssh-agent so you end up having to restart would-be-persistent processes, or you have to run a large number of ssh-agents which is not ideal.

Enter “ssh-reagent.” stick this in your shell configuration (e.g. ~/.bashrc or ~/.zshrc) and run ssh-reagent whenever you have an agent session running and a terminal that can’t see it.

ssh-reagent () {
  for agent in /tmp/ssh-*/agent.*; do
      export SSH_AUTH_SOCK=$agent
      if ssh-add -l 2>&1 > /dev/null; then
         echo Found working SSH Agent:
         ssh-add -l
         return
      fi
  done
  echo Cannot find ssh agent - maybe you should reconnect and forward it?
}

It’s magic.

SSHFS and SFTP

Typically we think of ssh as a way to run a command or get a prompt on a remote machine. But SSH can do a lot more than that, and the OpenSSH package that probably the most popular implementation of SSH these days has a lot of features that go beyond just “shell” access. Here are two cool ones:

SSHFS creates a mountable file system using FUSE of the files located on a remote system over SSH. It’s not always very fast, but it’s simple and works great for quick operations on local systems, where the speed issue is much less relevant.

SFTP, replaces FTP (which is plagued by security problems,) with a similar tool for transferring files between two systems that’s secure (because it works over SSH) and is just as easy to use. In fact most recent OpenSSH daemons provide SFTP access by default.

There’s more, like a full VPN solution in recent versions, secure remote file copy, port forwarding, and the list could go on.

SSH Tunnels

SSH includes the ability to connect a port on your local system to a port on a remote system, so that to applications on your local system the local port looks like a normal local port, but when accessed the service running on the remote machine responds. All traffic is really sent over ssh.

I set up an SSH tunnel for my local system to the outgoing mail server on my server. I tell my mail client to send mail to localhost server (without mail server authentication!), and it magically goes to my personal mail relay encrypted over ssh. The applications of this are nearly endless.

Keep Alive Packets

The problem: unless you’re doing something with SSH it doesn’t send any packets, and as a result the connections can be pretty resilient to network disturbances. That’s not a problem, but it does mean that unless you’re actively using an SSH session, it can go silent causing your local area network’s NAT to eat a connection that it thinks has died, but hasn’t. The solution is to set the “ServerAliveInterval [seconds]” configuration in the SSH configuration so that your ssh client sends a “dummy packet” on a regular interval so that the router thinks that the connection is active even if it’s particularly quiet. It’s good stuff.

/dev/null .known_hosts

A lot of what I do in my day job involves deploying new systems, testing something out and then destroying that installation and starting over in the same virtual machine. So my “test rigs” have a few IP addresses, I can’t readily deploy keys on these hosts, and every time I redeploy SSH’s host-key checking tells me that a different system is responding for the host, which in most cases is the symptom of some sort of security error, and in most cases knowing this is a good thing, but in some cases it can be very annoying.

These configuration values tell your SSH session to save keys to `/dev/null (i.e. drop them on the floor) and to not ask you to verify an unknown host:

UserKnownHostsFile /dev/null
StrictHostKeyChecking no

This probably saves me a little annoyance and minute or two every day or more, but it’s totally worth it. Don’t set these values for hosts that you actually care about.

I’m sure there are other awesome things you can do with ssh, and I’d live to hear more. Onward and Upward!

Rather than authenticate to a SMTP server to send email, which is fraught with potential security issues and hassles, I use a SSH tunnel to the machine running my mail server. This is automatic, easy to configure both for the mail server and mail client, and incredibly secure. It’s good stuff.

The downside, if there is one, is that the tunnel has to be active to be able to send email messages, and SSH tunnels sometimes disconnect a bit too silently particularly on unstable (wireless) connections. I (and others, I suspect) have had some success with integrating the tunnel connection with pre- and post- connection hooks, so that the network manager automatically creates a tunnel after connecting to the network. but this is a flawed solution that produces uneven results.

Recently I’ve discovered this program called “AutoSSH,” which creates an SSH tunnel and tests it regularly to ensure that the tunnel is functional. If it isn’t, AutoSSH recreates the tunnel. Great!

First start off by getting a copy of the program. It’s not part of the OpenSSh package, so you’ll need to download it separately. It’s in every pacakge management repository that I’ve tried to get it from. So installation, will probably involve one of the following commands at your system’s command line:

apt-get install autossh
pacman -S autossh
yum install autossh
port install autossh

When that’s done, you’ll issue a command that resembles the following

autossh -M 25 -f tychoish@foucault.cyborginstitute.net -L 25:127.0.0.1:25

Really, the important part here is the “autossh -M 25” part of the command. This tells autossh to watch (“monitor”) port number 25 on the local system for a tunnel. The rest of the command (e.g. “-f -L 127.0.0.1:25:127.0.0.1:25 mailserver@tychoish.com -N”) is just a typical call to the ssh program.

Things to remember:

• If you need to create a tunnel on a local port with numbered lower than 1000, you’ll need to run the autossh command as root.
• SSH port forwarding only forwards traffic from a local port to a remote port, through an SSH connection. All traffic is transmitted over the wire on port 22. Unless you establish multiple tunnels, only traffic sent to the specific local port will be forwarded.
• Perhaps it’s obvious, but there has to be some service listening on the specified remote end of the tunnel, or else the tunnel won’t do anything.
• In a lot of ways, depending on your use case autossh, can obviate the need for much more complex VPN setups for a lot of deployments. Put an autossh command in an @reboot cronjob, with an account that has ssh keys generated, and just forget about it for encrypting things like database traffic and the like.

Onward and Upward!