For a while, I’ve been envious of some of the project and file navigation features in emacs for browsing bigger projects/programs, things like imenu and tags have always seems awesome but given that I spend most of time editing restructured text and markdown files (I’m a technical writer), these tools have been distant and not a part of my day to day work.

It’s not that it would be impossible to write interfaces for imenu or etags, for the formats I use regularly, but more that I’ve never gotten around to it until now.

We’re still a ways away on the question of etags, but it turns out that when I wasn’t looking rst mode got imenu support, and with the following little bit of elisp you can get imenu for markdown.

(setq markdown-imenu-generic-expression
   '(("title"  "^\\(.*\\)[\n]=+$" 1)
     ("h2-"    "^\\(.*\\)[\n]-+$" 1)
     ("h1"   "^# \\(.*\\)$" 1)
     ("h2"   "^## \\(.*\\)$" 1)
     ("h3"   "^### \\(.*\\)$" 1)
     ("h4"   "^#### \\(.*\\)$" 1)
     ("h5"   "^##### \\(.*\\)$" 1)
     ("h6"   "^###### \\(.*\\)$" 1)
     ("fn"   "^\\[\\^\\(.*\\)\\]" 1)
))

(add-hook 'markdown-mode-hook
   (lambda ()
      (setq imenu-generic-expression markdown-imenu-generic-expression)))

Pretty awesome! I hope it helps you make awesome things.

I’ve recently gone from being someone who uses git entirely “on my own,” to being someone who uses git with a lot of other people at once. I’ve also had to introduce git to the uninitiated a few times. These have both been notable challenges.

Git is hard, particularly in these contexts: not only are there many concepts to learn, but there’s no single proscribed workflow and a multitude of workflow possibilities. Which is great from a philosophy perspective, and arguably good from a “useful/robust tool perspective,” but horrible from a “best practices” perspective. Hell, it’s horrible for a “consistent” and “sane” practices perspective.

There are a lot of solutions to the “finding a sane practice,” when working with git and large teams. Patching or reformulating the interface is a common strategy. Legit is a great example of this, but I don’t think it’s enough, because the problem is really one of bad and unclear defaults. For instance:

• The “master” branch, in larger distributed systems is highly confusing. If you have multiple remotes (which is common), every remote has its own master branch, which all (necessarily) refer to different possible points in a repository’s history.
• The names of a local branch do not necessarily refer to the names of the remote branch in any specific repository. The decoupling of local and remote branches, makes sense from a design perspective, but it’s difficult to retain this mapping in your head, and it’s also difficult to talk about branch configurations because your “view of the universe,” doesn’t often coincide with anyone else’s?

Here are some ideas:

1. Have two modes of operation: a maintainer’s mode that resembles current day git with a few basic tweaks described in later options, and a contributors mode, that is designed with makes the following assumptions:

   • The “mainline” of the project’s development will occur in a branch to which this user only has read-only access.
   • Most of this user’s work will happen in isolated topic branches.

2. Branch naming enforcement:

   • All branches will be uniquely named, relative to a username and/or hostname. This will be transparent (largely) to the history, but will make all conversations about branches less awkward. This is basically how branches work now, with [remote]/[branch], except that all local branches need self/[branch], and the software should make this more transparent.
   • Remote branches will implicitly have local tracking branches with identical names. You could commit to any of the local tracking branches, and pull will have the ability to convert your changes to a self/[branch] if needed.
   • All local branches, if published, will map to a single remote branch. One remote, will be the user’s default “publication target,” for branches.

   This is basically what the origin remote does today, so again, this isn’t a major change.

3. When you run git clone, this remote repository should be the upstream repository, not the origin.

   Use the origin remote, which should be the default “place to publish my work,” and would be configured separately.

4. Minor tweaks.

   • Map local branches directly to remote branches.
   • Be able to specify a remote branch as a “mirror” of another branch.
   • Make cherry-picked commits identifiable by their original commit-id internally. The goal is to push people to cherry-pick commits as much as possible to construct histories without needing to rebase.¹
   • Have sub-module support automatically configured, without fussing.
   • Have better functions to cleaning up branch cruft. Particularly on remotes.
   • Have some sort of configurable “published pointer,” that users can use as a safe block against rebases before a given point.

   The goals here are to:

   • Make working with git all about branches and iteration rather than a sequence of commits.
   • Provide good tools to prevent people from rebasing commits, which is always confusing and rarely actually required.
   • Make branch names as canonical as possible. The fact that there can be many possible names for the same thing is awful.

Who’s with me? We can sort out the details, if you want in comments.

I recently took some of my writing time to create a makefile (Novel Makefile) to manage work I hope to be doing on a new novel project. I’ve started outlining and researching the story in earnest after having spent the past few couple of years talking about it, and I think writing will commence soon. In another post I’d like to write up some thoughts on the tooling and technology of writing non-technical/non-manual long-form.

This post, drawing from the spending some time buried deep in production is about the state of (conceptually) longer form work in digital mediums. Or, at least a brief commentary on same.

The tools that I use to write technical materials do all sorts of cool things, like:

• provide instant cross referencing,
• generate great indexes, and
• automatically generate and link glossaries.

This is not particularly unusual, and in fact Sphinx is somewhat under-featured relative to other documentation generation systems like DocBook.¹

And yet people publish ebooks that virtually identical to paper books. Ebooks seem to say “*this electronic book is the best facsimile of a paper book that we can imagine right now,*” while totally ignoring anything more that a *hyper*text rightfully might be.

I enjoy reading ebooks, just as I have enjoyed reading paperbooks, but mostly because ebooks basically are paperbooks. I’ve written posts in the past challenging myself,and fiction writers in general, to actually do hypertext rather than recapitulating prior modalities in digital form.

At various points I’ve thought that wikis might be a good model of how to do hypertext, because the form is structurally novel. Any more, I don’t think that this is the case: wikis are unstructured and chaotic, and I’ve come to believe that the secret to hypertext is structure. There are so many possibilities in hypertext, and I think much experimentation in hypertext has attempted to address the chaos of this experience. This does serve to highlight the extent to which “the future is here,” but it obscures the fact that structure makes narratives understandable. Think about how much great, new, innovative (and successful!) fiction in the past decade (or so) is not structurally experimental or chaotic. (Answer: there’s a lot of it.)

The not-so-secret of hypertext, is (I suspect,) tooling: without really good tools the mechanics of producing a complex, interactive textual experience² is difficult for a single writer, or even a small group of writers. Most tools that manage the publication and writing of text are not suited to helping the production of large-multi-page and mutli-component texts. One potential glimmer of hope is that tools for developing programs (IDEs, build systems, compilers, templating systems, introspection tools, pattern matching, etc.) are well developed and could modified for use in text production.

The second non-so-secret of hypertext is probably that hypertext is an evolution of text production and consumption, not a revolution. Which only seems reasonable. We have the technology now to produce really cool text product. While tooling needs to get better, the literature needs to do some catching up.

Lets start making things!

Access control is both immensely useful and incredibly broken.

Access control, or the ability to constrain access to data and programs in a shared system is the only way that we, as users of shared systems, can maintain our identities, personal security, and privacy. Shared systems include: databases, file servers, social networking sites, virtualized computing systems, vendor accounts, control panels, management tools, and so forth all need robust, flexible, granular, and scalable access control tools.

Contemporary access control tools--access control lists (ACL,) and access control groups--indeed the entire conceptual framework for managing access to data and resources, don’t work. From a theoretical practice, ACLs that express a relationship between users or groups of users and data or resources, represent a parsimonious solution to the “access control problem:” if properly deployed only those with access grants will have access to a given resource.

In practice these these kinds of relationships do not work. Typically relationships between data and users is rich and complex and different users need to be able to do different things with different resources. Some users need “read only” access, others need partial read access, some need read and write access but only to a subset of a resource. While ACL systems can impose these kinds of restrictions, the access control abscration doesn’t match the data abstraction or the real-world relationships that it supposedly reflects.

Compounding this problem are two important factors:

1. Access control needs change over time in response to social and cultural shifts among the users and providers of these resources.
2. There are too many pieces of information or resources in any potential shared system to allocate access on a per-object or per-resource basis, and the volume of objects and resources is only increasing.

Often many objects or resources have the same or similar access control patterns, which leads to the “group” abstraction. Groups make it possible to describe a specific access control pattern that apply to a number of objects, and connect this pattern with specific resources.

Conceptual deficiencies:

• There’s a volume problem. Access control data represents a many-to-many-to-many relationship. There are many different users and (nested) groups, many different kinds of access controls that systems can grant, and many different (nested) resources. This would be unmanageably complex without the possibility for nesting, but nesting means that the relationships between resources and between groups and users are also important. With the possibility for nesting access control is impossible.

• ACLs and group-based access control don’t account for the fact that access must be constantly evolving, and current systems don’t contain support for ongoing maintenance. (we need background threads that go through and validate access control consistency.) Also all access control grants must have some capacity for automatic expiration.

• Access control requirements and possibilities shift as data becomes more or less structured, and as data use patterns change. The same conceptual framework that works well for access control in the context of a the data stored in a relational database, doesn’t work so when the data in question is a word processing document, an email folder, or a spread sheet.

  The fewer people that need access to a single piece of data, the easier the access control system can be. While this seems self evident, it also means that access control systems are difficult to test in the really large complex systems in which they’re used.

• Group-based access control systems, in effect, normalize data about access control, in an effort to speed up data access times. While this performance is welcome, in most cases granting access via groups leads to an overly liberal distribution of access control rights. At once, its too difficult to understand “who has access to what” and too easy to add people to groups that give them more access than they need.

So the solution:

1. Denormalize all access control data,
2. don’t grant access to groups, and
3. forbid inheritance.

This is totally counter to the state of the art. In most ways, normalized access control data, with role/group-based access control, and complex inheritance are the gold standard. Why would it work?

• If you have a piece of data, you will always be able to determine who has access to data, without needing to do another look-up.

• If you can deactivate credentials, then a background process can go through and remove access without causing a large security problem. (For partial removes, you would freeze an account, let the background process modify access control and then unfreeze the account.)

  The down side is that, potentially, in a large system, it may take a rather long time for access grants to propagate to users. Locking user accounts makes the system secure/viable, but doesn’t make the process any more quick.

  As an added bonus, these processes could probably be independent and wouldn’t require any sort of shared state or lock, which means many such operation could run in parallel, and they could stop and restart at will.

• The inheritance option should be fuzzy. Some sort of “bucket-based” access control should be possible, if there’s a lot of data with the same access control rules and users.

  Once things get more complex, buckets are the wrong metaphor, you should use granular controls everywhere.

Problems/Conclusion:

• Denormalization might fix the problems with ACLs and permissions systems, but it doesn’t fix the problems with distributed identity management.

  As a counterpoint, this seems like a cryptography management problem.

• Storing access control information with data means that it’s difficult to take a user and return a list of what these credentials have access to.

  In truth, centralized ACL systems are subject to this flaw as well.

• A huge part of the problem with centralized ACL derives from nesting, and the fact that we tend to model/organize data in tree-like structures, that often run counter to the organization of access control rights. As a result access control tools must be arbitrary.

I tell people that I’m a professional writer, but this is a bit misleading, because what I really do is figure out how to organize information so that it’s useful and usable. Anyone, with sufficient training and practice, can figure out how to convey simple facts in plain language, but figuring out how to organize simple facts in plain language into a coherent text is the more important part of my job and work.

This post and the “Information Debt” wiki page, begin to address some of these the problem of information resource maintenance, organization, and institutional practices with regards to information and knowledge resources.

Organization is hard. Really hard. One of the challenges for digital resources is that they lack all of the conventions of /technical-writing/books, which would seem to be freeing: you get more space and you get the opportunity to do really flexible categorization and organization things.

Great right?

Right.

Really flexible and powerful taxonomic systems, like tagging systems have a number of problems when applied to large information resources:

• the relationship between the “scope” of the tag, and the specificity of the tag matters a lot. Too much. Problems arise when:
• tags are really specific, pages include a number of pieces of information, and tags can only map to pages.
• tags are general and the resources all address similar or related topics.
• the size of the tag “buckets” matters as well. If there are too many items with a tag, users will find not the tag for answering their questions.
• if your users or applications have added additional functionality using tags, tags begin to break as a useful taxonomic system. For example, if your system attaches actions to specific tags (i.e. send email alerts when content with a specific tag,) or if you use a regular notation to simulate a hierarchy, then editors begin adding content to tags, not for taxonomic reasons, but for workflow reasons or to trigger the system.

The resulting organization isn’t useful from a textual perspective.

• If you have to have multiple tagging systems or namespaces.

  Using namespaces is powerful, and helps prevent collisions. At the same Sat Aug 16 10:50:00 2014, if your taxonomic system has collisions, this points to a larger problem.

• If the taxonomy ever has more than one term for a conceptual facet, then the tagging system is broken.

These problems tend to exacerbate as:

• the resource ages.
• the number of contributors grow.

There’s this core paradox in tagging systems: To tag content effectively, you need a fix list of potential tags before you begin tagging content, and you need to be very familiar with the corpus of tagged contentbefore* beginning to tag content.*

And there’s not much you can do to avoid it. To further complicate the problem, it’s essentially impossible to “redo” a taxonomic system for sufficiently large resources given the time requirements for reclassification and the fact that classification systems and processes are difficult to automate.

The prevalence of tagging systems and the promises of easy, quick taxonomic organization are hard to avoid and counteract. As part of the fight against information debt it’s important to draw attention to the failure of broken taxonomy systems. We need, as technical writers, information custodians, and “knowledge workers,” to develop approaches to organization that are easy to implement and less likely to lead to huge amounts of information debt.

Joey Hess’ blog of his work on git-annex-assitant has been a great inspiration to me. Basically, Joey had a very successful Kickstarter campaign to fund his work on a very cool tool of his that he wants to expand to be a kind of drop-box replacement. As part of the project, he’s been blogging nearly every day about the work, his progress, and the problems he’s faced.

I really admire this kind of note taking, and think it’s really interesting to see how people progress on cool projects. More than that, I think it’s a cool way to make yourself accountable to the world, and help ensure things get done.

By the same token, when your project work is writing, increasing daily workload by writing notes/posts rather than actually doing work is a problem, on the other hand, given the right kind of templates, and a good end of day habit, it might be easy and a good habit.

Anyone else interested in this? Thoughts on using rhizome versus some other area of tychoish.com? So many choices!

This is really a short rant, and should come as a surprise to no one.

I hate DOC files, and RTF files, to say nothing of ODF, DOCX, and their ilk because they have two necessarily conflicting properties:

1. They’re oriented at producing documents on paper. Which is crazy. Paper is an output, but it’s not the only output in common use, so it’s nuts that generic document representation formats would be so tightly coupled with paper.

2. The rendering of the content is editor specific, particularly with regards to display options. If I compile a document and send it to you, I have no guarantee whatsoever about the presentation or display of the document on your system, particularly if I’m not certain that your system is similarly configured. Particularly with respect to fonts, page breaks, etc.

This is particularly idiotic with respect to 1.

It’s not that PDF is great, or especially usable, but it’s consistent and behaves as expected. Furthermore, it does a good job of appropriately expressing the limitations of paper.

So use PDF and accept no substitutions.

Things you should know about, dear readers:

1. I published a book and released a mess under the banner of the new Cyborg Institute. These projects are:

• Systems Administration for Cyborgs

  This is a high level guide that introduces key systems administration knowledge domains and concepts. Possible subtitles include: “How to think like a systems administrator,” or “Just add man pages.”

  Read it now. Tell your friends. And please, send feedback and comments.

• Taskfile

  Rather than use org-mode, I’ve taken to using a sort of hacked together makefile that aggregates task items from other textfiles. It works surprisingly well.

• stl

  The idea behind this script is that it would provide a good way to report activity in a regular way so that I’d be able to look back over a log file and see how much I’d written (i.e. report word counts,) and also do a bunch of other straightforward reporting and logging.

  I plan to do a revision of this that’s a bit less complicated and a bit more modular, but it’s a nice proof of concept.

• stack

  I’ve been meaning to publish this for a long time, but this project contains configuration files for emacs and StumpWM, with other systems to follow.

2. I’ve made a bunch of wiki pages on Information Debt. Current pages include:

   • Books
   • Taxonomy
   • Wiki
   • Document Management
   • Hierarchy

3. I was planning on speaking at the Open Help Conference, on August 10-15th,

   2012. But then, United Airlines canceled my flight and I spent the day in Terminal C of Newark Airport and ultimately couldn’t make it for the conference. Which is a huge shame. My talk was going to be is about involving developers in documentation processes, about maintaining documentation, and about working with documentation “as code.”

   The site I made for the talk is at: http://tychoish.net/oh-2012

   And the last draft of the slides are at: http://tychoish.net/oh-2012/slides/

(I was speaking on behalf of `10gen <http://10gen.com/>`_, my employer, `MongoDB <http://docs.mongodb.org/>`_ `documentation project <http://docs.mongodb.org/>`_ that I work on by day (and sometimes night.))

There’s a folk song with the line “let the toast go free,” to which an assembled crowd of singers will often stand up and cry “free the toast!” in response.

I’ve been spending the morning going through email, reading feeds, tweaking some of the issues raised in the site improvements page, and trying to rock getting things done. In the mean time I have a few links to share.

• The Everyday Lives Of Video Game Developers
• The Price of Free
• This article, and the one before, are both open on my “things to read list. Imagine that’s what I’m doing this afternoon.
• Branchable - ikiwiki (like the wiki that powers this site) hosting service, based on publicly available tools. I kinda want to set up a branchable instance.
• Pybtex - a python/yaml drop in replacement for BibTeX, which is really cool, and I suspect that this will probably be my next citation management tool. I’ve been looking for one for a long time.
• Kindle Feeder - a tool to crawl your feeds and move them to your kindle. I’m currently sifting through my feeds and seeing what might make sense to read on the kindle. Might as well, and the automatic delivery makes a lot of sense.
• I played with ehsell for the first time today in response to this post, and I have to say that I really like it, and it’s much more intuitive and usable than I thought it’d be. I like the idea of being able to write shell functions in emacs-lisp. As an aside the post is, I think, an example of really great casual technical writing.

There are two parts to this post: first, the relationship or non-relationship between the ability to write code and technical literacy; and second, the pedagogical methods for teaching people how to program/code.

In some ways, I’ve been writing about this and related topics for quite a while: see /posts/objective-whatsis for an earlier iteration in this train of thought.

Programming and Technical Literacy

Programmers and other technical folks talk a lot about teaching young people to code as the central part of any young technical person’s education and basic computer literacy. Often this grows out of nostalgia for their own experience learning to program, but there are other factors at play.¹

In some cases, they even start or point to projects like Codecademy. Which are, in truth, really cool ideas, but I think that effectively equating the ability to write code with technical literacy is fraught:

• There are many different kinds of technical literacy and writing code is really such a small part. Sure code gives us a reasonable way to talk about things like design and architecture, but actually writing code is such a small part of developing technology.

• Writing code isn’t that important, really. In a lot of ways, code is just an implementation detail. Important as a way of describing some concepts pretty quickly, important because it’s impossible to iterate on ideas without something concrete to point to, but the implementation isn’t nearly as important as the behavior or the interface.

• For the last ~40 years, code has been the way that people design behavior and specify interfaces for software. While there are a lot of reasons why this predominantly takes the form of code, there’s not particular reason that we can’t express logic and describe interfaces using other modalities.

  There are many people who are very technically literate and productive who don’t write code, and I think that defining literacy as being able to write code, is somewhat short sighted. Also, there is another group of people who are actually programmers who don’t think of the things they do as “programming,” like people who do crazy things with spreadsheets, most librarians, among others. These non-coding programmers may shy away from programming or are mostly interested in the output of the program they write and less interested in the programming itself.

This is a huge problem. I hope that this /posts/computer-literacy-project that I’ve been planning will start to address some of these issues, but there’s even more work to do.

How to Teach People to Code

(This section of the post derives from and summaries the “How to Teach People to Program” wiki page.)

Most of the way that programming books and courses teach programming are frustrating and somewhat dire, for a few reasons:

• Most examples in programming books are dumb.
• Basic computer science/engineering knowledge is fundamental to the way that accomplished programmers think about programming but aren’t always required to teach people how to program.
• Syntax isn’t that important, but you can’t ignore it either.
• Slow reveals are really frustrating.
• The kinds of code that you write when learning to programming bear little resemblance to the actual work that programmers do.

The solutions to these problems are complex and there are many possible solutions. As a starting point:

• Separate the way you present core concepts (i.e. data structures, typing, functions, classes, etc.) from actual code examples and from actual explanations of the syntax.

  Interlink/cross reference everything, but if you give people the tools to answer their own questions they’ll learn what they actually need to know, and you can then do a better job of explaining the syntax, basic concepts, and practical examples.

• Provide longer examples that aren’t contrived.

  Examples don’t need to start from first principals, and don’t need to be entirely self contained. Programming work rarely starts from first principals (relative,) and is rarely actually self contained. It’s foolish, then to use these sorts of pedagogical tools.

Thoughts?