This is a rough sketch of some things that I’ve learned about the Sphinx documentation generation system. I should probably spend some time to collect what I’ve learned in a more coherent and durable format, but this will have to do for now:

• If you describe a type in parameter documentation it will automatically link to the Python documentation for that type when using the Python Domain and if you have intersphinx connected. That’s pretty cool.

• Sphinx let’s you define a scope for a file in some cases. If you’re documenting command-line options to a program. (i.e. with the “program” with subsidiary “option” directives,) or if you’re documenting Python objects and callables within the context of a module, the module and program directives have a scoping effect.

  Cool but it breaks the reStructuredText idiom, which only allows you to decorate and provide semantic context for specific nested blocks within the file. As in Python code, there’s no way to end a block except via white-space,¹ which produces some very confusing markup effects.

  The “default-domain” directive is similarly… odd.

• Sphinx cannot take advantage of multiple cores to render a single project, except when building multiple outputs (i.e. PDF/LaTeX, HTML with and/or directories.) if with a weird caveat that only one builder can touch the doctree directory at once. (So you either need to put each builder on its own doctree directory, or let one build complete and then build the reset in parallel.)

  For small documentation sets, under a few dozen pages/kb, this isn’t a huge problem, for larger documentation sets this can be quite frustrating.²

  This limitation means that while it’s possible to write extensions to Sphinx to do additional processing of the build, in most cases, it makes more sense to build custom content and extensions that modify or generate reStructuredText or that munge the output in some way. The include directive in reStructuredText and milking the hell out of make are good places to started.

• Be careful when instantiating objects in Sphinx’s conf.py file: since Sphinx stores the pickle (serialization) of conf.py and compares that stored pickle with the current file to ensure that configuration hasn’t changed (changed configuration files necessitate a full rebuild of the project.) Creating objects in this file will trigger full (and often unneeded) rebuilds.

• Delightfully, Sphinx produces .po file that you can use to power translated output, using the gettext sphinx builder. Specify a different doctree directory for this output to prevent some issues with overlapping builds. This is really cool.

Sphinx is great. Even though I’m always looking at different documentation systems and practices I can’t find anything that’s better. My hope is that the more I/we talk about these issues and the closer I/we’ll get to solutions, and the better the solutions will be.

Onward and Upward!

Like technical debt, information debt is a huge problem for all kinds of organizations, and one that all technical writers need to be aware and able to combat directly. Let’s backup a little…

Information debt is what happens when there aren’t proper systems, tools, and processes in place to maintain and create high quality information resources. A number of unfortunate and expensive things result:

• People spend time recreating documents, pages, and research that already exists. This is incredibly innefficent and leads to:
• Inaccurate information propagates throughout the organization and to the public.
• Information and style “drifts,” when information and facts exist in many places.
• Organizations spend more money on infrastructure and tools as a band-aid when data is poorly organized.
• People lose confidence in information resources and stop relying on them, preferring to ask other people for information. This increases the communication overhead, noise level, and takes longer for everyone, than using a good resource.

To help resolve information debt:

• Dedicate resources to paying back information debts. It takes time to build really good resources, to collect and consolidate information, and to keep them up to date. But given the costs of the debt, it’s often worth it.
• Documents must be “living,” usefully versioned, and there must be a process for updating documents. Furthermore, while it doesn’t make sense to actually limit editing privileges, it’s important that responsibility for editing and maintaining documents isn’t diffused and thus neglected.
• Information resources, must have an “owner” within an organization or group who is responsible for keeping it up to date, and making sure that people know it exists. You can have the best repository for facts, if no one uses it and the documents are not up to date, it’s worthless.
• Minimize the number of information resources. While it doesn’t always make sense to keep all information in the same resource or system, the more “silos” where a piece of information or document might live the less likely a reader/user will find it.

… and more.

I’m working on adding a lot of writing on information debt in the technical writing section of the wiki. I’ll blog more about this, while I continue to work through some of these ideas, but I’m quite interested in hearing your thoughts on this post and on the information-debt pages as well.

Onward and Upward!

I tell people that I’m a professional writer, but this is a bit misleading, because what I really do is figure out how to organize information so that it’s useful and usable. Anyone, with sufficient training and practice, can figure out how to convey simple facts in plain language, but figuring out how to organize simple facts in plain language into a coherent text is the more important part of my job and work.

This post and the “Information Debt” wiki page, begin to address some of these the problem of information resource maintenance, organization, and institutional practices with regards to information and knowledge resources.

Organization is hard. Really hard. One of the challenges for digital resources is that they lack all of the conventions of /technical-writing/books, which would seem to be freeing: you get more space and you get the opportunity to do really flexible categorization and organization things.

Great right?

Right.

Really flexible and powerful taxonomic systems, like tagging systems have a number of problems when applied to large information resources:

• the relationship between the “scope” of the tag, and the specificity of the tag matters a lot. Too much. Problems arise when:
• tags are really specific, pages include a number of pieces of information, and tags can only map to pages.
• tags are general and the resources all address similar or related topics.
• the size of the tag “buckets” matters as well. If there are too many items with a tag, users will find not the tag for answering their questions.
• if your users or applications have added additional functionality using tags, tags begin to break as a useful taxonomic system. For example, if your system attaches actions to specific tags (i.e. send email alerts when content with a specific tag,) or if you use a regular notation to simulate a hierarchy, then editors begin adding content to tags, not for taxonomic reasons, but for workflow reasons or to trigger the system.

The resulting organization isn’t useful from a textual perspective.

• If you have to have multiple tagging systems or namespaces.

  Using namespaces is powerful, and helps prevent collisions. At the same Sat Aug 16 10:50:00 2014, if your taxonomic system has collisions, this points to a larger problem.

• If the taxonomy ever has more than one term for a conceptual facet, then the tagging system is broken.

These problems tend to exacerbate as:

• the resource ages.
• the number of contributors grow.

There’s this core paradox in tagging systems: To tag content effectively, you need a fix list of potential tags before you begin tagging content, and you need to be very familiar with the corpus of tagged contentbefore* beginning to tag content.*

And there’s not much you can do to avoid it. To further complicate the problem, it’s essentially impossible to “redo” a taxonomic system for sufficiently large resources given the time requirements for reclassification and the fact that classification systems and processes are difficult to automate.

The prevalence of tagging systems and the promises of easy, quick taxonomic organization are hard to avoid and counteract. As part of the fight against information debt it’s important to draw attention to the failure of broken taxonomy systems. We need, as technical writers, information custodians, and “knowledge workers,” to develop approaches to organization that are easy to implement and less likely to lead to huge amounts of information debt.

There are two parts to this post: first, the relationship or non-relationship between the ability to write code and technical literacy; and second, the pedagogical methods for teaching people how to program/code.

In some ways, I’ve been writing about this and related topics for quite a while: see /posts/objective-whatsis for an earlier iteration in this train of thought.

Programming and Technical Literacy

Programmers and other technical folks talk a lot about teaching young people to code as the central part of any young technical person’s education and basic computer literacy. Often this grows out of nostalgia for their own experience learning to program, but there are other factors at play.¹

In some cases, they even start or point to projects like Codecademy. Which are, in truth, really cool ideas, but I think that effectively equating the ability to write code with technical literacy is fraught:

• There are many different kinds of technical literacy and writing code is really such a small part. Sure code gives us a reasonable way to talk about things like design and architecture, but actually writing code is such a small part of developing technology.

• Writing code isn’t that important, really. In a lot of ways, code is just an implementation detail. Important as a way of describing some concepts pretty quickly, important because it’s impossible to iterate on ideas without something concrete to point to, but the implementation isn’t nearly as important as the behavior or the interface.

• For the last ~40 years, code has been the way that people design behavior and specify interfaces for software. While there are a lot of reasons why this predominantly takes the form of code, there’s not particular reason that we can’t express logic and describe interfaces using other modalities.

  There are many people who are very technically literate and productive who don’t write code, and I think that defining literacy as being able to write code, is somewhat short sighted. Also, there is another group of people who are actually programmers who don’t think of the things they do as “programming,” like people who do crazy things with spreadsheets, most librarians, among others. These non-coding programmers may shy away from programming or are mostly interested in the output of the program they write and less interested in the programming itself.

This is a huge problem. I hope that this /posts/computer-literacy-project that I’ve been planning will start to address some of these issues, but there’s even more work to do.

How to Teach People to Code

(This section of the post derives from and summaries the “How to Teach People to Program” wiki page.)

Most of the way that programming books and courses teach programming are frustrating and somewhat dire, for a few reasons:

• Most examples in programming books are dumb.
• Basic computer science/engineering knowledge is fundamental to the way that accomplished programmers think about programming but aren’t always required to teach people how to program.
• Syntax isn’t that important, but you can’t ignore it either.
• Slow reveals are really frustrating.
• The kinds of code that you write when learning to programming bear little resemblance to the actual work that programmers do.

The solutions to these problems are complex and there are many possible solutions. As a starting point:

• Separate the way you present core concepts (i.e. data structures, typing, functions, classes, etc.) from actual code examples and from actual explanations of the syntax.

  Interlink/cross reference everything, but if you give people the tools to answer their own questions they’ll learn what they actually need to know, and you can then do a better job of explaining the syntax, basic concepts, and practical examples.

• Provide longer examples that aren’t contrived.

  Examples don’t need to start from first principals, and don’t need to be entirely self contained. Programming work rarely starts from first principals (relative,) and is rarely actually self contained. It’s foolish, then to use these sorts of pedagogical tools.

Thoughts?

Other than shortening sentences, inserting lists, and using document structure, there are a couple of “easy edits” that I make to most documents that other send to me for review:

1. Remove all first person, both singular and plural.

2. Remove all passive sentences, typically by making the sentences more imperative.

In practice these changes are often related.

Expunge the First Person

Removing the first person is important less because it’s “more formal” to avoid the first person and more because it’s always unclear in documentation: Who are “we,” and who is “I”? Should I read “I” as “me” or as the author of the documentation? What if my experiences and environment isn’t like “ours?” While we can resolve these confusion points pretty quickly it gives users another set of information that they must track. And given that technical subjects can be difficult without confusing language, there’s no reason to make things more confusing.

People tend to think that this makes their documentation “friendlier,” “personable,” or “intimate.” People used to interacting directly with users (i.e. people doing user support) are particularly susceptible to first person problems. In support cases, that little bit of personal touch is incredibly valuable and goes a long way toward making people feel comfortable.

Those people are wrong. Don’t do it. Speak simply. Write about the product and the processes you’re documenting, not yourself. Convey facts, not opinions. Provide context, not perspective. If you’re writing the official documentation for a product, your perspective is obvious to readers; if you’re not writing the official documentation, that’s also apparent and probably not your job to disclaim.

Use Good Verbs

Passive sentences and weak verbs are a huge problem. Huge. People with science and engineering back rounds seem to prefer passive sentences because they think that passive sentences convey objectivity, and that this objectivity is desirable.

Passive sentences do convey a sense of objectivity, and there are some cases where there’s no way to avoid describing a property of a thing except passively. That doesn’t make the passive voice generally acceptable. Related to the reason above, passive voice tends to provide a level of “syntatic indirection,” and means that complicated sentences become unnecessarily difficult to comprehend.

In documentation, unlike some other forms, it’s possible (and desirable!) to use imperative verbs, which provides some relief. One of the main projects of documentation is to inculcate “best practices” (i.e. values and conventions,) in users. Imperative verbs are great for this purpose.

In short: Do it!

.. that developers tell.

1. All the documentation you’d need is in the test cases.
2. My comments are really clear and detailed.

3. I’m really interested and committed to having really good documentation.

4. This code is easy to read because its so procedural.
5. This doesn’t really need documentation.

6. I’ve developed a really powerful way to extract documentation from this code.

7. The documentation is up to date.
8. We’ve tested this and nothing’s changed.
9. This behavior hasn’t changed, and wouldn’t affect users anyway.
10. The error message is clear.

11. This entire document needs to be rewritten to account for this change.

12. You can document this structure with a pretty clear table.

Often this is true, more often these kinds of comments assume that it’s possible to convey 3-5 dimension matrixes clearly on paper/computer screens.

13. I can do that.
14. I will do that.
15. No one should need to understand.

I’m not sure I’d fully realized it before, but the key problems in systems administration--at least the kind that I interact with the most--are really manifestations of a tension between complexity and reliability.

Complex systems are often more capable flexible, so goes the theory. At the same time, complexity often leads to operational failure, as a larger number of moving parts leads to more potential points of failure. I think it’s an age old engineering problem and I doubt that there are good practical answers.

I’ve been working on this writing project where I’ve been exploring a number of fundamental systems administration problem domains, so this kind of thing is on my mind. It seems, that the way to address the hard questions often come back to “what are the actual requirements, and are you willing to pay the premiums to make the complex systems reliable?”

Trade-offs around complexity also happen in software development proper: I’ve heard more than a few developers talk in the last few months weigh the complexity of using dynamic languages like Python for very large scale projects. While the quests and implications manifest differently for code, it seems like this is part of the same problem.

Rather than prattle on about various approaches, I’m just going to close out this post with a few open questions/thoughts:

• What’s the process for determining requirements that accounts for actual required complexity?

• How do things that had previously been complex, become less complex?

  Perhaps someone just has write the code in C or C++ and let it mature for a few years before administrators accept it as stable?

• Is there an corresponding level of complexity threshold in software development and within software itself? (Likely yes,) and is it related to something intrinsic to particular design patterns, or to tooling (i.e. programming language implementations, compilers, and so forth.)

Might better developer tooling allow us to programs of larger scope in dynamic languages (perhaps?)

Reader submitted questions:

• Your questions here.

Answers, or attempts thereat in comments.

A friend asked me a question (several weeks ago by publication) on a technical topic and I spent most of the next few days writing a missive on database administration strategy. That seemed like a normal response. I was delighted to find that: I liked the voice, I enjoyed writing the longer document, and there are a dozen or so other related topics that I wanted to explore. So, apparently, I’m writing a book. This is exactly what I need: more projects. Not.

But it’s a good thing: I find the writing inspiring and invigorating. I have a perspective and collection of knowledge that hasn’t been collected and presented in a single place. I like long form writing. The larger piece might also be a good contribution to my portfolio (such as it is.)

I think this kind of writing suits my attention span.

This has left me without a lot of spare time for blogging, and (as I’m prone to do every so often,) rethinking the future of my efforts on tychoish.com and as a blogger. This is boring for all of you, but I’ll give some higher level stuff here and we can follow up in comments:

• Blogging is fun, and even though I’ve not been posting regularly, I’m always writing blog posts. Sometimes I find myself writing posts in emails to friends, but I’m never really going to stop writing blog posts.

• The general explosion of blog publishing that we saw a few years ago has declined. Audience fragmentation happened, readership got entrenched. I feel like I weathered the storm pretty well and I’m really happy with the site and readers I have, but I’m also pretty confident that blogging isn’t going to be the means by which I “level up.”¹

• eBooks have finally happened. For the last decade most people have been saying that ebooks are great for reference material (given search-ability,) and for providing an introduction to a text that people will eventually buy in a paper edition. That may be true, but I think it’s changing rapidly, and with kindles and tablets and smart-phones, I think eBooks have effectively won, such as it is.

  In another ten years, perhaps, we’ll just call them books.

• I’m pretty clear that keeping a blog, and perhaps most of the writing I do in my spare time is for my own enjoyment and betterment and helps to develop a personal portfolio and account of my work. I have no (real) interest in using my writing on tychoish.com or any other side that I maintain, as a way of supporting myself to any greater or lesser extent.

I want to be in the business of writing things and working with technology and ideas and people, not the business of publishing. While the line is not always clear between “writing projects that you publish yourself online,” and “new media publisher,” I want to stay away from the later as much as possible.

So I think this means that most of my “tychoish,” writing time will go to writing this book project, and to fiction, and once my blog post backlog is fully depleted (heh,) most of my postings will either be announcements/progress-reports or a bunch of shorter more off-the-cuff notes.

Here’s hoping at least.

On Outer Alliance Podcast #8, David Levine talked about having worked as a technical writer for some 15 years and then said something to the effect of “It’s a point of great personal pride that I’ve never put a bulleted list in a piece of fiction.”

I laughed out loud. Perhaps frightening a woman walking her dog nearby.

In most ways, the kind of writing that I do for work, API references, tutorials, administration overviews, best-practice descriptions, is very different from the kinds of things I write away from work, or at least I like to think so.

The truth is that I’ve learned a bunch about writing and about communicating in general from writing documentation. While my “professional background,” doesn’t include formal technological training, I definitely “broke in” because I was familiar with technology and could write, rather than being a particularly skilled or trained writer. Any more (just 2.5 years on,) I think the inverse is more true, but that’s conjecture.

Technical writing has definitely shaped the evolution of my taste: a couple years ago, I found myself most drawn to complex tightly constructed prose in fiction. These days I mostly go for sparse clear concise prose that isn’t particularly ornamented. Perhaps it’s only really possible to tune the internal editor for one kind of style at a time.

Having said that, I will confess to feeling--and resisting--the urge to put a bulleted list or some other structured convention of software manuals in fiction.

It’s the little things, really.

I’ve written before about different types of documentation, and the different purposes and goals that each type services. Rather than rehash what documentation is, I’m interested in using this post to think about ways of managing and organizing the documentation process to produce better documentation more easily, with the end goal of being able to increase both maintainability and usability of documentation resources.

Different groups of users--say: administrators, end-users, and developers--interact with technology in overlapping but distinct ways. For some technologies, the differences between the classes of users is not significant and one set of documentation is probably good do every one, plus or minus a very small set. In most other cases, multiple resources are required to be able to address the unique needs of different user groups. Figuring out effective ways to address the different kinds of questions that various groups of users ask, but in a way that makes sense to those users is often the primary challenge in writing documentation.

Having said that, writing different sets of documentation for different users is a lot of work, but given time its not insurmountable. The problem is after six months or more (say,) or a couple of new releases when its time to update the documentation, there are three manuals to update instead of one. This is pretty much horrible. Not only is it more work, but the chances for errors skyrockets, and it’s just a mess.

The solution, to my mind, is to figure out ways to only ever have to write one set of documentation. While it might make theoretical sense to split the material into multiple groups, do everything you can to avoid splitting the documentation. Typically, a well indexed text can be used by multiple audiences if its easy enough for users to skip to read only the material they need.

The second class of solutions revolves around taking a more atomic approach to writing documentation. In my own work this manifests in two ways:

• Setting yourself up for success: understanding how software is going to be updated, or how use is likely to change over time allows you to construct documents that are organized in a way that makes them easy to update. For example: Separate processes from reference material, and split up long processes into logical chunks that you can interlink to remove redundancies.

  Unfortunately, in many cases, it’s necessary to learn enough about a project and the different use patterns before you have the background needed to predict what the best structure of the documentation ought to be.

• Separate structure from content: This is a publishing system requirement, at the core, but using this kind of functionality must be part of the writer’s approach. Writers need to build documentation so that the organization (order, hierarchy, etc.) is not implicit in the text, but can be rearranged and reformed as needed. This means writing documentation “atoms” in a structurally generic way. Typically this also leads to better content. As a matter of implementation, documentation resource would require a layer of “meta files” that would provide organization that would be added at build time.

In effect this approach follows what most systems are doing anyway, but in practice we need another processing layer. Sphinx is pretty close in many ways but most document formats and build systems don’t really have support for this kind of project organization (or they require enough XML tinkering to render them unfeasible.) Once everything’s in place and once all of the atoms exist, producing documents for a different audience is just a matter of compiling a new organization layer and defining an additional output target.

This also produces problems for organization. If content is segregated into hundreds of files for a standard-book-length manual (rather than dozens, say) keeping the files organized is a challenge. If nothing else, build tools will need to do a lot more error checking and hopefully documentation writers will develop standard file organizations and practices that will keep things under control.

Thoughts? Onward and Upward!