I've been thinking about what I do as a software engineer for a while, as there seems to be a common thread through the kinds of projects and teams that I'm drawn toward and I wanted to write a few blog posts on this topic to sort of collect my thoughts and see if these ideas resonated with anyone else.

I've never been particularly interested in building new and exciting features. Hackathon's have never held any particular appeal, and the things I really enjoy are working on are on the spectrum of "stabilize this piece of software," or "make this service easy to operate" or "refactor this code to make support future development" and less "design and build some new feature." Which isn't to say that I don't like building new features or writing code, but that I'm more driven by the code and supporting my teammates than I am by the feature.

I think it's great that I'm different from software engineers who are really focused on the features, both because I think the tension between our interests pushes both classes of software engineer to do great things. Feature development keeps software and products relevant and addresses users' needs. Stabilization work makes projects last and reduces the incidence of failures that distract from feature work, and when there's consistent attention paid to aligning infrastructure [1] work with feature development of the long term, infrastructure engineers can significantly lower the cost of implementing a feature.

The kinds of projects that fall into these categories inculde the following areas:

• managing application state and workload in larger distributed contexts. This has involved designing and implementing things like configuration management, deployment processes, queuing systems, and persistence layers.
• concurrency control patterns and process lifecycle. In programming environments where threads are available, finding ways to ensure that processes can safely shut down, and errors can be communicated between threads and processes takes some work and providing mechanisms to shutdown cleanly, communicate abort signals to worker threads, and handle communication patterns between threads in a regular and expected way, is really important. Concurrency is a great tool, but being able to manage concurrency safely and predictably and in descret parts of the code are useful.
• programming model and ergonomic APIs and services. No developers produces a really compelling set of abstractions on the first draft, particularly when they're focused on delivering different kinds of functionality. The revision and iteration process helps everyone build better software.
• test infrastructure and improvements. No one thinks tests should take a long time or report results non-deterministically, and yet so many test are. The challenge is that tests often look good or seem reasonable or are stable when you write them, and their slow runtimes compound overtime, or orthogonal changes make them slower. Sometimes adding an extra check in some pre-flight test-infrastructure code ends ups causing tests that had been just fine, thank you to become problems. Maintaining and structure test infrastructure has been a big part of what I've ended up doing. Often, however, working back from the tests, it's possible to see how a changed interface or an alternate factoring of code would make core components easier to test, and doing a cleanup pass of tests on some regular cadence to improve things. Faster more reliable tests, make it possible to develop with greater confidence.

In practice this has included:

• changing the build system for a project to produce consistent artifacts, and regularizing the deployment process to avoid problems during deploy.
• writing a queuing system without any extra service level dependencies (e.g. in the project's existing database infrastructure) and then refactoring (almost) all arbitrary parallel workloads to use the new queuing system.
• designing and implementing runtime feature flagging systems so operators could toggle features or components on-and-off via configuration options rather than expensive deploys.
• replacing bespoke implementations with components provided by libraries or improving implementation quality by replacing components in-place, with the goal of making new implementations more testable or performant (or both!)
• plumbing contexts (e.g. Golang's service contexts) through codebases to be able to control the lifecycle of concurrent processes.
• implementing and migrating structured logging systems and building observability systems based on these tools to monitor fleets of application services.
• Refactoring tests to reuse expensive test infrastructure, or using table-driven tests to reduce test duplication.
• managing processes' startup and shutdown code to avoid corrupted states and efficiently terminate and resume in-progress work.

When done well (or just done at all), this kind of work has always paid clear dividends for teams, even when under pressure to produce new features, because the work on the underlying platform reduces the friction for everyone doing work on the codebase.

[1]

It's something of an annoyance that the word "infrastructure" is overloaded, and often refers to the discipline of running software rather than the parts of a piece of software that supports the execution and implementation of the business logic of user-facing features. Code has and needs infrastructure too, and a lot of the work of providing that infrastructure is also software development, and not operational work, though clearly all of these boundaries are somewhat porous.

My friend Jo wrote a post a while ago that addressed the subject of building an audience for your scholarly work. You can read the post on her blog, here.

One of the things that I think Jo is really great at is thinking practically about academic careers and trajectories in light of the current academic job market. While people working in traditional academic spaces and on a traditional academic course have a different set of challenges than folks like me, her points still resonate.

How do you build networks and audiences? Two things:

1. You talk to people.

Audiences are built on relationships. While we might like to think that writers and scholars are able to attract audiences purely on the basis of their work, in practice additional work is required.

2. You make sure you have something to show for yourself.

Everyone's got ideas, and projects that they'd like to work on. People love to talk about their ideas. Success, I think, comes when you have something to show for yourself and your projects, and give people some level of confidence that your can make good on your ideas.

In sort, write more, publish more. While quality matters some, being more than someone to talks well at parties is really important.

I think this approach is useful for people doing any kind of creative or intellectual work that engages an audience, but I'm interested in your thoughts.

All of my friends who have taught composition are appalled when they hear me say that I want to teach writing. But it's true: I would be interested in having the opportunity to give people the kind of writing education that I never got to have. I've even collected a few of these ideas on a very rough "pedagogy" page. This post, by contrast, will be a list of "things I wish I could have learned before I got a job writing."

• How to write in long form. The skils and process for writing something that's a hundred pages is fundamentally different from the process for writing pieces that are a few hundred words or a few pages. Project management, planning, and organization are totally different skills.
• Working with editors. In school, the editing process is very conversational. Editors, comment and ask you to make changes if agree with their judgment. Writers need to learn how to gather requirements, write the best possible content, and then hand it over to an editor who will modify the text without comment. Not only is it important to learn how to "get over this," but also in how to learn from this kind of editing
• How to revise work. While I've learned to avoid making a number of mistakes to which I'm particularly prone, and spot those errors when the slip through, it's really the process of applying for jobs that has taught me how to revise my own writing. Revision is probably the hardest writing skill, and I think there are probably better ways to teach revision than some sort of idealized "drafting process."
• How to write at volume, even when you're not feeling inspired. We're pretty good at teaching people to write when they're inspired or have done a lot of research. But writing Writing needs to be as instinctive as speech and the kind of thing that you don't need to be inspired to be able to do. Not because anyone writes that much, but it's a comfort thing.
• How to document things. Which is to say, how to record a practice, wprocedure, or interface, to tell people (and your future self,) how to do something. I had to figure this out on my own, and I think people would be much better writers for being 10% worse at writing essays and 10% better at writing processes.

That would do it! I've included some work in this direction in the pedagogy page, but comments, are always valuable.

I read this post by one of the partners in one of the coolest web services around, you should open that in a new window and then come back.

Back? Great!

Lars, proposes crowd-funding as a way to support free software development. Basically, run a "sponsor me to develop stuff" program, but rather than fund free software as a start-up around a single project or work for a big vendor.

It's a nifty idea, and it's got me thinking about micro-entrepreneurship. This would be where you make or do things, but not on a big scale. The businesses you create are small, and probably aren't completely full-time equivalent, but in aggregate it's good enough. While this is not the most prominent form of entrepreneurship on the internet, my sense is that it's way bigger than most people think.

We're too used to seeing multi-million dollar venture capital fund raising, IPOs, big acquisition deals, to realize the multitude of people who are making a few to several tens of thousands of dollars doing much smaller amounts of work.

I suppose I could write a whole post on good enough economics in the vein of this post on patronage from JamesGovernor but I'll just leave a place holder link to a wiki page, in case someone else wants to fill things in.

Thoughts:

• Service-businesses don't scale particularly well, any individual work can only produce so much work, and it's hard to make individuals any more productive. In light of that, large service-based firms are unlikely to form.
• Most people have pretty specialized skills and abilities. Self-employment, particularly full time employment makes it difficult for people to spend most of their time doing what their best at. Specialization and differing skills is also what creates a market for service-based endeavors.
• Lacking health care and other benefits of traditional employment, it's hard for people to be more self-employed and less conventionally-employed. Given this, doing entrepreneurial projects on a smaller scale makes more sense.
• Some kinds of entrepreneurial activities are attractive because, while they may not produce the same level of income as a salaried position, they allow more freedom and flexibility. This is the conventional justification for self-employment, and also the reason that most aligns with the "good enough" policy.

The problem with these kinds of "little businesses," is that it's too easy to focus on income earning work (e.g. freelance, and client work,) at the expense of doing basic work (e.g. developing core free software, doing basic research, writing fiction.) While the crowd sourcing notion makes a lot of sense, it requires a lot of faith in the crowd. I'm also unsure of how sustainable it is: while individuals can justify small amounts of money for such purposes, organizations cannot. Without organizational support, revenue is much lower, and it probably puts the larger financial burden on the smaller users, relatively speaking.

Continuing from the discussion regarding intellectual practice, I've been talking with a number of people (my father in particular) about graduate school and the prospect of "bootstrapping" a scholarly practice using "new media," like blogging, and wiki making. I want to explore both my thoughts graduate school and bootstrapping with new media, and as you'd expect both of these ideas are rather intertwined. My initial gloss follows:

What I want, it seems to me, isn't a career--I have one of those--but to sustain intellectual life and practice. I would like to be able to ask questions, read seriously, participate in important conversations, and to write about this work and practice effectively for an audience that is invested in these discussions. This post is a follow up to my "career pathways" post.

I have a blog and wiki, I can read, and my writing continues to improve. How hard can it be to achieve these goals and establish this practice on my own? Famous last words.

The thing is, I hate auto-didacticism as an approach to knowledge production and learning. Sure it works, sometimes, and professionally I think I've been able to succeed on the basis of being able to learn things on my own. At the same time, self teaching at more advanced levels, and avoiding formal study feels like a mechanism for people to use to avoid challenging themselves or their assumptions about the world. The challenge here, in addition to discipline (in a number of senses of the word,) is to avoid scholarly isolationism.

Conversely, it might be true that sufficiently advanced study is always already self-lead and self-taught anyway. That's not a conjecture I have the experience or specialty to comment upon, but it's a possibility.

In any case, my success at being able to do meaningful and fulfilling work, hinges upon:

• being able to write and interact effectively for your communications medium. In my case this means, use blogging and wikis well.
• being able to maintain an active presence and participation in the discussions and work you want to do. This means posting regularly, in addition to writing, reading, and thinking about various projects. Work needs to be sustained and ongoing.
• being able to make leisure time sacrifices to support the work. There's only so much time in the day, and I think it's also important to manage expectations somewhat in recognition of this fact.
• being able to find or establish and interact with a community of peers. Regardless of interest or focus, it's important to find colleagues who do work that is enough like yours to allow them to grasp the intricacies of your work and different enough to infuse the conversation with useful context and ideological breadth.

At least, that's my hope. What am I forgetting?

I always thought that I would go to graduate school sometime in my twenties. I ask questions that are pretty geeky and difficult to answer, I think learning and research are pretty important, and I want to talk to people about ideas, projects, and theories. There are skills I need to be able to address the questions I have and background literates that I wish I were way more familiar with.

It seems like a good fit. Right?

Right. Well, right only if we accept that graduate school is a mechanism for personal betterment. While that has to be part of it, mostly graduate school is a job and the first in a long line of possible jobs. The academic career path has merits and demerits, but it's still work, and I think to ignore this, makes it possible to accept atrocious labor practices in the academic world.

Somehow, without much intentionally on my part, I've found a career that I enjoy. Even more curious is the fact that being a professional writer with some technical background is the kind of thing that enjoys a certain kind of perpetual demand. And better yet, it's impossible to get a degree to support this career: as near as I can tell literature degrees, history degrees, theology degrees, theater degrees, and psychology degrees are all equally relevant and irrelevant.

While I'm not convinced that I never want to teach, if the "getting a job" portion of going to graduate school is somewhat moot, then I'm left with a couple of questions:

• If additional schooling doesn't affect career options and possibilities, then does it make sense to spend significant time in pursuit of an advanced degree?
• How do I develop and maintain an intellectual and scholarly practice without graduate school?
• How do I prevent my career from stagnating and from getting stuck in less than ideal jobs in mid-career and late career stages?
• I work in field where the need for human labor is constantly (and ideally) being automated away. The conventional wisdom is "develop specialties, but don't get too cemented in a particular function so that you have options for after your job gets replaced. Combined with the orthogonal issue that writing and the work of writers is horribly misunderstood by just about everyone, figuring out "career paths is not necessarily easy. How do I deal with this long term concern in a more manageable way while being mindful of the future concerns.

While I sometimes feel like this blog can stray into the "overly meta", I think that prefer intentionally over aimless wandering. Indeed, I think this career issue might have been a great deal easier for me had I figured some of these things out earlier. I know that we don't always find clear and definitive answers to these problems and that solutions come in pieces and very slowly.

This series is about thinking about these issues to increase the possibility of intentionally and to document my process so that people can provide feedback. With luck, this will also help form a model for people who want to think about ways of contributing to scholarly conversations and grow intellectually, but needn't do that in the context of the academic training and labor market.

I look forward to hearing from you and working with you all!

I'm a technical writer. This is a realization that has taken me some time to appreciate and understand fully.

Technical writing is one of those things that creators of technology, a term that I will use liberally, all agree is required, but it's also something that's very difficult to do properly. I think this difficulty springs from the following concerns: What constitutes describing a technology or process in too much detail? Not enough? Are all users of a technology able to make use of the same level of documentation sets? If users are too diverse, what is the best way to make sure that their needs are addressed: do we write parallel documentation for all users, or do we try and equalize less advanced users so that the core documentation is useful to everyone?

The answers to these questions vary of course with the needs of the product being documented and the use cases, but I think resolving these concerns presents a considerable challenge to any kind of technical documentation project, and the way that the documentation developers resolve these issues can have a profound effect not only on the documentation itself but the value and usefulness of the documentation itself. As I've been thinking about the utility and value of technical writing, a professional hazard, I've come up with a brief taxonomy of approaches to technical writing:

• First, there's the document everything approach. Starting with a full list of features (or even an application's source) the goal here is to make sure that there's no corner unturned. We might think of this as the "manual" approach, as the goal is to produce a comprehensive manual. These are great reference materials, particularly when indexed effectively, but the truth is that they're really difficult for users to engage with, even though they may have all the answers to a users questions (e.g. "RTFM.") I suspect that the people who write this kind of documentation either work closely with developers or are themselves developers.
• Second, there's what I think of as the systems or solutions document, which gives up comprehensiveness, and perhaps even isolation to a single tool or application, and documents outcomes and processes. They aren't as detailed, and so might not answer underlying questions, but when completed effectively they provide an ideal entry point into using a new technology. In contrast to the "manual" these documents are either slightly more general interest or like "white papers." This class of documentation, thus, not simply explains how to accomplish specific goals but illuminates technical possibilities and opportunities that may not be clear from a function-based documentation approach. I strongly suspect that the producers of this kind of documentation are very rarely the people who develop the application itself.
• In contrast to the above, I think documentation written for education and training purposes, may appear to be look either a "manual" or a "white paper," but have a fundamentally different organization and set of requirements. Documentation that supports training is often (I suspect) developed in concert with the training program itself, and needs to impart a level of deeper understanding of how a system works (like the content of a manual,) but doesn't need to be comprehensive, and needs mirror the general narrative and goals of the training program.
• Process documentation finally, is most like solution documentation, but rather than capture unrealized technological possibilities or describe potentially hypothetical goals, these kinds of documents capture largely institutional knowledge to more effectively manage succession (both by future iterations of ourselves, and our replacements). These documents have perhaps the most limited audience, but are incredibly valuable both archival (e.g. "How did we used to do $*?") and also for maintaining consistency particularly amongst teams as well as for specific tasks.

I think the fundamental lesson regarding documentation here isn't that every piece of technology needs lots and lots of documentation, but rather that depending on the goals for a particular technology development program or set of tools, different kinds of documentation may be appropriate, and more useful in some situations.

As a secondary conclusion, or direction for more research: I'd be interested in figuring out if there are particular systems that allow technical writers (and development teams) to collect multiple kinds of information and produce multiple documentation for different organizations. Being able to automatically generate different wholes out of documentation "objects" if we may be so bold.

I must look into this. Onward and Upward!