tl;dr> I’ve decided to take a new job as an early engineering manager/tech lead, at a data analytics/database startup. And I hope to begin blogging/journaling here more about software development and the way that software and databases interact with other things that grab my interest.

This post doesn’t really have a narrative. It’s just a collection of thoughts. I’m sort of out of the habit of writing blog posts, and one of the things that’s gotten me stuck on writing in the past year or so is feeling like the narrative of any given blog post is not quite right.¹ But we have to start somewhere, and the story will happen over the next few posts.

In the middle of 2020 I left a job at a company that I’d been at for almost 9 years. It seemed like the world was on fire, what it meant to work at a globally distributed tech company was changing (and still is!), and it felt like a good time to make a change. But to what? I’ve spent the last few years working on different projects and exporing different things: I’ve worked on cool distributed systems problems, I’ve worked on product-focused (backend) engineering, I’ve worked on familiar latency-sensitive compute provisioning and orchestration, and a little bit this and that between.

I’ve identified a few things about jobs and working in this process:

• while remote work is (and has been for a while) definitely a reality of our world,² the one thing you can’t really accommodate for is timezones. A friend, zmagg, claims (and I believe this) that time differences between 9 hours and 16 hours are basically unworkable as there isn’t really enough overlap to really collaborate.
• if the center of gravity in a company is in a place, or in an office, remote teams really have to be intentional about making sure to include people who are outside of that bubble.
• the interesting parts of software engineering is what happens between people building software on a team: programming is a means to an end, and while the tools change every so often, how you build together is really fascinating (and hard)!

My new job is different from any job I’ve ever had: First, I’m going to be working on building and developing teams and helping support the engineers and the engineering teams rather than working directly/full-time on the software. I probably will end up doing some software work too. This isn’t that novel: I’ve definitely done “the management thing” a few times of times, but more of my time will be doing this, and I get to be more intentional about how I approach it. Second, my work (mostly) has been in support of products that other people--largely my coworkers develop, in the sense that the work that I’m drawn to is the internal infrastructure of software. I like infrastructure a lot, and I think I tend to focus on these kinds of problems because I feel like it gives me the biggest opportunity to make an impact. In this sense, in a lot of ways this role is very similar to my previous jobs: building the infrastructure to enable and support the people working around me. Only this time it is (not entirely) software. I’m actually looking forward to mapping concrete business goals to actual research and development problems and projects, helping to provide context and prioritize projects without needing to filter through layers of indirection.

I have been blogging in one form or another (mostly this form) to greater and lesser (mostly lesser) extent for a (really) really long time. The landscape of the way we interact on the internet is changing: twitter is really a shell of its former self, “big” social media networks (facebook, instagram, ticktock, twitter, etc.) have really segmented by generation and region, and little social media sites are still niche.

For my part, I’ve been spending more time in “group chat,” with sort of wacky combinations of random friends. This isn’t new for me, and I take a lot of joy of building out very small communities of folks. Also, as I think about it, the thing I most want to do is have coversations with people. Sometimes that conversation is longer form (like this,) and sometimes, it’s more one-to-many (like this sort of half baked telegram channel idea that’s just sort of a “things I would have said on twitter” but as a chat,) but rarely is it “I would like to become a publisher,” or I want to use blogging as a way to market some other entrepreneurial effort. Not that there are anything wrong with these reasons for blogging.

It’s easy (as a software engineer, though I think this probably holds true for anyone who process for “building” thing requires a lot of just “thinking,” about things,) to do a lot of the thinking part in private and to only show up with the “right” solution (with maybe a few alternatives for rhetorical effect,) or to mostly focus our writing on conclusions reached rather than questions asked.

So perhaps, (and we’ll see if holds,) I can use this space more as a journal and as a collection of questions and (perhaps) failed attempts and less of news feed.

I’ve been thinking about what I do as a software engineer for a while, as there seems to be a common thread through the kinds of projects and teams that I’m drawn toward and I wanted to write a few blog posts on this topic to sort of collect my thoughts and see if these ideas resonated with anyone else.

I’ve never been particularly interested in building new and exciting features. Hackathon’s have never held any particular appeal, and the things I really enjoy are working on are on the spectrum of “stabilize this piece of software,” or “make this service easy to operate” or “refactor this code to make support future development” and less “design and build some new feature.” Which isn’t to say that I don’t like building new features or writing code, but that I’m more driven by the code and supporting my teammates than I am by the feature.

I think it’s great that I’m different from software engineers who are really focused on the features, both because I think the tension between our interests pushes both classes of software engineer to do great things. Feature development keeps software and products relevant and addresses users' needs. Stabilization work makes projects last and reduces the incidence of failures that distract from feature work, and when there’s consistent attention paid to aligning infrastructure¹ work with feature development of the long term, infrastructure engineers can significantly lower the cost of implementing a feature.

The kinds of projects that fall into these categories inculde the following areas:

• managing application state and workload in larger distributed contexts. This has involved designing and implementing things like configuration management, deployment processes, queuing systems, and persistence layers.
• concurrency control patterns and process lifecycle. In programming environments where threads are available, finding ways to ensure that processes can safely shut down, and errors can be communicated between threads and processes takes some work and providing mechanisms to shutdown cleanly, communicate abort signals to worker threads, and handle communication patterns between threads in a regular and expected way, is really important. Concurrency is a great tool, but being able to manage concurrency safely and predictably and in descret parts of the code are useful.
• programming model and ergonomic APIs and services. No developers produces a really compelling set of abstractions on the first draft, particularly when they’re focused on delivering different kinds of functionality. The revision and iteration process helps everyone build better software.
• test infrastructure and improvements. No one thinks tests should take a long time or report results non-deterministically, and yet so many test are. The challenge is that tests often look good or seem reasonable or are stable when you write them, and their slow runtimes compound overtime, or orthogonal changes make them slower. Sometimes adding an extra check in some pre-flight test-infrastructure code ends ups causing tests that had been just fine, thank you to become problems. Maintaining and structure test infrastructure has been a big part of what I’ve ended up doing. Often, however, working back from the tests, it’s possible to see how a changed interface or an alternate factoring of code would make core components easier to test, and doing a cleanup pass of tests on some regular cadence to improve things. Faster more reliable tests, make it possible to develop with greater confidence.

In practice this has included:

• changing the build system for a project to produce consistent artifacts, and regularizing the deployment process to avoid problems during deploy.
• writing a queuing system without any extra service level dependencies (e.g. in the project’s existing database infrastructure) and then refactoring (almost) all arbitrary parallel workloads to use the new queuing system.
• designing and implementing runtime feature flagging systems so operators could toggle features or components on-and-off via configuration options rather than expensive deploys.
• replacing bespoke implementations with components provided by libraries or improving implementation quality by replacing components in-place, with the goal of making new implementations more testable or performant (or both!)
• plumbing contexts (e.g. Golang’s service contexts) through codebases to be able to control the lifecycle of concurrent processes.
• implementing and migrating structured logging systems and building observability systems based on these tools to monitor fleets of application services.
• Refactoring tests to reuse expensive test infrastructure, or using table-driven tests to reduce test duplication.
• managing processes' startup and shutdown code to avoid corrupted states and efficiently terminate and resume in-progress work.

When done well (or just done at all), this kind of work has always paid clear dividends for teams, even when under pressure to produce new features, because the work on the underlying platform reduces the friction for everyone doing work on the codebase.

My friend Jo wrote a post a while ago that addressed the subject of building an audience for your scholarly work. You can read the post on her blog, here.

One of the things that I think Jo is really great at is thinking practically about academic careers and trajectories in light of the current academic job market. While people working in traditional academic spaces and on a traditional academic course have a different set of challenges than folks like me, her points still resonate.

How do you build networks and audiences? Two things:

1. You talk to people.

Audiences are built on relationships. While we might like to think that writers and scholars are able to attract audiences purely on the basis of their work, in practice additional work is required.

2. You make sure you have something to show for yourself.

Everyone’s got ideas, and projects that they’d like to work on. People love to talk about their ideas. Success, I think, comes when you have something to show for yourself and your projects, and give people some level of confidence that your can make good on your ideas.

In sort, write more, publish more. While quality matters some, being more than someone to talks well at parties is really important.

I think this approach is useful for people doing any kind of creative or intellectual work that engages an audience, but I’m interested in your thoughts.

All of my friends who have taught composition are appalled when they hear me say that I want to teach writing. But it’s true: I would be interested in having the opportunity to give people the kind of writing education that I never got to have. I’ve even collected a few of these ideas on a very rough “pedagogy” page. This post, by contrast, will be a list of “things I wish I could have learned before I got a job writing.”

• How to write in long form. The skils and process for writing something that’s a hundred pages is fundamentally different from the process for writing pieces that are a few hundred words or a few pages. Project management, planning, and organization are totally different skills.
• Working with editors. In school, the editing process is very conversational. Editors, comment and ask you to make changes if agree with their judgment. Writers need to learn how to gather requirements, write the best possible content, and then hand it over to an editor who will modify the text without comment. Not only is it important to learn how to “get over this,” but also in how to learn from this kind of editing
• How to revise work. While I’ve learned to avoid making a number of mistakes to which I’m particularly prone, and spot those errors when the slip through, it’s really the process of applying for jobs that has taught me how to revise my own writing. Revision is probably the hardest writing skill, and I think there are probably better ways to teach revision than some sort of idealized “drafting process.”
• How to write at volume, even when you’re not feeling inspired. We’re pretty good at teaching people to write when they’re inspired or have done a lot of research. But writing Writing needs to be as instinctive as speech and the kind of thing that you don’t need to be inspired to be able to do. Not because anyone writes that much, but it’s a comfort thing.
• How to document things. Which is to say, how to record a practice, wprocedure, or interface, to tell people (and your future self,) how to do something. I had to figure this out on my own, and I think people would be much better writers for being 10% worse at writing essays and 10% better at writing processes.

That would do it! I’ve included some work in this direction in the pedagogy page, but comments, are always valuable.

I read this post by one of the partners in one of the coolest web services around, you should open that in a new window and then come back.

Back? Great!

Lars, proposes crowd-funding as a way to support free software development. Basically, run a “sponsor me to develop stuff” program, but rather than fund free software as a start-up around a single project or work for a big vendor.

It’s a nifty idea, and it’s got me thinking about micro-entrepreneurship. This would be where you make or do things, but not on a big scale. The businesses you create are small, and probably aren’t completely full-time equivalent, but in aggregate it’s good enough. While this is not the most prominent form of entrepreneurship on the internet, my sense is that it’s way bigger than most people think.

We’re too used to seeing multi-million dollar venture capital fund raising, IPOs, big acquisition deals, to realize the multitude of people who are making a few to several tens of thousands of dollars doing much smaller amounts of work.

I suppose I could write a whole post on good enough economics in the vein of this post on patronage from JamesGovernor but I’ll just leave a place holder link to a wiki page, in case someone else wants to fill things in.

Thoughts:

• Service-businesses don’t scale particularly well, any individual work can only produce so much work, and it’s hard to make individuals any more productive. In light of that, large service-based firms are unlikely to form.
• Most people have pretty specialized skills and abilities. Self-employment, particularly full time employment makes it difficult for people to spend most of their time doing what their best at. Specialization and differing skills is also what creates a market for service-based endeavors.
• Lacking health care and other benefits of traditional employment, it’s hard for people to be more self-employed and less conventionally-employed. Given this, doing entrepreneurial projects on a smaller scale makes more sense.
• Some kinds of entrepreneurial activities are attractive because, while they may not produce the same level of income as a salaried position, they allow more freedom and flexibility. This is the conventional justification for self-employment, and also the reason that most aligns with the “good enough” policy.

The problem with these kinds of “little businesses,” is that it’s too easy to focus on income earning work (e.g. freelance, and client work,) at the expense of doing basic work (e.g. developing core free software, doing basic research, writing fiction.) While the crowd sourcing notion makes a lot of sense, it requires a lot of faith in the crowd. I’m also unsure of how sustainable it is: while individuals can justify small amounts of money for such purposes, organizations cannot. Without organizational support, revenue is much lower, and it probably puts the larger financial burden on the smaller users, relatively speaking.

Continuing from the discussion regarding intellectual practice, I’ve been talking with a number of people (my father in particular) about graduate school and the prospect of “bootstrapping” a scholarly practice using “new media,” like blogging, and wiki making. I want to explore both my thoughts graduate school and bootstrapping with new media, and as you’d expect both of these ideas are rather intertwined. My initial gloss follows:

Bootstrapping for Success

The “new media,” even 10 or more years on, is still quite new. The media shift and technological changes have had a pretty clear impact on economic and industry practices. At the same time, reading, participation, and writing are still in flux. People say, “oh look, blogging and wikis; we can use this as a teaching and learning tool!” and then there are classes, tools, and software to integrate blogging into courses and learning management systems, but the media itself is still in flux and I’m not sure that anyone has blogging and wikis (as an example) figured out.

While the changes in new media are important, the changes to education itself is probably more important. Educators of all kinds have begun to take this we begin to think about the was that traditional education has changed and will change. Given new media, a changing job market, and the shifting economics of education it’s hard to think that education isn’t changing.

I’m not sure it’s changing that much.

There are cases of successful auto-didacts, and people who’ve been able acheive success

I’d love to be wrong about this, but I’m pretty sure that the only people who blog/wiki and have found real success in fields are people with some other more conventional route to success: people who are already successful and figure out how to use new media, people who have conventional training or have achieved success in traditional media and then moved to using blogs:

Some examples: Cory Doctorow began publishing fiction conventionally and doing freelance work for Weird magazine, and became a blogger and used that to multiply existing success. John Scalzi (and Tobias Buckell) published non-fiction and had successful professional writing careers before beginning to blog and write and publish fiction. The Valve is a successful academic blog/publication/forum, but as near as I can tell all of the contributors have traditional literary training, and all/most have academic postings. Bitch, Ph.D. has/had a formal background.

Samuel Delany doesn’t have formal training but has had a scholarly career, and while his is an inspiring story there’s not much that’s reproducible from it given some historical constraints: he started publishing before the demise of SF pulp magazines and Ace Double, because creative writing hadn’t been established when he entered the academy, etc.

I’m certainly willing to believe that my sample is skewed, and that people have been able to move in the other direction (from online success to conventional success, or been able to bootstrap their own success online,) but I can’t think of a single anecdote. I’d love to be proved wrong here.

Disciplining and Formal Education

I think that working as a technical writer is something to which I am very suited, something that provides a great deal of value, gives me access to the kinds of people that I’m interested in talking with (software developers, admins.) And writing experience and skill is largely fungible, so the skill I’m honing and developing is very transferable.

So, while I’m not opposed to doing academic work eventually, I’m pretty sure that no matter what kind of industry work I end up doing (product management, community management/organization, training, etc.) I’ll sill basically be a technical writer. And here’s the thing, if graduate school has no effect on my career except dominating my time and earning potential for a few years? It becomes very difficult to justify.

The equation that keeps going through my head is: two job searches within a few years years¹ and a hundred thousand dollars or more,² for what amounts to a personal betterment project. It’s not getting any easier to justify.

Here’s the catch: I’m a decent writer and I’m getting better all the time can I write or help people write books, articles, essays, stories, and a whole host of more specific forms. I’m not really sure that I could write a quality academic paper without an unreasonable amount of effort. I don’t know the process, I don’t know how to start, which literature to look at for resources, or for models, I’m not sure where the line between concision and complexity is in academic prose, and so forth. That’s the kind of knowledge that I’m certain I could get out of graduate education. And perhaps I’ve been a technical writer for too long, but I think not being able to “write like a scholar” makes it hard to participate in scholarly discussions.

The Remains of the Practice

I’m not sure where this leaves me. I’m thinking about seeing if I can take a seminar and a methods class at CUNY in the next year I might be able to get what I need. The right collaborative project might be a good way to build the required skills, but that’s even more complicated. As far as using the blog/wiki to build and participate in a conversation about new media practices, collaboration, and digital labor practices… there is much work left to be done.

What I want, it seems to me, isn’t a career--I have one of those--but to sustain intellectual life and practice. I would like to be able to ask questions, read seriously, participate in important conversations, and to write about this work and practice effectively for an audience that is invested in these discussions. This post is a follow up to my “career pathways” post.

I have a blog and wiki, I can read, and my writing continues to improve. How hard can it be to achieve these goals and establish this practice on my own? Famous last words.

The thing is, I hate auto-didacticism as an approach to knowledge production and learning. Sure it works, sometimes, and professionally I think I’ve been able to succeed on the basis of being able to learn things on my own. At the same time, self teaching at more advanced levels, and avoiding formal study feels like a mechanism for people to use to avoid challenging themselves or their assumptions about the world. The challenge here, in addition to discipline (in a number of senses of the word,) is to avoid scholarly isolationism.

Conversely, it might be true that sufficiently advanced study is always already self-lead and self-taught anyway. That’s not a conjecture I have the experience or specialty to comment upon, but it’s a possibility.

In any case, my success at being able to do meaningful and fulfilling work, hinges upon:

• being able to write and interact effectively for your communications medium. In my case this means, use blogging and wikis well.
• being able to maintain an active presence and participation in the discussions and work you want to do. This means posting regularly, in addition to writing, reading, and thinking about various projects. Work needs to be sustained and ongoing.
• being able to make leisure time sacrifices to support the work. There’s only so much time in the day, and I think it’s also important to manage expectations somewhat in recognition of this fact.
• being able to find or establish and interact with a community of peers. Regardless of interest or focus, it’s important to find colleagues who do work that is enough like yours to allow them to grasp the intricacies of your work and different enough to infuse the conversation with useful context and ideological breadth.

At least, that’s my hope. What am I forgetting?

I always thought that I would go to graduate school sometime in my twenties. I ask questions that are pretty geeky and difficult to answer, I think learning and research are pretty important, and I want to talk to people about ideas, projects, and theories. There are skills I need to be able to address the questions I have and background literates that I wish I were way more familiar with.

It seems like a good fit. Right?

Right. Well, right only if we accept that graduate school is a mechanism for personal betterment. While that has to be part of it, mostly graduate school is a job and the first in a long line of possible jobs. The academic career path has merits and demerits, but it’s still work, and I think to ignore this, makes it possible to accept atrocious labor practices in the academic world.

Somehow, without much intentionally on my part, I’ve found a career that I enjoy. Even more curious is the fact that being a professional writer with some technical background is the kind of thing that enjoys a certain kind of perpetual demand. And better yet, it’s impossible to get a degree to support this career: as near as I can tell literature degrees, history degrees, theology degrees, theater degrees, and psychology degrees are all equally relevant and irrelevant.

While I’m not convinced that I never want to teach, if the “getting a job” portion of going to graduate school is somewhat moot, then I’m left with a couple of questions:

• If additional schooling doesn’t affect career options and possibilities, then does it make sense to spend significant time in pursuit of an advanced degree?
• How do I develop and maintain an intellectual and scholarly practice without graduate school?
• How do I prevent my career from stagnating and from getting stuck in less than ideal jobs in mid-career and late career stages?
• I work in field where the need for human labor is constantly (and ideally) being automated away. The conventional wisdom is “develop specialties, but don’t get too cemented in a particular function so that you have options for after your job gets replaced. Combined with the orthogonal issue that writing and the work of writers is horribly misunderstood by just about everyone, figuring out “career paths is not necessarily easy. How do I deal with this long term concern in a more manageable way while being mindful of the future concerns.

While I sometimes feel like this blog can stray into the “overly meta”, I think that prefer intentionally over aimless wandering. Indeed, I think this career issue might have been a great deal easier for me had I figured some of these things out earlier. I know that we don’t always find clear and definitive answers to these problems and that solutions come in pieces and very slowly.

This series is about thinking about these issues to increase the possibility of intentionally and to document my process so that people can provide feedback. With luck, this will also help form a model for people who want to think about ways of contributing to scholarly conversations and grow intellectually, but needn’t do that in the context of the academic training and labor market.

I look forward to hearing from you and working with you all!