I end up reviewing a lot of code, and while doing code review (and getting reviews) used to take up a lot of time, I think I’ve gotten better at doing reviews, and knowing what’s important to comment on and what doesn’t

• The code review process is not there to discover bugs. Write tests to catch bugs, and use the code review process to learn about a specific change, and find things that are difficult to test for.
• As yourself if something is difficult to follow, and comment on that. If you can’t figure out what something is doing, or you have to read it more than once, then that’s probably a problem.
• Examine and comment on the naming of functions. Does the function appear to do what the name indicates.
• Think about the interface of a piece code:
  • What’s exported or public?
  • How many arguments do your functions take?
• Look for any kind of shared state between functions, particularly data that’s mutable or stored in globally accessable, or package local structures.
• Focus your time on the production-facing, public code, and less on things like tests and private/un-exported APIs. While tests are important, and it’s important that there’s good test coverage (authors should use coverage tooling to check this), and efficient test execution, beyond this high level aspect, you can keep reading?
• Put yourself in the shoes of someone who might need to debug this code and think about logging as well as error handling and reporting.
• Push yourself and others to be able to get very small pieces of code reviewed at a time. Shorter reviews are easier to process and while it’s annoying to break a logical component into a bunch of pieces, it’s definitely worth it.

After, my post on what I do for work I thought it’d be good to describe the kinds of things that make software easy to work on and collaborative. Here’s the list:

1. Minimal Documentation. As a former technical writer this is sort of painful, but most programning environments (e.g. languages) have idioms and patterns that you can follow for how to organize code, run tests and build artifacts. it’s ok if your project has exceptional requirements that require you to break the rules in some way, but the conventions should be obvious and the justification for rule-breaking should be plain. If you adhere to convention, you don’t need as much documentation. It’s paradoxical, because better documentation is supposed to facilitate accessibility, but too much documentation is sometimes an indication that things are weird and hard.
2. Make it Easy To Run. I’d argue that the most difficult (and frustrating) part of writing software is getting it to run everywhere that you might want it to run. Writing software that runs, even does what you want on your own comptuer is relatively, easy: making it work on someone else’s computer is hard. One of the big advantages of developing software that runs as web apps means that you (mostly) get to control (and limit) where the software runs. Making it possible to easily run a piece of software on any computer it might reasonably run (e.g. developer’s computers, user’s computers and/or production environments.) Your software itself, should be responsible for managing this environment, to the greatest extent possible. This isn’t to say that you need to use containers or some such, but having packaging and install scripts that use well understood idioms and tools (e.g. requirements.txt, virtualenv, makefiles, etc.) is good.
3. Clear Dependencies. Software is all built upon other pieces of software, and the versions of those libraries are important to record, capture, and recreate. Now it’s generally a good idea to update dependencies regularly so you can take advantage of improvements from upstream providers, but unless you regularly test against multiple versions of your dependencies (you don’t), and can control all of your developer and production environments totally (you can’t), then you should provide a single, centralized way of describing your dependencies. Typically strategies involve: vendoring dependencies, using lockfiles (requirements.txt and similar) or build system integration tends to help organize this aspect of a project.
4. Automated Tests. Software requires some kind of testing to ensure that it has the intended behavior, and tests that can run quickly and automatically without requiring users to exercise the software manually is absolutely essential. Tests should run quickly, and it should be possible to run a small group of tests very quickly to support iterative development on a specific area of the code. Indeed, most software development can and should be oriented toward the experience of writing tests and exercising new features with tests above pretty much everything else. The best test suites exercise the code at many levels, ranging from very small unit tests to ensure the correct behavior of the functions and methods, to higher level tests that test the functionality of higher-order functions and methods, and full integration tests that test the entire system.
5. Continuous Integration. Continuous integration system’s are tools that support development and ensure that changes to a code pass a more extensive range of tests than are readily available to developers. CI systems are also useful for automating other aspects of a project (releases, performance testing, etc.) A well maintained CI environment provide the basis for commonality for larger projects with a larger number for projects larger groups of developers, and is all but required to ensure a well supported automated test system and well managed dependency.
6. Files and Directories Make Sense. Code is just text in files, and software is just a lot of code. Different languages and frameworks have different expectations about how code is organized, but you should be able to have a basic understanding of the software and be able to browse the files, and be able to mostly understand what components are and how they relate to each other based on directory names, and should be able to (roughly) understand what’s in a file and how the files relate to eachother based on their names. In this respect, shorter files, when possible are nice, are directory structures that have limited depth (wide and shallow,) though there are expections for some programming language.
7. Isolate Components and Provide Internal APIs. Often when we talk about APIs we talk about the interface that users access our software, but larger systems have the same need for abstractions and interfaces internally that we expose for (programmatic) users. These APIs have different forms from public ones (sometimes,) but in general:
   • Safe APIs. The APIs should be easy to use and difficult to use incorrectly. This isn’t just “make your API thread safe if your users are multithreaded,” but also, reduce the possibility for unintended side effects, and avoid calling conventions that are easy to mistake: effects of ordering, positional arguments, larger numbers of arguments, and complicated state management.
   • Good API Ergonomics. The ergonomics of an API is somewhat ethereal, but it’s clear when an API has good ergonomics: writing code that uses the API feels “native,” it’s easy to look at calling code and understand what’s going on, and errors that make sense and are easy to handle. It’s not simply enough for an API to be safe to use, but it should be straightforward and clear.

i am a self taught programmer. i don’t know that i’d recommend it to anyone else there are so many different curricula and training programs that are well tested and very efficacious. for lots of historical reasons, many programmers end up being all or mostly self taught: in the early days because programming was vocational and people learned on the job, then because people learned programming on their own before entering cs programs, and more recently because the demand for programmers (and rate of change) for the kinds of programming work that are in the most demand these days. and knowing that it’s possible (and potentially cheaper) to teach yourself, it seems like a tempting option.

this post, then, is a collection of suggestions, guidelines, and pointers for anyone attempting to teach themselves to program:

• focus on learning one thing (programming language and problem domain) at a time. there are so many different things you could learn, and people who know how to program seem to have an endless knowledge of different things. knowing one set of tools and one area (e.g. “web development in javascript,” or “system administration in python,”) gives you the framework to expand later, and the truth is that you’ll be able to learn additional things more easily once you have a framework to build upon.

• when learning something in programming, always start with a goal. have some piece of data that you want to explore or visualize, have a set of files that you want to organize, or something that you want to accomplish. learning how to program without a goal, means that you don’t end up asking the kinds of questions that you need to form the right kinds of associations.

• programming is all about learning different things: if you end up programming for long enough you’ll end up learning different languages, and being able to pick up new things is the real skill.

• being able to clearly capture what you were thinking when you write code is basically a programming super power.

• programming is about understanding problems¹ abstractly and building abstractions around various kinds of problems. being able break apart these problems into smaller core issues, and thinking abstractly about the problem so that you can solve both the problem in front of you and also solve it in the future are crucial skills.

• collect questions or curiosities as you encounter them, but don’t feel like you have to understand everything, and use this to guide your background reading, but don’t feel like you have to hunt down the answer to every term you hear or see that you don’t already know immediately. if you’re pretty rigorous about going back and looking things up, you’ll get a pretty good base knowledge over time.

• always think about the users of your software as you build, at every level. even if you’re building software for your own use, think about the future version of yourself that will use that software, imagine that other people might use the interfaces and functions that you write and think about what assumptions they might bring to the table. think about the output that your program, script, or function produces, and how someone would interact with that output.

• think about the function as the fundamental building block of your software. lower level forms (i.e. statements) are required, but functions are the unit where meaning is created in the context of a program. functions, or methods depending, take input (arguments, ususally, but sometimes also an object in the case of methods) and produce some output, sometimes with some kind of side-effect. the best functions:

  • clearly indicate side-effects when possible.
  • have a mechanism for reporting on error conditions (exceptions, return values,)
  • avoid dependencies on external state, beyond what is passed as arguments.
  • are as short as possible.
  • use names that clearly describe the behavior and operations of the function.

  if programming were human language (english,) you’d strive to construct functions that were simple sentences and not paragraph’s, but also more than a couple of words/phrases, and you would want these sentences to be clear to understand with limited context. if you have good functions, interfaces are more clear and easier to use, code becomes easier to read and debug, and easier to test.

• avoid being too weird. many programmers are total nerds, and you may be too, and that’s ok, but it’s easier to learn how to do something if there’s prior art that you can learn from and copy. on a day-to-day basis, a lot of programming work is just doing something until you get stuck and then you google for the answer. If you’re doing something weird--using a programming language that’s less widely used, or in a problem space that is a bit out of mainstream, it can be harder to find answers to your problems.

Notes

Editoral Note: this is a follow up to my earlier Principles of Test Oriented Software Development post.

In software development, we write tests to make sure the code we write does what we want it to do. Great this is pretty easy to get behind.

Tests sometimes fail.

The goal, is that, most of the time when tests fail, it’s because the code is broken: you fix the code and the test passes. Sometimes when test fail there’s a bug in the test, it makes an assertion that can’t or shouldn’t be true: these are bad because they mean the test is broken, but all code has bugs, and test code can be broken so that’s fine.

Ideally either pass or fail, and if a test fails it fails repeatedly, with the same error. Unfortunately, this is of course not always true, and tests can fail intermittently if they test something that can change, or the outcome of the test is impacted by some external factor like “the test passes if the processor is very fast, and the system does not have IO contention, but fails sometimes as the system slows down.” Sometimes tests include (intentionally or not) some notion of “randomnesses,” and fail intermittently because of this.

A test suite with intermittent failures is basically the worst. A suite that never fails isn’t super valuable, because it probably builds false confidence, a test suite that always fails isn’t useful because developers will ignore the results or disable the tests, but a test that fails intermittently, particularly one that fails 10 or 20 percent of the time, means that developers always will always look at the test, or just rerun the test until it passes.

There are a couple of things you can do to fix your tests:

• write better tests: find sources of non-determinism in your test and rewrite tests to avoid these kinds of “flaky” outcomes. Sometimes this means restructuring your tests in a more “pyramid-like” structure, with more unit tests and fewer integration tests (which are likely to be less deterministic.)
• run tests more reliably: find ways of running your test suite that produce more consistent results. This means running tests in more isolated environments, changing the amount of test parallelism, ensure that tests clean up their environment before they run, and can be as logically isolated as possible.

But it’s hard to find these tests and you can end up playing wack-a-mole with dodgy tests for a long time, and the urge to just run the tests a second (or third) time to get them to pass so you can merge your change and move on with your work is tempting. This leaves:

• run tests multiple times: so that a test doesn’t pass until it passes multiple times. Many test runner’s have some kind of repeated execution mode, and if you can combine with some kind of “stop executing after the first fail,” then this can be reasonably efficient. Use multiple execution to force the tests to produce more reliable results rather than cover-up or exacerbates the flakiness.
• run fewer tests: it’s great to have a regression suite, but if you have unreliable tests, and you can’t use the multi-execution hack to smoke out your bad tests, then running a really full matrix of tests is just going to produce more failures, which means you’ll spend more of your time looking at tests, in non-systematic ways, which are unlikely to actually improve code.

I want to like test-driven-development (TDD), but realistically it’s not something that I ever actually do. Part of this is because TDD, as canonically described is really hard to actually pratice: TDD involves writing tests before writing code, writing tests which must fail before the implementation is complete or correct, and then using the tests to refactor the code. It’s a nice idea, and it definitely leads to better test coverage, but the methodology forces you to iterate inefficiently on aspects of a design, and is rarely viable when extending existing code bases. Therefore, I’d like to propose a less-dogmatic alternative: test-oriented-development.¹

I think, in practice, this largely aligns with the way that people write software, and so test oriented development does not describe a new way of writing code or of writing tests, but rather describes the strategies we use to ensure that the code we write is well tested and testable. Also, I think providing these strategies in a single concrete form will present a reasonable and pragmatic alternative to TDD that will make the aim of “developing more well tested software” more achievable.

1. Make state explicit. This is good practice for all kinds of development, but generally, don’t put data in global variables, and pass as much state (configuration, services, etc.) into functions and classes rather than “magicing” them.
2. Methods and functions should be functional. I don’t tend to think of myself as a functional programmer, as my tendencies are not very ideological, in this regard, but generally having a functional approach simplifies a lot of decisions and makes it easy to test systems at multiple layers.
3. Most code should be internal and encapsulated. Packages and types with large numbers of exported or public methods should be a warning sign. The best kinds of tests can provide all desired coverage, by testing interfaces themselves,
4. Write few simple tests and varry the data passed to those tests. This is essentially a form of “table driven testing,” where you write a small sequence of simple cases, and run those tests with a variety of tests. Having test infrastructure that allows this kind of flexibility is a great technique.
5. Begin writing tests as soon as possible. Orthodox TDD suggests that you should start writing tests first, and I think that this is probably one of the reasons that TDD is so hard to adopt. It’s also probably the case that orthodox TDD emerged when prototyping was considerably harder than it is today, and as a result TDD just feels like friction, because it’s difficult to plan implementations in a test-first sort of way. Having said that, start writing tests as soon as possible.
6. Experiment in tests. Somehow, I’ve managed to write a lot of code without working an interactive debugger into my day-to-day routine, which means I do a lot of debugging by reading code, and also by writing tests to try and replicate production phenomena in more isolated phenomena. Writing and running tests in systems is a great way to learn about them.

It’s not real secret that I’m red-green colorblind. It’s not really a major life obstacle: I’ve got a wardrobe of clothes in colors that I can easily tell apart, I have developed a number of heuristics for guessing colors that are right enough, and mostly it just creates funny stories where I get a confused look if I try to describe something that might be purple, or have to convince a coworker into reading a graph for me.

One challenge historically, however, has been various kinds of text editing color themes: so often they end up having some kind of low contrast situation that’s hard to read, or two different aspects of code that should be highlighted differently but aren’t. I’ve tried lots of themes out, and I would always end up just going back and using default emacs themes, which might not have been great, but I always found it useable.

Until Protesilaos Stavrou’s Modus Themes, that is.

These are super compelling and I never really knew how good an editor could look until I started using it. Like is this what it’s really like for the rest of you all the time? Things are clear: I never get confused between type names and function names any more. There are rarely situations where I feel like the highlighting color and text color are the same, which used to happen all the time.

The real win, I think, is that Modus’ makes dark themes accessible to me, in a way that they never were before. For the most part “dark themes” which have been so popular recently, are just impossible to see clearly (for me), I also find that it’s less taxing to spend time in front of screens when darker backgrounds, so being able to spend most of my time doing work in an environment that’s easy to read. I tend to keep to light backgrounds when using a laptop and dark backgrounds otherwise.

The second piece is that, I think I’ve caved in and decided to increase the size of the font, by default in my text editor, at least when I’m using my desktop/external monitor. I think my vision is as good as it’s been, though I should probably get that checked out post-pandemic. I think there’s a balance between “small fonts let you see more of the file you’re working on,” and “larger fonts let you focus on the area that you’re editing.” When I’m writing English, the focus is great, and when writing software I tend to want more context. There’s a balance also in wanting to keep an entire line length viable at once, and an ideal words-per-line limit for text that’s useful for making things easier to read. So there’s some tuning there, depending on what your workload looks like.

I guess if there is any lesson in this it’s that: Comfort matters, and you shouldn’t push yourself into uncomfortable display situations if you can.

Cloud computing, and with it most of tech, has been really hot on the idea of “serverless” computing, which is to say, services and applications that are deployed, provisioned, and priced separate from conventional “server” resources (memory, storage, bandwidth.) The idea is that we can build and expose ways of deploying and running applications and services, even low-level components like “databases” and “function execution”, in ways that mean that developers and operators can avoid thinking about computers qua computers.

Serverless is the logical extension of “platform as a service” offerings that have been an oft missed goal for a long time. You write high-level applications and code that is designed to run in some kind of sandbox, with external services provided in some kind of ala carte model via integrations with other products or services. The PaaS, then, can take care of everything else: load balancing incoming requests, redundancy to support higher availability, and any kind of maintains on the lower level infrastructure. Serverless is often just PaaS but more: provide a complete stack of services to satisfy needs (databases, queues, background work, authentication, caching, on top of the runtime,) and then change the pricing model to be based on request/utilization rather than time or resources.

Fundamentally, this allows the separation of concerns between “writing software,” and “running software,” and allows much if not all of the running of software to be delegated to service providers. This kind of separation is useful for developers, and in general runtime environments seem like the kind of thing that most technical organizations shouldn’t need to focus on: outsourcing may actually be good right?

Well maybe.

Let’s be clear, serverless platforms primarily benefit the provider of the services for two reasons:

• serverless models allow providers to build offerings that are multi-tenant, and give provider the ability to reap the benefit of managing request load dynamically and sharing resources between services/clients.
• utilization pricing for services is always going to be higher than commodity pricing for the underlying components. Running your on servers (“metal”) is cheaper than using cloud infrastructure, over time, but capacity planning, redundancy, and management overhead, make that difficult in practice. The proposition is that while serverless may cost more per-unit, it has lower management costs for users (fewer people in “ops” roles,) and is more flexible if request patterns change.

So we know why the industry seems to want serverless to be a thing, but does it actually make sense?

Maybe?

Makers of software strive (or ought to) make their software easy to run, and having very explicit expectations about the runtime environment, make software easier to run. Similarly, being able to write code without needing to manage the runtime, monitoring, logging, while using packaged services for caching storage and databases seems like a great boon.

The downsides to software producers, however, are plentiful:

• vendor lock-in is real, not just because it places your application at the mercy of an external provider, as they do maintenance, or as their API and platform evolves on their timeframe.
• hosted systems, mean that it’s difficult to do local development and testing: either every developer needs to have their own sandbox (at some expense and management overhead), or you have to maintain a separate runtime environment for development.
• application’s cannot have service levels which exceed the service level agreements of their underlying providers. If your serverless platform has an SLA which is less robust than the SLA of your application you’re in trouble.
• when something breaks, there are few operational remedies available. Upstream timeouts are often not changeable and most forms of manual intervention aren’t available.
• pricing probably only makes sense for organizations operating at either small scale (most organizations, particularly for greenfield projects,) but is never really viable for any kind of scale, and probably doesn’t make sense in any situation at scale.
• some problems and kinds of software just don’t work in a serverless model: big data sets that exceed reasonable RAM requirements, data processing problems which aren’t easily parallelizable, workloads with long running operations, or workloads that require lower level network or hardware access.
• most serverless systems will incur some overhead over dedicated/serverfull alternatives and therefore have worse performance/efficiency, and potentially less predictable performance, especially in very high-volume situations.

Where does that leave us?

• Many applications and bespoke tooling should probably use serverless tools. Particularly if your organization is already committed to a specific cloud ecosystem, this can make a lot of sense.
• Prototypes, unequivocally make sense to rely on off-the-shelf, potentially serverless tooling, particularly for things like runtimes.
• If and when you begin to productionize applications, find ways to provide useful abstractions between the deployment system and the application. These kinds of architectural choices help address concerns about lock-in and making it easy to do development work without dependencies.
• Think seriously about your budget for doing operational work, holistically, if possible, and how you plan to manage serverless components (access, cost control, monitoring and alerting, etc.) in connection with existing infrastructure.

Serverless is interesting, and I think it’s interesting to say “what if application development happened in a very controlled environment with a very high level set of APIs.” There are clearly a lot of cases where it makes a lot of sense, and then a bunch of situations where it’s clearly a suspect call. And it’s early days, so we’ll see in a few years how things work out. In any case, thinking critically about infrastructure is always a good plan.

I made a tongue-in-cheek comment on twitter a while back that, k8s is just the contemporary API for mainframe computing., but as someone who is both very skeptical and very excited about the possibilities of kube, this feels like something I want to expand upon.

A lot of my day-to-day work has some theoretical overlap with kube, including batch processing, service orchestration, and cloud resource allocation. Lots of people I encounter are also really excited by kube, and its interesting to balance that excitement with my understanding of the system, and to watch how Kubernetes(as a platform) impacts the way that we develop applications.

I also want to be clear that my comparison to mainframes is not a disparagement, not only do I think there’s a lot of benefit to gain by thinking about the historic precedents of our paradigm. I would also assert that the trends in infrastructure over the last 10 or 15 years (e.g. virtualization, containers, cloud platforms) have focused on bringing mainframe paradigms to a commodity environment.

Observations

• clusters are static ususally functionally. I know that the public clouds have autoscaling abilities, but having really elastic infrastructure requires substantial additional work, and there are some reasonable upper-limits in terms of numbers of nodes, which makes it hard to actually operate elastically. It’s probably also the case that elastic infrastructure has always been (mostly) a pipe dream at most organizations.
• some things remain quite hard, chiefly in my mind:
  • autoscaling, both of the cluster itself and of the components running within the cluster. Usage patterns are don’t always follow easy to detect patterns, so figuring out ways to make infrastructure elastic may take a while to converse or become common. Indeed, VMs and clouds were originally thought to be able to provide some kind of elastic/autoscaling capability, and by and large, most cloud deployments do not autoscale.
  • multi-tenancy, where multiple different kinds of workloads and use-cases run on the same cluster, is very difficult to schedule for reliably or predictably, which leads to a need to overprovision more for mixed workloads.
• kubernettes does not eliminate the need for an operations team or vendor support for infrastructure or platforms.
• decentralization has costs, and putting all of the cluster configuration in etcd imposes some limitations, mostly around performance. While I think decentralization is correct, in many ways for Kubernetes, applications developers may need systems that have lower latency and tighter scheduling abilities.
• The fact that you can add applications to an existing cluster, or host a collection of small applications is mostly a symptom of clusters being over provisioned. This probably isn’t bad, and it’s almost certainly the case that you can reduce the overprovisioning bias with kube, to some degree.

Impact and Predictions

• applications developed for kubernettes will eventually become difficult or impossible to imagine or run without kubernettes. This has huge impacts on developer experience and test experience. I’m not sure that this is a problem, but I think it’s a hell of a dependency to pick up. This was true of applications that target mainframes as well.
• Kubernetes will eventually replace vendor specific APIs for cloud infrastructure for most higher level use cases.
• Kubernetes will primarily be deployed by Cloud providers (RedHat/IBM, Google, AWS, Azure, etc.) rather than by infrastructure teams.
• Right now, vendors are figuring out what kinds of additional services users and applications need to run in Kubernetes, but eventually there will be another layer of tooling on top of Kubernetes:
  • logging and metrics collection.
  • deployment operations and configuration, particularly around coordinating dependencies.
  • authentication and credential management.
  • low-latency offline task orchestration.
• At some point, we’ll see a move multi-cluster orchestration, or more powerful tools approach to workload isolation within a single cluster.

Conclusion

Kubernetes is great, and it’s been cool to see how, really in the last couple of years, it’s emerged to really bring together things like cloud infrastructure and container orchestration. At the same time, it (of course!) doesn’t solve all of the problems that developers have with their infrastructure, and I’m really excited to see how people build upon Kubernetes to achieve some of those higher level concerns, and make it easier to build software on top of the resulting platforms.