The Finish Line Is Just Another Milestone

Speaking of remodeling updating your code, If you’re releasing a new version of your code, that means you’ve already released a version of your code. That means that at some time in the past, you finished your code. Or at least thought you did. But now, here you are releasing a new version. So I guess you weren’t done. Or to put it another way,

The finish line is just another milestone.

Yup. Unlike a race, the finish line with software almost never means you’re done.¹ Especially with version 1, but really, for all versions. EVERY new project I’ve working on that made it to the first release, has had things that I knew could have been done better. Features I thought should have been added. Errors that needed better handling. From personal projects to school projects to AAA games to multi-year multi-million dollar defense projects released the initial version with known work I (and the team) wanted to do. Those things didn’t make the initial release, for very good reasons. Reasons like boredom and more pressing issues, homework due dates, hitting a release date for game sales, to contractual obligations. Time is fleeting, but there’s always an important date in there somewhere.

First, you work hard. Then you release. You take a break and regroup. You listen to what your users are saying. You combine that with what you know you didn’t do but thought you should. Then you do it all over again.

This time, the list of things you didn’t do is smaller, but it’s more targeted, because you know more. More about the problem space. More about the constraints you added the first time. More about what your users really want to do. You’re starting from a different place, but the process is pretty similar. You develop the code. You learn more along the way. You make mistakes and fix them. Then, you release again.

But again, you’re not done. There are still things you wanted to do. User requests you didn’t get to. Rough edges that you want to fix. And the cycle begins again.

And it doesn’t matter how long your cycle is. It could be a 2-week sprint. It could be 5-year plan. It doesn’t matter. Release is not the same as done.

That’s the real lesson here. Release is not a finish line. It’s just another milestone along the path. And like all those other milestones, there’s another one coming up. So be ready for it.

Home Ownership and Software Updates

Seen on Bluesky

Homeownership is a lifelong commitment to discovering what it takes to modernize a legacy system and the lengths to which you will go and compromises you will make in doing so.

It’s often been said that software development is like home construction. You have a site, some requirements, a plan, you start to build, then you adjust as needed. There’s some truth to that, and from far enough away, it’s accurate.

However, when you look closer the analogy falls apart. First, we’ve been building homes a lot longer than we’ve been writing programs. The number of things that are known to home builders as tacit knowledge vastly outnumbers the tacit knowledge of software developers. Most developers have built a handful of programs, or the same program many times. Especially developers that work on their own or in a small company. On the other hand, your typical home building company has a much larger legacy behind them. They made those mistakes before. And the industry made some of them so long ago that the fix is documents. As building codes.

Which is the second big difference. Home building is a regulated industry. Even if you’re building everything yourself, there aren’t many places left where you don’t have to follow the same rules and regulations as every other builder, and, your work will be inspected by a third party when it’s done. There aren’t many software projects where the local government software inspector comes by and makes sure you don’t have any dangling pointers, missing null checks, or all the known security patches. That almost certainly happened to your house though. At multiple times during the build, the inspector came out and checked the plumbing, the electrical, the framing, the layout of the stairs, and even how many electric outlets you have.

Those differences aside, one way that homes and software are similar is updates. At some point, potentially long after the original builder and owner are no longer involved, the current owner is going to reach out to a new builder and say “Can you change X because I want to do Y”. And the builder is going to say yes. Then figure out how to do it. Or at least try to, And find that there’s a pipe right where the new owner wants a doorway. Or the walls just aren’t strong enough to support a second floor. Or someone secretly added another set of outlets to an existing circuit, and plugging in one more phone charge blows the main breaker for the entire house.

So the builder figures out how to make things work. They convince the new owner that want the door four feet to the left, or they add a parallel electric system to the house, or the build a second story on stilts so it’s really a new house carefully place just above the current house, so the stairways line up, but in fact they’re completely separate.¹ Then it happens a few months later, and not only does the builder have to deal with the original build, but they also must deal with the compromises and issues that came from the first change. When you do a home remodel, you not only have to design in the things you want, you have to design in the choices of the original builder (and any change between now and then).

With software it’s kind of the same thing. Especially legacy software. Decisions were made. For probably good reasons at the time. To deal with a known requirement. As with Chesterton’s Fence, unless you know exactly why something was done and that it’s no longer needed, you shouldn’t change it. So when you have to add a new feature to a piece of software, the first thing you need to do is make sure you haven’t broken any of your current users (Hyrum’s Law).

That means each time you need to change something, whether it’s a building or software, unless you work very hard, and do things that are not directly related to the thing you’re trying to enable, you make it harder to make the next change.

So THINK before you make that change. Think about what you’re trying to do. Think about how you can do it without impacting other users. And especially, think about future developers and what they’ll need to do when they have to make yet another change.

Verification First Development

From the “Listen to Hillel” file, an article on Verification First Development. I really like the idea of Test Driven Development, but I have to be honest. I don’t always do it. Sometimes I practice Test After Development. Meaning I write my tests to verify that the code I wrote does what I wanted it to do. But after reading Hillel’s article, I realized that even though I sometimes write my tests after, I almost always know how I’m going to verify it. I have a positive test case in place, and I often have a large handful of known error cases. Having a name to put on things makes me feel good.

Another good thing about verification first is that it’s a great definition of done. And knowing when you’re done is critical. You’ve added some kind of value when you’re done. You get to stop what you’re doing when you’re done. You get to build up some momentum and celebrate when you’re done. You get to move on to the next thing when you’re done. These are all good things. And it’s part of the Purpose that Pink talked about in Drive. It helps motivate us.

And thinking about verification first development of course makes me think of my current job. At Aurora, our mission is to deliver the benefits self-driving technology, safely, quickly, and broadly. To do that, Aurora has a safety case. That safety case, which was built long before the code was finished, defines done. And not just in big broad terms, but also in detail. With specific requirements. To be done, we need to close the safety case. That means that a lot of work goes into building and executing the plan to verify that things are safe. So while different teams have their own flavors of development style, it’s all guided by a verification first mindset.

So, depending on what I’m developing, I might be using BDD, DDD, TDD, Test After Development, or sometimes I’m just experimenting to learn about a space and to figure out what I really want to do. Regardless of how I’m doing it though, there’s always at least a plan for how I verify the results are what I want.

The question now is, as that other Hillel would say,

If not now, then when?

Safety Nets and Guardrails

Safety nets and guardrails sound like the same thing, but they’re not. They are very similar though. They both help prevent bad things from happening. Where they differ is in how and when they operate.

Safety nets help after something bad has happened. It’s what you do when things go wrong. Your traditional safety net catches something that has fallen. It could be a person off a roof, or a trapeze artist that missed a catch. It could also be a process that helps recover from a negative event. Like insurance (be it life, health, auto, home, or unemployment). It doesn’t mean the bad thing can’t happen, or that there will be no consequences, but it minimizes the negative impact/damage caused by the bad thing happening.

Or in the software world it could be a top-level error handling routine or executing a series of SQL statements inside a transaction so you can safely roll things back if there’s an error. Put another way, it’s using both a belt and suspenders even when your pants fit. Normally, the pants stay in place by themselves, but if they don’t for some reason, you’ve got the belt to hold your pants up. And if the belt snaps, there’s still the set of suspenders to hold them up. In terms of ilities, it’s resilience. ¹

Guardrails, on the other hand, help prevent something bad from happening. Like the guardrails along the highway. They work to keep cars on the road and heading in the right general direction. It doesn’t mean you can’t force your way off the road, or that everything will still be perfect if you end up needing the guardrail, but things will be much better with the guardrail, and you’ll probably still get to your destination. It’s Poka Yoke, which I’ve talked about before. And just like you can have multiple levels of safety nets, you can have multiple guardrails. Like the rumble strips on a highway that tell you you’re drifting, before the guardrail pushes you back on track, both of them help you do the right thing.

In software, guardrails come in multiple flavors. It’s using types instead of primitive obsession. Sure, you could use a string to store a value that is one of a limited set, but it can also store many invalid strings. If you instead use an ENUM that only supports the limited set, the user simply can’t set the value to something invalid. Another guardrail is using a builder to initialize something so that it either works or tells you immediately that it can’t be initialized instead of leaving you with something that won’t work. There are lots of other guardrails you can add to your software.

And remember, while safety nets and guardrails have the same basic goal, keep something terrible from happening, they are, in fact, orthogonal ideas. Which means you can (and should) use them both. Use guardrails that make it easy to use your API/functions the right way and hard to use them incorrectly. But recognize that it can still happen. So also include safety nets, so that when something is wrong, it gets handled the best way possible.

With Apologies to Uncle Bob

From the design by counter-example file, this might be the most practical definition of “unclean” code I’ve ever seen.

In the kitchen, the stuff I left on the counter is fine, I know why it’s there. Everything my family leaves on the counter is mess.

In our own software, we don’t trip over the rough edges, we can fix those later. For everyone else, our software is rough.

– Jessica Joy Kerr

In your own kitchen, you don’t see your own clutter. It’s not bad, but there’s not a lot of available workspace. Similarly, that’s why everyone else’s code is not clean, but your code is. At least in your eyes.

Clean code is good goal. And there are lots of heuristics and rules of thumb to help you get write clean code. You should always be thinking about them. Not blindly following them, but thinking about them. And you need to be aware of your biases and blind spots.

And one of the biggest, the one that makes identifying your own unclean code, is the same one that makes it hard to accurately and effectively edit your own writing. It’s a problem of context. When you’re writing, whether it’s code, a novel, an email, or a text message, you have an immense amount of context. When you go back to review/edit that text, unless it’s been a long time, you still have all that context. And even if it has been a long time, that context will come back pretty quickly. That means you don’t see the missing or doubled words. You don’t see the misspellings. You don’t notice that your functions are long, that you have complicated conditionals, or that function and variable names no longer match what they actually do.

Unfortunately, unless you’re the copy editor for someone, everyone else has much less context. They don’t know what you know and they see those things immediately. It makes it hard for them to understand your code. Just like it makes it hard for you to understand their code.

Or to work in someone else’s kitchen.

So the next time you’re reviewing your own code prior to getting someone else to review it, make sure you’re looking at not just with your own context, but also the context of someone who hasn’t seen it before.

Testing Schedules

Yesterday I talked about different kinds of tests, [unit, integration, and system](/posts/2025/04/02]. I mentioned that not only are there different kinds of tests, but those tests have different characteristics. In addition to the differences in what you can learn from tests by classification, there are also differences in execution times and execution costs.

NB: This is semi-orthogonal to test driven development. When you run the tests is not the same as when you write the tests.

These differences often lead to different schedules for running these tests. Like any dynamic system, one important tool to maintain stability is to have a short feedback loop. The faster the tests run, and the more often you run them, the shorter you can make your feedback loop. The shorter your feedback loop, the faster you can get to the result you want.

Luckily, unit tests are both cheap and fast. That means you can run them a lot. And get results quickly. The questions are, which ones do you run, and what does “a lot” mean? If you’ve got a small system, and running all the tests takes a few seconds, run them all. While building and running any individual test is fast, in a more realistic setting, building and running them all can take minutes or hours. And that’s just not practical. You need to do something else. This is where a dependency management system, like Make or bazel can help. You can set them up to only run the tests that are directly dependent on the code that changed. Combine that with some thoughtful code layout and you can relatively easily keep the time it takes to run the relevant (directly impacted) tests down.

Running quickly is important, but when do you run them? Recommendations vary from “every time a file is changed/saved” to every time changes are stored in your shared version control system. Personally, I think it’s every time there’s a logical change made locally. Sometimes logical changes span files, so testing based on any given file doesn’t make sense. You want to run the tests before you make a change to make sure the tests are still valid, you want to run the tests after each logical step in the overall change to make sure your changes haven’t broken anything, and you want to run the tests when you’re done to make sure that everything still works. That’s a good start, but it’s not enough. In a perfect world your unit tests would cover every possible combination of use cases for the SUT. But we don’t live in a perfect world, and as Hyrum’s Law tells us, someone, somewhere, is making use of some capability you don’t know you’ve exposed. A capability you don’t have a unit test for. So even when all your unit tests pass, you can still break something downstream. At some point you need to run all the unit tests for all the code that depends on the change. Ideally before anyone else sees the change. You run all those tests just before you push your changes to the shared version control system.

Unfortunately, unit tests aren’t enough. Everything can work properly on its own, but things also must work well together. That’s why we have integration tests in the first place. When do you run them? They cost more and take longer than integration tests, but the same basic rule applies. You should run them when there’s a complete logical change. That means when any component of any integration test changes, run the integration test. And again, just running the directly impacted integration tests isn’t enough. There will be integration tests that depend on the things that depend on the integrations you’re testing. You need to test them as well. Again, ideally before anyone else sees the change.

Then we have system level, or end-to-end tests. Those tests are almost always slow, expensive, and take real hardware. Every change impacts the system, but it’s just not practical to run them for every change. And even if you did, given the time it takes to run those tests (hours or days if there’s real hardware involved), running them for every change would slow you down so much you’d never get anything done. Of course, you need to run your system level tests for every release, or you don’t know what you’re shipping, but that’s not enough. You need to run the system tests, or at least the relevant system tests, often enough that you’ve got a fighting chance to figure out which change made the test fail. That’s dependent on the rate of change of your system. For systems under active development that might be every day or at least multiple times per week, for systems that rarely change, it might be much less frequently.

There you have it. Unit tests on changed code run locally on every logical change, before sharing, and centrally on everything impacted by the change after/as part of sharing with the rest of the team. Integration tests run on component level changes locally before sharing, and centrally on everything impacted by the change after/as part of sharing with the rest of the team. System level tests run on releases and on a schedule that makes sense based on the rate of change of the system.

Bonus points for allowing people to trigger system tests when they know there’s a high likelihood of emergent behavior to check for.

Test Classification

Whether you are thinking about unit vs integration vs system tests, or build (or save) time vs check in time vs release time tests, what you’re really thinking about is test classification and test hierarchy. Or put another way, you’re thinking about why you’re running that test and what the goal of the test is.

Of course you want the test to pass. And you want that pass to mean something. Even if the result you’re looking for is a failure in the system under test (SUT), you want to see that failure so your test passes. But I’m not talking about what makes a good test. That’s a different topic for a different time.

The topic for today is, instead, what is the purpose of the test. What are level of functionality are you trying to test? Knowing the purpose of the test can help you figure out how to classify it. That can then help you figure out how and when to run it.

First, some basics on test classification. There are many types of tests, but in broad strokes, you can think of them applying at 3 levels, unit, integration, and end-to-end or system. To make things more real, let’s consider a clock application.

Unit tests are tests that validate things at the functional level. The typically live and execute at the function/class level. Do these handful of things work well insolation? Do they do what they say they will do, and handle failure gracefully? They typically take less than a second to set up, run, and tear down. Things at the unit level might include configuration, storage, accessing the host’s notion of date and time.

Integration tests are the tests that validate things at the boundaries of domains, functions, or classes. How does class A work with class B? How does your CRUD layer work with the underlying database? How does your logging/timing/resource management system work with its consumers? These tests might take a couple of seconds, and might require access to some host system, For the clock app, you might test reading and writing configuration with the storage system.

End-to-end or System tests are the tests that validate how the system as a whole works. Does it meet the end user’s expectations, or at least not surprise them greatly when it doesn’t. System tests are the ones that validate that even though a bunch of things failed along the way, the system managed to do the right thing, or at least avoided doing the wrong thing. This is where you’ll test emergent behavior as the different parts of the system interact. It’s often only at the system level that you can test what happens when 4 different things fail in a specific way. Because the system level test is the only one where those 4 different components are working together. These tests can take much longer, and often require the real system, or at least a trusted emulator. For that clock, it might be setting up an alarm and making sure it sounds at the appropriate time.

I’ve mentioned the time it takes for these various test types, but it’s not just time that changes. Cost also changes. Running unit tests is almost free. Just a few CPU cycles and that’s it. System tests, on the other hand, can be very expensive. You need to build the entire system and deploy it. To not just the appropriate hardware, but special hardware that you have extra access to for debugging. That all takes time. Any money. Unless you’re doing manual testing. Which takes even more time and money.

Most tests fit reasonably well into one of these three buckets. If one your tests doesn’t, think about breaking the test up into multiple tests to that it does. Once you know which bucket to put your tests in you can move on to the next step, figuring out when you should be running it. I’ll cover that in a different post.

On the other hand, if most of your tests don’t, think about your test design. If your test design seems reasonable, but your tests themselves don’t fit into those three buckets, think about the underlying system design. If your system is untestable at the unit level, that’s not a testing problem, that’s a design/architecture problem. Fix that first. Then recognize that you’re practicing Test Driven Development.

And that’s a good thing.

I'm Back

It’s been a while since I’ve published a new entry. Not because I haven’t thought of things, but because I got sidetracked with life and work for a bit, then I got out of the habit of writing. Which is a great topic to write about. So here I go. Talking about writing. And habits. And personal lessons.

We all have free will. We get to decide what we want to do. Not in a vacuum of course. There is always an impact to our choices. You need to balance the costs and benefits of a choice. The better visibility you have into those costs and benefits, the better decision you can make. Just remember that making a good decision is not the same as having a good outcome of the decision.

In my case, back in the middle of last year I got busy. Busy at work and busy outside work. As people familiar with Spoon Theory know, when you run out of spoons you need to stop, so you need to use your spoons thoughtfully. Looking at the things I needed to do, the things I wanted to do, and the things I could do, I decided to stop working on this blog.

And in retrospect, it was a good decision. The things that needed to be done got done. I was able to do the things I wanted to do that were most important to me, and I was able to put enough effort into them to do them well. I’m happy with the choice I made, and in the same situation, I’ll go through the same process.

However, in retrospect, one thing I missed in my decision was how much momentum and habit play into things. One of the reasons I was writing so much was that I was I the habit of writing. I had some momentum, and that kept me going. When I stopped, I got out of the habit and lost the momentum. Even worse, I got in the habit of saying “I’ll get back to it soon”. And that’s a dangerous habit to have.

What I should have done was extend my decision with some exit criteria. That would have helped me not get into the habit of not writing. Instead of realizing it’s been 9 months since I posted a new entry, I would have had both reminders and a reason to get back to it. Because I do like writing. And sharing. And hopefully others are getting something out of it as well. So here we are. I’m writing blog posts again, and working on building back that habit.

And to bring this back to helping you, my Friendgineers, it’s something that we need to remember as software developers. When we write, whether it’s emails, docs, blog posts, or code, we have habits. Generally, our habits help us by keeping us from having to decide every little detail. One space or two between sentences? (One) Oxford comma or not (Yes) Indent or blank line between paragraphs (Blank line). Those habits are useful.

But sometimes, when we make a decision, like deciding to move quickly to get something working right now, that should have exit criteria but doesn’t, we end up without an important habit, or possibly worse, with a new habit, that gets in our way later. Like the habit of not thinking about forward or backward compatibility, not worrying about separation of concerns, or not writing unit tests. Or maybe hard-coding configurations, or choices? Sometimes you do those things for speed, or expediency, but those are not things you want to make a habit of.

So when you do make those decisions, know your exit criteria, and follow them. If you don’t have them, create them. And above all, be careful what habits your pick up. Or lose.

On Busyness

As I’ve mentioned previously, Winnie the Pooh makes a great coach for folks interested in extreme programming. Things are the way they are and we have to live with what is. We can learn from it. We can change it. But we have to deal with the reality of what is.

There’s another part of the Tao Of Pooh we can learn from. It’s the Bisy Backson.

In the book, Rabbit is looking for Christopher Robin, but instead of finding him, Rabbit finds the note above. He can’t figure out exactly what it means and becomes a “Bisy Backson” trying to find Christopher Robin.

In the story, the Bisy Backson has to always be moving. Always doing something, going somewhere, full of sound and fury, signifying nothing. Just to prove, to themselves and others, that they’re important. Because if you’re doing something important, you must be important.

Unfortunately, that’s one of those things we all know that just ain’t so. And ain’t so on multiple levels. First, and most straightforward, there’s no transitive relationship between the importance of the work and the importance of who’s doing it. The work is important, and getting it done is important, but in most cases, it doesn’t matter who does it. A better way to look at this is through the value lens. Not “I am doing important things so I am important”, but “I am valuable because I am doing things that add value.” A subtle, but important, difference.

Second, and probably the most insidious thing, is that the Bisy Backson is a great example of the difference between outputs and outcomes. They do lots of things and there’s lots of activity, but not a lot of results. And from the outside it looks like progress. That’s the sound and fury part. To make matters worse, that appearance of progress is often incentivized by the systems we work in. this one is hard to manage because it requires self-awareness. Again, the value lens is a good way to combat this. Is what you’re doing high value or not? It doesn’t matter if it’s high output if the value is low.

Third, and hardest to see, is the opportunity cost of being busy. I’ve talked about the importance of slack time, and this is still a great explanation of how busyness and parallelization can work against reducing overall time. The Bisy Backson doesn’t see this. They’re too busy doing things to see that doing less might be faster. And it’s certainly faster than doing something that’s just going to sit in unfinished inventory for a while, or worse, doing the wrong thing because we don’t know what the right thing is yet. The value lens helps here, as it usually does, but it’s not enough. One of the things that traps the Bisy Backson is the local maximum (or minimum) problem. If you don’t take the time to look at the bigger picture the Bisy Backson will quickly find themselves on a peak looking across the valley at the higher peak they should have been moving towards. The antidote here is to step back and look at the bigger picture and understand what it is you’re really trying to do.

On a personal note, there’s another kind of time when I deal with the Bisy Backson inside myself. That’s when something significant enough happens that I need to take time to process it, but I’m not ready to process it headfirst in real time. At times like that I’ll often choose to be the Bisy Backson to engage the high-order processing nodes in my head and let the issue rattle around and clarify itself. That’s where I’ve been the past week. Someone at work passed away unexpectedly. Someone I’ve worked closely with for over 3 years. There are lots of little reminders of the loss, and each one is a distraction. I’ve been using my inner Bisy Backson to give me the time and space to work through it at my own pace.

So while busyness for its own sake might not be the best thing, busyness as a tool can be useful. The hard part is knowing which situation you’re in and making the appropriate choice.

That's The Way It Is

I’ve said before that It Depends is just E_INSUFFICIENT_CONTEXT written so humans can understand it. There’s another common phrase that often hides a much deeper meaning.

The thing about that sentence is how passive and accepting it is. Particularly in the word just¹. Without just it’s a description of the current state. Adding just adds another whole dimension. It changes the sentence for a description of what is to a comment on what is.

And implicit in that comment is context. The context that says not only are things the way they are, but that you’re powerless to do anything about it. I assert that that last part is untrue.

There may be limits on how much you can do, but it’s not nothing. At the very least, if you know that things are that way, you can expect it. And plan for it. Since I’m a software developer I’ll use a car analogy. Say you’re on a road trip and a road you want to use is closed.

You can drive right up to the sign, then stop and wait for someone to open the road, tell you to turn around and go home, or provide a detour. Or, depending on when you find out, you can plan a different route, decide not to go or to go somewhere else instead, or maybe decide a phone call gets you enough of what you want, and do that instead.

The difference is agency. If that’s just the way it is, you have no agency. On the other hand, if that’s the way it is, you have some control over your destiny. You can do something.

Coming back to software development, the same thing applies. There are events that happen that are outside your control. You do have to accept them. Requirements change. Hardware fails. You get bad input. What you do about it is up to you.

Depending on how much control you have, what you do is different. Sometimes you have enough control to prevent the problem. Or at least prevent the problem from impacting you. Ensure there are redundant systems to mitigate hardware issues. Sanitize your inputs when you get them, and if possible, where they are generated. Knowing that requirements change, leave some slack in the schedule. You’ll still run out of time (Hofstadter’s Law), but it won’t be as bad as it might have been.

Or maybe all you can do is add a bit of resilience to the system. Knowing that your inputs are unreliable, even after doing some sanitizing, reject them. Instead of crashing or passing on the problem to someone else, stop what you’re doing and return some kind of error to someone who can do something about it. If you can’t do that, at least log enough information so that you know what happened. And automate the recovery process. Or if you can’t do that, script it. There have been many times where I wasn’t in a position to prevent a problem from happening, but once I knew it could happen, I can’t think of a single time where there was nothing I could do to make things easier to diagnose and/or recover from the situation.

What makes it possible is the mindset change that comes from dropping the just. From changing a comment that makes you powerless to a statement of reality that you can do something about.

Come to think of it, that’s good advice not just for software development, but for life in general.