Deploying vs. Releasing

Today’s thought is courtesy of the inimitable @mipsytipsy (Charity Majors).

Deploys and releases are two different things:

DEPLOY – building, testing, and rolling out new code changes; hopefully small, incremental ones, very often

RELEASE – changing user experience in some meaningful way (not just minor bug fixes)

Sounds simple, no? In reality we confuse the two all the time. Or at least conflate them. Every deployment changes how things behave. And the only way to change it is to deploy again. This might not be a big deal if your change/build/validate/deploy cycle is on the order of minutes, but when it’s on the order of hours (or days or weeks) that’s a real problem. You become afraid to deploy, so the cycle time gets longer, more things end up in each deploy, and things get even longer.

It’s the exact opposite of a virtuous cycle. It’s a death spiral that ends up with huge deployment/releases that don’t do what you want, don’t let you respond to feedback, slow down the development cycle, and upset users. I’m pretty sure that’s not how we want things to go. I know I’d much rather be able to release a small change to my users and have them see a small change that I can iterate on. And undo if they don’t like it. Or it has some unintended side effect. Something I can be comfortable releasing now, whenever now is, knowing that if there’s a problem and quickly undo it.

That gives me confidence to try things. Things that I expect make things better for users, but I need more feedback on. Things that a focus group liked but might not be broadly applicable. Things that change internal data flow but have different operational characteristics than how things are now. Or any other change, really. The easier it is to go back through that two-way door, the more doors I can go through.

Of course, making that happen isn’t easy. You need a rock-solid build/validate/deploy system, which is not a simple thing to build. You need a robust system to distribute and use feature flags (or the equivalent). Also, a non-trivial solution.

You also need a way to clean up after yourself. You can’t leave everything behind a feature flag forever. If nothing else, the combinatorial explosion of possibilities will make it impossible to validate even a majority of the possible configurations. Some of them will be contradictory. Some will just break. The code gets too ugly and hard to understand. You end up with nested configs and then you can’t turn one thing on/off by itself, which defeats the purpose of doing this in the first place.

Now not every change should have deployment separated from release. It depends. Sometimes there are underlying changes that must happen at the same time as the code change. Sometimes the change is so pervasive that making it optional doesn’t make sense. But those kinds of things are much rarer than you think.

So instead of just assuming Deploy == Release, think about what it would take to make deployment and release two entirely separate actions.

Let It Flow

Like many things, I’ve talked about Flow and WIP before. The idea that what you want to optimize for is getting things done, not doing things. That’s a pretty subtle difference, but it’s an important one.

Or as it was said in Principle of Product Development Flow,

In product development, our problem is virtually never motionless engineers. It is almost always motionless work products.

– Donald Reinertsen

So much of how we work is designed to keep us busy. Blocked on waiting for someone else to do something? Start another task. Waiting for some tests to run? Start refactoring your code. Whatever you do, don’t just do nothing.

Now to be clear, I’m NOT recommending you sit around and do nothing if you can’t work directly on whatever it is you’re doing. That is not going to help.

But instead of doing nothing or starting the next task, maybe help the person you’re waiting for. They might get done sooner, which helps them get back to what they were supposed to be doing. Even if it makes things take a bit longer, the next time you’re in that situation you won’t need to bother them. You won’t be blocked, and they won’t be interrupted. Better for everyone.

Or another typical case. You need to run an integration test. It takes 20 minutes, so as soon as you start it, you go do something else. It takes you 15 minutes to context switch and get back up to speed. Just as you start getting somewhere, the test finishes. But you don’t notice because you’re busy. 10 minutes later you notice and go back to the first task. After another 15-minute context switch. You’re back to where you were before the test started. Think about it. You just spent 40+ minutes (2 context switches and a bit of time actually working) for 10 minutes of progress. Do that 3 or 4 times a day and you’ve wasted way more time than you’ve been productive.

Another quote from the book is

Since high capacity utilization simultaneously raises efficiency and increases delay cost, we need to look at the combined impact of these two factors. We can only do so if we express both factors in the same unit of measure, life-cycle profits. If we do this, we will always conclude that operating a product development process near full utilization is an economic disaster.

I know there are a lot of management buzzwords in there, and SAFe has picked up on this quote in particular, but that doesn’t mean it’s wrong. Like most things engineering, it’s a trade-off, and the answer always starts with “It depends …”. And with all trade-offs, the way to make it is to understand what you’re trading off against something else, and what your priorities are. It comes back to are you trying to optimize for doing things or for getting things done?

Here are some more interesting quotes from the book. There’s a lot to think about there.

K. I. S. S.

This is something I keep coming back to. Whether it’s talking about The Unix Way or the difference between complicated and complex, we seem to like complexity.

Developers are drawn to complexity like moths to a flame, often with the same outcome

– Neal Ford

It’s not surprising. It’s in the nature of the problems we’re trying to solve. We’re often dealing with large problems in real-world situations. People are involved, and people are complex. While they may be somewhat predictable in an aggregate sense, even Hari Seldon’s psychohistory didn’t (and couldn’t) predict the Mule. Or as Ian Malcom said, Life finds a way. Complexity is all around us, and we often feel like the only way to tame it is to build something even more complex.

But what it it’s not? What if the best way to deal with complexity is to make things simpler? To build things that are in and of themselves very simple. They do one thing and do it well. The do it in response to an input. You can reason about them. You can make predictions about them. They don’t surprise you. And when the fail, they fail in very specific, predicable, handleable ways. Which means if it does something unexpected, it’s easy to figure out why. Then you can figure out how keep it from doing that. And make it even more predictable.

If you build something that way, something that has very narrow inputs and only a few outputs, the space it operates in, its operational domain, is very small. The smaller the operational domain, the less complex the behavior. And if one part is less complex, the rest of the system can be less complex. The more things you can make less complex the more you can make other parts less complex. It’s a virtuous cycle.

I can hear you saying that making things less complex is great and all, but the problems we’re looking at are complex, and we need to deal with that complexity. That’s true. Things are complex. We need to deal with them. The thing is, we need to deal with them at the system level, not the component level. We all want to build the big, new, shiny, complex, solution to all the world’s problems in one fell swoop, but that’s unlikely to be the best answer.

You can combine all those simple components in complicated ways. Consider the computer I’m using to create this entry. At its core, it’s a collection of gates. 1’s and 0’s. There are lots of them, and they’re connected in very complicated ways. And I can do almost anything with them. But complication is something we can deal with. Complicated things are knowable. And that’s the key.

We can almost always handle complex behavior with a complicated arrangement of simple, straightforward steps. It’s only for the rest of the cases that you need to build a complex solution.

So is your complex problem one that can be solved with a complicated arrangement of simple things, or do you need a complex solution. The answer to that question is “It Depends”.

Adding Estimates

I’ve talked about estimation many times before. I’ve talked about Kent Beck, and I’ve talked about Hillel Wayne. Today I’m going to talk about all three of them at once. I’m not sure how directly this applies to development, but I’m pretty sure there’s something there. Maybe I’ll have it by the end of the post.

Beck recently wrote about private estimates and public progress. What it basically came down to was that the entire team gets together on Monday and decides what they think they can get done that week, given all they know of the situation, the priorities, and the goals. Then, at the end of the week, the team gets together again, looks at what was done, then answers the question “How did we do?”.¹ It’s a simple idea. He then goes on to give a list of pros you get from doing that and some things to avoid.

You get limits. You know what you’re NOT doing. You know what done looks like. You get focus and alignment.

On the other hand, you can’t add weeks. Or compare them. Or compare teams. It just doesn’t work.

But it also got me to thinking about something Hillel said. You can’t add times or temperatures and get something meaningful, but you can subtract them and average them. Huh? On the surface, that doesn’t make any sense. I mean it’s true, and it feels right, but huh? Subtraction is just the inverse of addition. Why can you do one and not the other? Averaging is adding things up then dividing. So why can you add and divide, but not just add?

As he explains, the reason it works is that what you’re really doing is not staying in the domain of time or temperature. What you really end up doing is working in the domain of deltas in time or temperature. And that works because you can add and subtract deltas. When you add two times the result isn’t a time. I’m not sure what it is, but it’s not a point in time. When you subtract two times you get the difference between them. The delta. And I know what that is. It’s a time period. That’s a real thing.

Taking the average of a group of times (or temperatures) is really taking the delta between each time and a common arbitrary time point, then adding the deltas, dividing that time period by the number of times you’re averaging, then adding that time period back to the same arbitrary time point. All of those are valid operations, and you end up with a single point in time that we all agree is the average of that collection of time points.

Now that I’m getting towards the end, here’s what I think. Estimates are kind of like times and temperatures in that they’re based on a known starting point. If you could somehow compare them all with the same starting condition then, just like time or temperature, you could find the delta and do your math in that space. Unfortunately, the starting point for an estimate isn’t an estimate, or a point in time, or anything else easily measurable. Instead, it’s the sum total of what you know about a situation. How things are, how you want them to be, the environment around you, and the expected interruptions among other things.

When you make an estimate for this week you know everything you know about the current state. How correct you are is a different question, but you know all you can know right now. If you could know more then you would. However, the estimate you want to make for next week relies on having that same level of knowledge. You might have that knowledge now, but when you get to the beginning of next week, when you should be making that estimate, the current state will be different. How different? I don’t know. You don’t know.

For next week, it will probably be something like what you estimate for this week, but not exactly. And over time the difference between you expectation and reality will grow, to the point where you have no idea what the starting condition is.

Which mean that you can’t pick a starting condition to subtract from all of the estimates, so you can’t generate a relative estimate delta. And if you can’t turn your estimate into a delta then there’s nothing you can add, or even compare.²

The same thing applies to comparing teams. Or doesn’t apply. Because the teams are different there can’t be any shared arbitrary point to use to create a delta from. And again, without that delta, there’s nothing to compare.

And that’s how Kent Beck, Hillel Wayne, and Estimates come together to explain why you can’t just add up a bunch of weekly estimates and come up with 12 month plan.

You Broke What?

They say with motorcycles there are riders that have laid their bike down at least once, and there are riders that will lay there bike down in the future. That there are no other kinds of riders. I’m not a motorcycle rider, so I can’t comment on the accuracy of that statement or not.

I am, however, a software developer, and I can say that there are 10 kinds of developers. Those that understand binary and those that don’t. And there are 2 other kinds of developers. Those that have broken production software at least once, and those that will.

There are two important things to take away from that. First, you will break production at some point. Don’t let that paralyze you. Second, don’t just YOLO it over the wall and walk away. Pay attention to what happens. Make sure things are doing what you expect, and not doing what you don’t expect. And be prepared to fix things.

Actually, there are a lot of things you should take from that besides those two important things. And the biggest of those revolve around 2 area. Preventing issues and recovering from issues.

You want to do everything you can to prevent issues from happening. The things you’ve seen happen. The things you’ve heard about happening. The things you can think of going wrong. The things others can think of going wrong.

Things like running unit and acceptance tests first. If possible, shadowing production. Then sending a percentage of production work to the new version. And things like making sure the sequencing is in the right order. You add things before you reference them. You stop referencing things before you delete them. Making sure everything is done before flipping to new versions. Those sorts of things. And don’t forget Hyrum’s Law. Somewhere, someone is using your system in ways you didn’t expect, and probably relying on something you didn’t realize was happening. When you find it you’ll need to deal with it.

Bottom line? You don’t want to keep getting bitten by the same kinds of problems when you can find and prevent them.

The other thing to take away is that you need to be prepared to deal with problems. The simplest one to be prepared for is the roll-back. Something bad happened? Putting it back the way it was should be a fast, one-click, operation. If you can’t do that, work towards it. Because it will happen. And don’t forget to periodically test the roll-back. There are much better times than when you’re under pressure to find out that you have a write-only backup.

And after the fact, don’t forget the blameless review. You might call it a post-mortem, an incident review, lessons learned, or something else entirely. Amazon calls them COE’s. Some parts of Microsoft call them Root Cause Analysis. Regardless of what you call them, you should do them. And pay attention to the results. Check out what the Pragmatic Engineer has to say about them.

How careful should you be? It Depends. It depends on how bad things can go and how fast you can recover. So think about what you’re doing. Think about how you can recover. Then make the change. And eventually, you’ll join the club of those who have broken production. I look forward to welcoming you.

The Dude Abides

The Big Lebowski is a cult classic from the late 90s. It’s all about Jeff “The Dude” Lebowski and a series of events and (mis)adventures that happen to and around him. There are many quotable lines, but one of the most quoted, one that shows up on posters, shirts, memes everywhere, and is the last line The Dude says in the movie, is “The Dude abides.” It’s a paraphrase of a line in Ecclesiastes

One generation goeth, and another generation cometh; but the earth abideth for ever.

Things change. Generations come and go. But the Earth abides. It doesn’t go away. The same with The Dude. Despite everything that happens in the movie, The Dude goes on as he is. Accepting life. Calmly enjoying it. Going bowling and drinking white russians.

Which brings me to today’s topic, Dude’s Law.

Value = Why / How

Dude’s law was coined by David Hussman, and he explains it in this short video. Dude’s law is simple. The value of doing something is proportional to why you’re doing it, and inversely proportional to how you’re doing it. For any give why/how pair, if you double the why you get twice the value. If, on the other hand, you double the how, you get half the value.

That’s pretty non-denominational, so let’s try to be a little more specific. Why measures the benefit. The thing you want and get more of. It’s based on the reason you’re doing something. It helps you to quantify the good things that happen if you do whatever it is. How measures the cost of doing it. That includes time, effort, money, and any other resource you put into doing the thing. Or talking about doing the thing.

It’s still pretty non-denominational, but at least now, value is benefit / cost. That’s a fairly common definition, and something that’s easy to get behind.

The important thing to remember here is that value is inversely proportional to the how, and the how in Hussman’s explanation includes time spent discussing how. If you spend too much time talking and thinking about how and don’t improve the how by enough, you’ve actually reduced value. An obvious example of this is XKCD’s Is It Worth The Time. Automation can help, but if you spend more time automating that the automation will save you then you haven’t added value.

It’s another local vs global maximization tradeoff. Consider making a technology choice for the next feature. You’ve got a system that was built using Go. There’s a cool new feature in Rust that you could use that would make the task easier. But the team doesn’t know Rust and the development environment doesn’t support Rust. The higher the how, the less value you create. So without changing the why you probably shouldn’t choose Rust in this case.

Unless… you also change the why. If there’s a good reason to move the entire system to Rust things are different. If your why goes up at the same rate as the how, then the value is not reduced. In that case, the higher how is justified by the higher why.

So yet again, we come back to it depends. Which is just another way of expressing Dude’s Law. Value is benefit / cost. That doesn’t change. It’s up to you to define why and how.

Skill Vs. Experience

Skill and experience can have a lot in common. If you have a lot of experience doing something, you can become very skilled at doing it. Alternatively, you can be very skilled at doing something, and that can take the place of spending time gaining experience.

Consider this video of a technically skilled mountaineer and a monk walking up a mountain.

The person with the camera knows the technical way to get up a mountainside. The monk on the other hand, just knows how to walk up the side of a mountain. End result, they both end up in the same place, but the path they take is very different.

Another way to say it is:

In theory, there’s no difference between theory and practice. In practice, there is.

Or maybe you heard it in an intro physics class, where it when something like this:

Everything I’m going to teach you in this class is wrong. Or more accurately, not exactly correct. It’s good enough for most things you do, but it’s not exact. Newton’s laws of motion are mostly correct, and the equations of motion are close enough for day to day things, but there’s a lot more depth to understand.

Knowing the rules is a skill. Knowing when you can and can’t apply them is experience.

And finally, lots of people know Arthur Clarke had at least 3 laws because they know his 3rd law, which is “Any sufficiently advanced technology is indistinguishable from magic”. But many people don’t know what his 1st law is.

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

That elderly scientist has enough skill and knowledge in a field to know what’s possible even if they don’t know how to do it. On the other hand, their lack of experience can sometimes blind them to possibilities,

All of those are ways of pointing out that just because you have some skill and training, you don’t know everything. Skills can get you pretty far, and in many (most?) cases far enough. But you need experience as well.

I’m a decent rough carpenter. I can frame a wall or hang a door. They’ll be plumb, square, and level. I have the skill to do it. But it takes me a while. And very often I get part way through and realize it would have been easier if I had done something else first. That’s because I’m not experienced enough. I don’t do finish carpentry or build furniture. That requires a level of experience I just don’t have.

The same applies in development. In software development (and many other areas) you want to be efficient. That often means only building what you need. When you find out you need more, you refactor and add what you need. That’s skill based. It works. Every time. And it’s way more efficient than building some elaborate thing you don’t need, then figuring out a way to shoehorn it in to the problem you actually have.

Consider this situation. You’re building a tool to transform some data. You know the input file name, so you just hard code it into the tool. Great. It works. Then you find out that need to know the name in a couple of places in the code, so make it a defined constant. Again, great. Then you find out that there are actually different files that you’ll need to read in different situations. So you make it an input parameter. That’s using your skills. But if you’ve got enough experience you realize that you’re going to need to handle multiple file names from the beginning and you build it that way up front. With the starting name as the default. That’s even more efficient and that’s what experience gets you.

Skill might also tell you that you could build the tool to handle an input file that is a list of files to process. And that each of those input files will need a different output name, so you make the input file contain that. And each of those input files might need different permissions to be read, so you figure out how to get the right secrets to read the file.

What experience can also tell you is while you could do those things, you shouldn’t. Those kinds of problems belong to someone (or something else). It’s experience that tells you to use The Unix Way. Or at least you shouldn’t do them until there’s a demonstrated need for them.

On the other hand, if all your experience is in doing the exact same thing, the exact same way, you end up blocked. You don’t have lots of experience. Instead, you have many instances of the same experience. That’s not true experience. And it’s probably not true skill either. Because true skill is made up of lots of sub-skills. By themselves, the sub-skills aren’t nearly as useful.

So next time you have an opportunity to use your skills to increase your experience, or to use your experience to get some new skills, take it. The more you have of both, the better you’ll do.

Time Periods

From Star Trek to Hofstadter’s Law, estimates are always a conundrum. There are lots of articles about it on the web, but here’s Ron Jeffries about it and it’s a really good place to start.

To me, the problem is not the estimates themselves, or the idea that you’re trying to measure something that is neither precisely or accurately measurable, but in what happens to after you make the estimate. Or to put it more bluntly, the problem is that very often inaccurate, imprecise estimates are taken as promises. When that happens, everyone is going to end up with some level of unhappiness.

If you’ve been reading these entries for any length of time, it’s probably obvious that I think it’s important to add value as soon as possible. And that you need to add the most value you can. The question is, how do you know what the most valuable thing to do is.

In fact, it’s not just the most valuable, it’s adding the most value, for some set of resources, over a certain amount of time, and knowing that for a given set of resources, every choice to do something is also a choice to NOT do a bunch of other things. That’s a good thing. Focus is good, and tends to get you to value sooner.

However, to maximize, or even go in the direction of maximization, you need to compare resources spent on all of the possible options. And since you need to do it before you’ve done most of the possible things, you can’t know how many resources it will take. Instead, you have to estimate it. Then you can do comparisons and make choices.

Sometimes, there really are dates you need to hit. My go-to example for that is delivering games for the holidays. Even today, with online games, internet delivery, and mobile gaming, in the US, if your product isn’t available by Thanksgiving, you’re going to lose out on a lot of sales. You NEED to work within that constraint. You can go a long way by sequencing work and doing the things that need to be done first, first, and leaving the rest for later.

But what if you’re making tradeoffs? Thing A is critical, things B, C, and D are nice to have and individually that’s the importance order. But together, C and D are more important than B and C. Should you do C and D first then? Only if you believe that you can get them both done. And to know that, you need an estimate.

Which brings us to my recommendation(s). Never give a precise time estimate. Instead, the precision of your estimate should be a function of the time period estimated. Something like no more accurate than ½ the time period. If you think the right unit of time for your estimate is days, the accuracy of your estimate should be +/- ½ day. If your estimate is 3 years, then it’s 3 years, +/- 6 months. And, reserve the right to change the estimate AND the precision as you learn more about the problem.

That’s not what the #noestimates zealots want, but it is as little estimation as you can get away with.

Doing It Well And Truly

I’ve always been a reader of science fiction. In particular, the kind of science fiction where knowledge and competence make a difference. Stories like Andy Weir’s The Martian. Or Lucifer’s Hammer. There’s nothing magical or particularly special about the protagonists, they’re just competent. And lucky. And most importantly, they do the thing. They do it well and truly. But most importantly, they do it. The don’t (only) talk about doing it. They don’t (only) talk about how important it is that it get done. They don’t (only) wish someone else do it for them. They don’t (only) complain about who’s not doing it. They just roll up their sleeves and do it.

Most of Lucifer’s Hammer is about how different groups of people prepare for, survive, then move forward after a comet hits the Earth. They don’t complain about it (too much). They don’t wait for others to do something for them. They plan, they execute, they adapt, and they move forward. Lots of folks have read the book, and different people remember different parts of it. But almost everyone remembers Gil, the surfer. His choice on that day was to do what he loved, which was surfing. So he was out on the ocean off the shore of Santa Monica when the comet hit. When that happened, instead of giving up, he did what he did best. He decided to surf the tsunami. The last thing we get in the book is Gil thinking about the story he could tell if he survived.

Sure, Gil was thinking about going out in style at first, but he realized that Doing it well and truly might actually get him out of it alive. In a an extra last chapter that was published in Analog almost 40 years after the book, it turns out, he did survive. Once again we see that doing the thing, instead of just talking about it, makes a difference.

As in art (and life), so in development and career. If you see something you think needs to be done, but no one is doing it, think about it and figure out why. It might be like Chesterton’s Fence. There really is a good reason, it’s just not obvious to you. It might be because of priority. Or, it might be that it just hasn’t been important enough to anyone. That’s where doing the thing becomes important.

If it’s important to you, you have (or can find) the time, and there’s not a good reason not to do it, then do it. You’ll be happier. Chances are others around you will be happier. And people will see you as someone who gets things done. Any of those is a good enough reason. Together they’re a great set of reasons.

There are lots of things that sound like doing the thing, but they’re really things that aren’t doing the thing. They don’t get things done. They don’t make you happier. They don’t make others happier. They make you look like a complainer or a procrastinator instead of a doer. The key thing here is to keep the difference in mind. It might be fun to do things that aren’t doing the thing. It’s probably easier to do those other things than it is to do the thing. You might even get more support that way. However, you won’t get the thing done. And that’s the most important thing to remember. If you’re trying to get a thing done, you have to do the thing.

So the next time you find yourself in a situation where you have a choice between doing the thing and not doing the thing, think about what you want, what’s important, and if it’s your choice. Then, do the thing, and do it well and truly.

A look at 2023

Big changes on the work front. I started the year working at Amazon on their Just Walk Out technology, but then got moved to a new initiative. The work was interesting, and from what I hear, going well, but Amazon and that project weren’t a good fit for me, so after a year at Amazon, I started looking around for something different. What I found wasn’t Amazon, but it wasn’t that different for me. I’m back at Aurora, now working on the Developer Experience team, focusing on software quality, bothg internal and external. I still get to walk to work, and I’m enjoying the challenges, and once again, I’m learning new things all the time.

With those work changes, I wasn’t able to do as much work on the blog as would have liked, only 50 new posts this year. It averages out to one a week, but it was way more bursty than that. My goal for this year is to be not just more productive on the blog, but also more consistent.

From a content standpoint, this year’s articles have been more about the what and the why, and less about they how. Oh sure, I have opinions on how everything should be done, and I’m happy to share them, but the posts have focused on what kinds of things you should be thinking about and doing and why they’re important, not how to do them.

With that said, here are the 10 most popular posts over the last 12 months:

1. Careers Pt 3 – Visualizing Progressions
2. Careers Pt 1 – Levels
3. Effective Software Engineers
4. McDonalds
5. Incident Response
6. Let Me Tell You What I Want
7. High Quality Quality
8. Seeing Like A State
9. Breaker Breaker Rubber Duck
10. 1 > 2 > 0

I’m really happy to see that my career posts are still getting good traction, as are the Eng Competencies. How you look at your career drives what you put into it, and from that flows what you get out of it. And to round things out, 2 more posted this year since #1 and #2 are from last year

• Shallow Hurry
• Any fool can write code the computer understands.

Hopefully you’ve found those, and other, articles interesting and useful. I know I found writing them helpful in clarifying and cementing my own understanding of the topics. Until next year, Happy New Year, and remember to enjoy whatever you’re doing. We only get one shot at this life, so make it worthwhile.