Deploying vs. Releasing

Today’s thought is courtesy of the inimitable @mipsytipsy (Charity Majors).

Deploys and releases are two different things:

DEPLOY – building, testing, and rolling out new code changes; hopefully small, incremental ones, very often

RELEASE – changing user experience in some meaningful way (not just minor bug fixes)

Sounds simple, no? In reality we confuse the two all the time. Or at least conflate them. Every deployment changes how things behave. And the only way to change it is to deploy again. This might not be a big deal if your change/build/validate/deploy cycle is on the order of minutes, but when it’s on the order of hours (or days or weeks) that’s a real problem. You become afraid to deploy, so the cycle time gets longer, more things end up in each deploy, and things get even longer.

It’s the exact opposite of a virtuous cycle. It’s a death spiral that ends up with huge deployment/releases that don’t do what you want, don’t let you respond to feedback, slow down the development cycle, and upset users. I’m pretty sure that’s not how we want things to go. I know I’d much rather be able to release a small change to my users and have them see a small change that I can iterate on. And undo if they don’t like it. Or it has some unintended side effect. Something I can be comfortable releasing now, whenever now is, knowing that if there’s a problem and quickly undo it.

That gives me confidence to try things. Things that I expect make things better for users, but I need more feedback on. Things that a focus group liked but might not be broadly applicable. Things that change internal data flow but have different operational characteristics than how things are now. Or any other change, really. The easier it is to go back through that two-way door, the more doors I can go through.

Of course, making that happen isn’t easy. You need a rock-solid build/validate/deploy system, which is not a simple thing to build. You need a robust system to distribute and use feature flags (or the equivalent). Also, a non-trivial solution.

You also need a way to clean up after yourself. You can’t leave everything behind a feature flag forever. If nothing else, the combinatorial explosion of possibilities will make it impossible to validate even a majority of the possible configurations. Some of them will be contradictory. Some will just break. The code gets too ugly and hard to understand. You end up with nested configs and then you can’t turn one thing on/off by itself, which defeats the purpose of doing this in the first place.

Now not every change should have deployment separated from release. It depends. Sometimes there are underlying changes that must happen at the same time as the code change. Sometimes the change is so pervasive that making it optional doesn’t make sense. But those kinds of things are much rarer than you think.

So instead of just assuming Deploy == Release, think about what it would take to make deployment and release two entirely separate actions.