Tag, You're It

Docker and immutability have an interesting relationship. Docker images are immutable. They even have a SHA, and when you pull an image with its associated SHA you always get the same one. Docker tags, on the other hand. are associations between a string (the tag) and an <repository>:<sha> pair. And that association is mutable. Or at least it's mutable according to the spec. That means that you could pull an image today, maybe docker pull debian:stretch-20200607 and get an image with a SHA256 of 554173b67d0976d38093a073f73c848dfc667d3b1b40b6df4adf4110f8a3e4ce. That's great. But tomorrow you could issue the same command and get an image with a different SHA. Because the author can change the tag to be associated with a different image while you weren't looking.

We like to think of docker images as a way to ensure that we always get the same environment. And that's sort of true. For a given image it's always the same. But a <repository>:<tag> can change. If you really want to be sure you need to pull by sha. So the correct command to ensure you always get the same image (assuming it hasn't been deleted) is docker pull debian@sha256:554173b67d0976d38093a073f73c848dfc667d3b1b40b6df4adf4110f8a3e4ce.

Then there's the magic latest tag. latest is especially fun when used with docker run. In this case it's not the latest image pushed. It's not the latest image:tag you ran. It's the latest image without a tag you ran. And unless you've got a very odd build/deploy/run process it's almost certainly not what you want.

If you think that's bad, throw Kubernetes into the mix and it gets weirder. Because Kubernetes acts like the tag is immutable, and will, by default assume that if there's an image with the right tag on the machine it must be the one you want. Yes, you can tell it to always pull, but what happens if the tag changes and then your pod dies? When the pod restarts you're going to get a different version of the image.

And to make matters worse, Kraken, which is very good at supplying images to 100s of nodes at once, has sort of mutable tags. When you create a new tag Kraken correctly writes down the SHA it refers to. And if you update the tag it will remember, in memory, what the new SHA is, but it never writes it down. So if the Kraken node ever restarts, it reads the old SHA from disk and starts serving the old one.

So the moral of the story is to treat your tags as immutable, even if they're not. Don't use latest, especially in production. If you feel the urge to use latest locally during development you're probably going to be OK, but don't push it to a remote docker registry. And don't ever re-use a tag. You, and everyone around you, will be happier.

FYI, here on the infra compute team we like to use <user>_<git branch>_<date>_<time> which is really hard to make collide, so we don't have to worry about mutability. We have that format baked into our bazel rules/makefiles so it just works. Yes, the tags are a bit long, but you  don't type them that often, and unlike a box of chocolates, you always know what you're going to get.