Every team has lived some version of this night. The deploy is green on staging. The same build, the same commit, the same everything, goes to production and immediately falls over. Someone senior squints at the logs and says the sentence that has ended more standups than any other in this industry: it works on my machine.
It always does. That is the problem. Your machine is a liar with good intentions.
01 — The GapThe gap is the bug
Production and your laptop differ in a hundred small ways, and any one of them can be the whole outage. The runtime is a patch version newer. An environment variable exists in one place and not the other. The database has ten million rows instead of ten. The timezone is UTC instead of wherever you happen to live. The code did not change. The world around the code did, and the world is part of the program whether you invited it or not.
- 01Runtime version. A patch version newer in production than on the laptop that shipped the code.
- 02Environment variables. Exists in one place, missing in the other.
- 03Data volume. Ten million rows in production. Ten on your machine.
- 04Timezone. UTC in production, wherever you happen to live on your laptop.
- 05Config source. Values and shape that drift because nobody wrote them down.
War stories in this trade almost never end with a clever algorithm. They end with a version number nobody wrote down.
02 — The DiffDiff the two worlds
When the five words get said on the outage call, stop arguing and run the diff. It takes two minutes, and it answers the first question before the archaeology starts.
$ cat .nvmrc # the runtime the repo says should run
$ env | grep -i proxy # and every other variable production swears it sets
$ git status # is the tree you tested the tree you shipped?
$ git rev-parse HEAD # the exact commit, in both places
$ date -u # whose clock, whose timezone
03 — The HabitsWhat the survivors do
Teams that stop having this night share a few unglamorous habits.
- 01Pin the runtime. Not "Node 20-ish." The exact version, in a file, in the repo, enforced by the tooling. If a new hire can install a different interpreter than production runs, you do not have a standard. You have a hope.
- 02Commit the lockfile and install from it. Dependencies drift quietly. The lockfile is the difference between "the versions we tested" and "whatever the registry felt like this morning."
- 03Make the dev environment boring. Containers, dev containers, a setup script that a tired person can run at 2 a.m. The goal is not elegance. The goal is that "my machine" stops being a unique snowflake with its own private physics.
- 04Print the versions where you can see them. Runtime version, build hash, and config source, right in the health endpoint and the startup log. When the outage call starts, the first question should be answerable in ten seconds, not after twenty minutes of archaeology.
- 05Treat config as code. Reviewed, versioned, and identical in shape across environments. The values differ. The shape never does.
04 — The Honest EndingThe honest ending
You will still say the five words someday. Everyone does. The goal is to say them about a machine that is documented, pinned, and reproducible, so the sentence becomes the start of a short debugging session instead of the title of a very long night.
The code is the easy part. The machine was always the program too.
