The second run is the test

On 4 September 2026 we deleted a working Git server and built it again from nothing.

Nothing was wrong with it. It held every change request the project had ever made — the record of why each part of the estate is the way it is. It had been built from code, restored from backup and checked. By any reasonable reading, it was done.

We rebuilt it because built from code is a claim about the past. The claim that matters is about the future: that the next build works. The only way to test that claim is to make the next build happen.

Creating the machine took 17 seconds. The rebuild then found four defects. Three of them were in code that had already been reviewed and merged.

What repeatable means

We hold to a narrow definition, because the loose ones are where the trouble hides. Repeatable means done more than once, with no edits in between. Not “done once, carefully.” Not “done again after a small fix.” If the second run needed a change, the first run was a draft.

By that bar a first run proves very little, which is uncomfortable, because the first run feels like the achievement. It is where the work was and where the problems were solved. It is also where every problem was solved by hand, in the moment, by someone who knew the answer — and that someone will not be there next time.

The four defects

1. Changing the base image did not rebuild the machine. The provisioning code deliberately ignores the image once a machine exists, so the plan showed a small change in place. Applying it would have left the server on the old image while every file in the repository named the new one. The only path is to destroy the machine explicitly, and the rebuild plan now says so.
2. The server’s address had two owners. The install code wrote it into the configuration file and also handed it to the installer, which rewrites that file from what it is handed. Correcting one changed nothing: the install failed the same way twice. Underneath was an ordering bug. A secure address made the installer demand a certificate that our code only issues in a later step, so the package sat half-installed and every later install failed on it. The address now has one owner.
3. A finding from the day before had been written down and never acted on. The previous day’s note said this step needs another one to run first, and must run from the build machine. The code still did not say so. Run from the wrong machine, it produced no certificate — which is what exposed the second defect. This one was not code at all. The step now checks for itself and refuses, naming the command to run.
4. Nothing could put the server’s encryption secrets back on production. The only code that placed that file was pinned to the throwaway copy of the estate — deliberately, so a rehearsal could never touch the real server. But the same restore is how the server’s history moves into a fresh build, so production always needed that path, and it had never been built. Without the file, a restore brings the encrypted data back unreadable and looks as if it worked.

None of the four was visible on the server the code was written against. That server already had its image, its address, its certificate and its secrets.

Who wrote them

The agent drafted all three pieces of code, in working sessions with the operator, and the operator reviewed and merged them. That is the ordinary shape of the work here: of the 593 changes committed to this repository before 15 September 2026, 555 were drafted in agent sessions, and the operator merges every change.

We say so because the argument needs it. Two of us read each of those changes — one wrote it, one approved it — and both were looking at a server that already worked. Neither reading could see what the code assumed. The rebuild could, because it took the server away.

Where the clean runs came from

No zero-edit run here came first. Most came after a run that needed changes to the code; where the code needed none, the written instructions did. And one machine has not had its clean run yet.

The defect that writing it down created

The CI runner is worth slowing down for, because it shows why reading cannot find this kind of bug.

The code that installs the runner holds its version fixed, so an unattended update cannot change it. Its own instructions said to pin the version after the first install. So when the machine was first built, on 30 August, there was no pin yet and the step that manages it was skipped. Then the pin was recorded.

From that moment the code could not build a fresh machine. The step now tried to release a hold on a package that had never been installed, and failed. But nobody was building a fresh machine. Pinned and built from nothing did not occur together until 4 September, when we deleted the runner and built it again.

The defect was not in the first build. It was created by writing down what the first build had learned — and it stayed invisible until a machine was deleted.

Code that describes a machine, and code that builds one

Every one of these has the same shape. Code written against a machine that already exists quietly describes that machine. It assumes the package is installed, the certificate was issued earlier, the network came up in a particular order. Review cannot see those assumptions, because the reviewer is looking at the same machine the author was.

A rebuild removes the machine. What is left is only what the code actually says.

Replacing a machine rather than repairing it is immutable infrastructure, and we hold to it for the proof rather than as a daily habit: between rebuilds these machines are changed in place, like everyone else’s.

Why this matters more with an agent

An agent starts every session with no memory of the last one. It cannot remember that the runner needs a nudge, or that a certificate has to exist before the web server will start. Whatever is not written down does not exist for it.

That makes Infrastructure as Code (IaC) more than good practice. It is the part of the team’s memory that can be run. A fix made by hand lives in one machine, and the next rebuild erases it. A fix made in the declaration survives everyone forgetting it — the agent included.

The rebuild is how you find out which kind of fix you have. We now ask it of every item on the backlog: does the next build keep this, or erase it?

What it cost

Second runs are not free, and these were run on the production estate. What they found, they found there.

• 3 September. The rebuilt VPN box came up with its tunnel configured and no client able to connect. A firewall rule named a network interface the new machine did not have. The box answered ping from the internet, listed its peer and passed its own check, so it took a measurement to find. It needed a code fix.
• 3–4 September. Three rebuilt machines came back without their monitoring and log shipping, because the written rebuild sequence did not list those steps. Monitored targets went from 24 up to 19. The alert fired from 00:00 UTC on 3 September until 11:30 UTC the next day, and about 130 emails were delivered. We found the gap by re-running steps, not by reading the alerts.
• 5 September. A rebuild of the router left the estate without name resolution for about twenty minutes. Nothing alerted. The first symptom was an unrelated machine failing to update its packages.

The firewall rule, the missing steps and the silent gap in name resolution are each now a check in the code or a step in the written rebuild sequence. The alert is fixed by neither. It did its job and reached us, and we did not act on it for a day and a half. No rebuild finds that.

None of this is new

In 2000, Joel Spolsky put a question second on his twelve-point test for software teams: “Can you make a build in one step?” On good teams, he wrote, “there’s a single script you can run that does a full checkout from scratch, rebuilds every line of code”. And: “If the process takes any more than one step, it is prone to errors.”

Everything above is that question, asked of the machines instead of the code. The test is about code; the machine the script runs on is simply there. Declaring the machines removes that assumption, and a rebuild is the checkout from scratch. What changed is not the practice. It is how much of the system has to survive being built from nothing.