Immutable infrastructure is the practice of never changing a machine once it is built. A fix, an upgrade or a new setting goes into the code and the image the machine starts from, and a new machine is built from them to replace the old one. Kief Morris described the end point in 2013:
An Immutable Server is the logical conclusion of this approach, a server that once deployed, is never modified, merely replaced with a new updated instance.
The shorter version is rebuild, don’t repair. Between a machine that is repaired forever and one that is never touched sits a middle step, the phoenix server, rebuilt from scratch often enough that nothing unrecorded survives on it. In Morris’s terms, “a phoenix server’s complete state is built from a combination of base image + automated configuration management + data”.
Why it matters more with an agent
A machine that has been repaired holds some of its truth outside the code: the fix someone made, the order things happened in, the package that was already there. A person who made those repairs might remember them. An agent starts every session with only the code, so on a repaired machine it is working from a description that is quietly incomplete.
A rebuild deletes everything the code does not say. What comes back is exactly what the next session will be able to read. That makes every rebuild a test of whether the code is the whole memory, and when it is not, the rebuild is where we find out.
None of this is new
In 2012, Martin Fowler wrote that “it is a good idea to virtually burn down your servers at regular intervals.” His reason was drift. Tools that re-apply configuration help, but “re-applying configuration like this can only spot drift in areas that you’ve defined that the tools control.” A rebuild has no such blind spot: “Since phoenixes start from scratch, however, they will pick up any drift from the source configuration.”
The idea’s best-known slogan turns up in Fowler’s post on snowflake servers, in a footnote that does not say whose it was: “Another metaphor I’ve heard for this is that you should treat your servers like cattle and not like pets.” Who said it first is disputed, and we found no source we could check, so we quote it only from there.
What rebuilding bought here
- Every machine built fresh from the code. The campaign that deleted and rebuilt each one finished on 4 September 2026. On the Git server alone it found defects in code already reviewed and merged, none of them visible on the machine the code had been written against.
- Second runs with nothing to fix. On 5 September 2026 two network machines were each rebuilt twice, and both second runs needed zero edits.
- A new image that replaced nothing. Pointing a machine at a new base image did not replace it, because the provisioning code deliberately ignores the image once a machine exists. Replacing a machine here is an explicit destroy, and the rebuild procedure says so.
- Proof of what is not disposable. Morris warned that “If servers are disposable, the data that lives on them often is not.” The Git server’s history cannot be rebuilt from code at any price, so its rebuild is also a restore. And on 4 September 2026 the sign-in service came back from a clean run with its groups and clients in place and zero users: a sign-in service nobody could log into. We had written that down before the rebuild, so it was a step, not an outage.
- A rebuilt machine is a stranger to its neighbours. The same rebuild issued new secrets to the services that sign in through it, and two of them had to be re-run to pick them up.
Where we fall short
We still repair in place, every day. Morris is plain about this: “Once you’ve spun up a server instance from a well-tested base image, you shouldn’t run configuration management tools, since they create opportunities for untested changes to the instance.” Here, every change is a person running the code against machines that are already running, and since 14 September 2026 certificates are renewed weekly on running machines. Rebuilding proves the code can build a machine. Day to day, the machine is converged, not replaced.
Rebuilds come in campaigns, not at regular intervals. A machine is as fresh as its last rebuild, and no schedule sets when the next one runs. Between campaigns, the hourly drift check looks for changes, and it sees only what it reads.
The image nobody tests. In Morris’s version the image is where changes go: “Each time you want a new update you modify the base instance and run it through an automated test harness.” Our machine images are built rarely, by hand, from a written procedure, and the image is the one owner nothing can verify. Every rebuild starts from it.