On 31 August 2026 two changes were reported merged, and we believed both were live. Both times, the machines were still running the old configuration. Nothing in this estate applies a merge. A machine changes only when a person runs the code against it.
A merge changes what the repository says. That is all it does. Routing every change through a request that someone reviews is change management, and it is the part we had right: an unbroken record of what changed, when, and why. But the repository is a declaration of what the machines should be, and the machines do not read it. Something has to carry each change from the declaration to the hardware and then confirm it arrived, and at the time nothing even reported whether the two agreed. The planning tool claimed one machine needed a change on every run, forever, so a real difference looked exactly like the permanent false one. That signal was repaired on 7 September.
Declared, merged, detected. None of the three is the same as done.
It kept happening
Once we started looking for the gap between declared and applied, we found it repeatedly, and once in the other direction.
-
The old address, still answering
The monitoring machine had moved that morning, and the declaration held its new address. The estate’s name service was still handing out the old one. Declared, not applied, the same day, and found by accident.
Declared · not served -
A guard on paper
The machine holding the certificate authority and every secret declared a guard against accidental deletion. The hypervisor reported the guard off. The declaration had simply never been applied, and the backlog row that claimed it was protected had measured a different machine. Applied that day and verified on the box.
Declared on · live off -
The fix that had not run
The change that made the public proxy restart itself after a failure was merged, and the machine still would not restart. Nothing had run against it. The evidence cited that it was still broken had itself been read from the machine next door. Applied the same day; the task reporting a change was the proof it had never been there.
Merged · never ran -
A value applied once
A certificate fingerprint was declared correctly, diffed cleanly and reviewed. The code applied it only when the storage was first created, never again, so when the certificate changed, the documented fix of updating the value and re-running changed nothing. The code now converges it, proven by setting a junk value on one machine and watching one run put it back.
Correct · unappliable -
The other direction
In one backlog session, six of the rows closed were closed by measurement rather than work. The fix had already shipped, and nobody had written it back. Twice that sent a session after work that did not exist.
Applied · not recorded
The record drifts both ways. A declaration can run ahead of the machines, and the machines can run ahead of the record. Either way, somebody plans on top of a claim that is not true.
Detection is not delivery
On 8 September a nightly backup failed. An existing alert rule matched it correctly, over a log path that was working. The alert then sat unread for eight hours, and was found by accident while someone looked at the alerting system for an unrelated reason. Every component had worked. Nothing had arrived.
Hours earlier, that same session had added a new detector to a system whose backlog already recorded, that day, that its alerts were not being read. Nobody decided to do that. Adding detection feels like progress, and repairing the last step, the one where a person acts, does not.
A loop is open until the signal has reached someone who acted on it. Detection that nobody reads is recorded as coverage, which is worse than nothing.
The check that closes the gap
On 14 September a declared-state check went live. Every hour it runs the code for four parts of the estate in a mode that changes nothing and reports whether anything would change. Its first run found real drift, on the router and the VPN box and nowhere else: two configuration files written by an older version of the tooling. The alert fired, the owning code was run against each machine in turn, one at a time, and by 20:24 the alert had cleared.
That was the whole path: found by nobody looking, fixed by the code that owns it, and confirmed clear. Not a staged demonstration of half of it.
How this site goes live
This page follows the same rule. A merge publishes nothing. Each piece on this site was merged, then applied by running the web server’s code from the estate’s build machine, then checked from outside by fetching every changed file and comparing it byte for byte with what the source renders. Only then is it live.
What it cost
- Two changes believed live that were not, on the same day.
- A deletion guard missing on the machine holding every secret, while the declaration and a backlog row both said it was there.
- A proxy fix merged and not running, with the evidence about it read from the wrong machine.
- A failure alert unread for eight hours, in a session that was busy adding detection.
- Sessions sent after work that was already done, because nothing wrote the fix back.
The rule it bought
Merged is not applied, applied is not verified, and detected is not delivered. Count a change done when it has been run against the machine and the machine’s live state has been read back. Count a loop closed when the signal has reached someone who acted on it. And build the check that compares declaration to reality on a schedule, so drift is found by nobody looking.
Where we might be wrong
If a loop only counts as closed once a signal has provably reached a person who acted, then almost every loop is permanently open, and the word stops telling a healthy channel from a dead one. There is also no obvious way to measure the last step: an alert that arrives and is rightly judged “nothing to do” looks identical to one that was never read.
What would settle it: whether anyone can name a loop here that is closed by this standard. If none can, the rule is unfalsifiable and should be weakened to something checkable, such as requiring every detector to name who receives it and how that was tested.
None of this is new
The third of W. Edwards Deming’s fourteen points for management is short: “Cease dependence on inspection to achieve quality”. The rest of the point says what to do instead: “Eliminate the need for inspection on a mass basis by building quality into the product in the first place.”
Deming’s target was a factory that made things and then inspected them. Ours is an estate that declares things and then detects drift. Detection is our inspection, and it has the same limit: it finds what went wrong without making anything go right. The part that builds quality in is the step people skip, which is running the change against the machine and reading the result back, every time, as part of the change.