The month the estate was built. Nothing here existed on 1 August: no machines, no code, no backups. By the end of it three hypervisors had been built from bare metal without editing the code, two restores had been proven, and most of the rules this site is about had been paid for.
heldwhat held, or what a failure bought brokewhat broke, or what was found broken
-
The network switch locked us out
A configuration change stranded its own management interface. It cost a factory reset, and the recovery that works — moving a cable to a port that still answers — is written down now.
Factory reset -
The last change went straight to the main branch
From this day on, every change arrived as a merge request: 525 of them by mid-September, with no direct push. It became a rule the server enforces a month later.
Every change reviewed -
A key was retired, and we believed it
Removed from every machine, then from the hosts, and refused at a real login attempt. The sentence written down was true about the places we had checked. The place we had not checked kept it until 4 September. The key that was retired everywhere.
True, and not the whole truth -
The first restore ever performed here
A machine was backed up, restored onto a different host, and booted with its address intact: 65 seconds to back up, 457 seconds to restore. Until that day the backups were presumed good. Restore testing.
Presumed → proven -
The second restore was slow, and that was the find
Incremental backups barely touch the wire, so they had been fast over a link that was not. The restore could not cheat, and pulled at about 900 KB/s. The backups had been hiding the network for weeks.
A link nothing measured -
One address, three machines, and nothing in the logs
Three copies of one machine existed across two hosts, each holding the same public address. Booting a second one took out routing, address translation and name resolution at once, and logged nothing. The rule it bought: before booting anything restored or cloned, list the whole lineage on every host, not just the one you restored from.
Routing, NAT and DNS -
Three hypervisors built from bare metal
Each installed, then re-run with nothing left to change and no edits to the code. On the third, the written procedure needed correcting three times: the code was repeatable, the instructions were not.
Code · 0 editsProcedure · 3 fixes -
Four production machines cold-restored
Backed up, restored onto another host and verified: host keys unchanged, the break-glass password still set, name resolution working. About 93 seconds to restore, about 20 minutes from shutdown to verified. Nobody has since said how fast a recovery must be.
Four for four -
The orientation file was cut for the first time
It had reached about 1,900 lines, and every session pays to read it. The rule that one document owns each fact was written the same day. By 29 August the file was 2,247 lines. A rule nothing checks.
Grew back in three days -
A throwaway copy of the estate, built and torn down three times
The third run landed on identical numbers with zero edits. Getting there filled a ledger of fourteen findings. The second run is the test.
Ledger · 14 findingsRun 3 · 0 edits -
Asked what stood between us and the goal, the agent invented half the answer
An eight-row table of gaps, every row plausible, half of them not real. Reviewed with the operator the same day. Half the rows were invented.
4 of 8 invented -
One machine is backed up, and that is a decision
Everything else here can be rebuilt from code. The Git server holds the change history, which cannot be rebuilt at any price, so it is the one machine with a backup — a pause with a named end, not a policy.
Named, not drifted -
The orientation file was cut again, to 627 lines
The second cut in three days. This time the rule was written a second time as a script that exits non-zero, because a rule that only lives in prose is recalled after the mistake, if at all.
Rule → script -
A reboot run to test something else left the router without its address
Everything reported healthy. The machine came back up and the interface did not, and the signal that should have said so said nothing. It looked healthy.
Healthy and wrong -
The checker had never run, and its first automated run proved it
It was committed without permission to execute, so every run had silently passed by doing nothing — the exact defect the checker exists to catch. A check nobody has watched fail is just another thing that looks healthy.
Never ranWatched it fail -
The Git server was restored, and counted
133 merge requests and 428 notes, equal to the backup’s own rows. We compared against the backup rather than the live server, which had moved on. The test itself could have deleted the real backup history, so the isolation went into the code before anything ran. Restore testing.
Restore proven -
The secrets store was restored into a throwaway copy
The proof was issuing a new certificate from the restored authority, because a certificate can come back while its private key does not, and reading it looks the same either way.
Authority proven -
Two changes were merged and believed live
Both times the machines still ran the old configuration. Merging is not applying, and it took until 14 September to have anything that checks. Merged is not applied.
Twice in one session -
The log shipper was pointed at a port that refused it
It ran, reported healthy, and shipped nothing. The first thing built to watch the estate needed watching itself.
Shipping nothing