On 3 September 2026 the estate’s sign-in service ran for 79 minutes and nobody could reach it.
Nothing about the service had failed. It was up, listening, ready to answer. What had died was the web proxy in front of it. After a power event that morning the proxy started before the estate’s name resolution was back, so it could not look up the address of the service it forwards to. Its configuration test failed, and this proxy’s default is not to retry. It stayed down after the network was repaired, and it came back with a single start command and no change to its configuration at all.
We had checked. After the power event we swept every physical machine, and the sweep reported zero failed services. That was true. The failed service was inside a virtual machine, and a check of the physical machines never looks there.
The catalogue as of 5 October 2026. The payload of an entry is the command that would have caught it.
Every signal was true
We keep a catalogue of these. By 5 October 2026 it held 79 entries: things on this estate that reported success while doing nothing.
What makes them expensive is not that a signal lies. In the six below, no signal was false. Each one was true about something other than the thing we needed to know.
-
A setting that is only read at boot
A line in the router’s network configuration was removed as a no-op. It was doing real work. For two days every apply succeeded and every check passed, because the running system still had the name from its previous boot. A reboot, run to test something else, brought the router back without its management address, route or name resolution.
2 days · every check passed -
A pass over nothing
Restoring the Git server from backup, the check that decides whether its encrypted data survived printed 0 failures. It prints exactly that whether the keys are right or there is nothing encrypted to test, and a freshly restored server is very nearly the empty case.
0 failures · proves nothing -
A healthy answer from the wrong page
The new monitoring marked the power monitor’s metrics endpoint healthy from its first run. The endpoint answered with 8,862 bytes of the metrics library’s statistics about itself and not one reading about power. The alert that endpoint existed to feed could never have fired.
Up · no readings -
An alert on a number that does not exist
The obvious alert for a broken log shipper watches its error count. With the shipper pointed at a port that refused it, the count did not read zero. It did not exist, and a rule over a missing number evaluates cleanly and never fires. Every alert built on that machine’s logs goes quiet with it, which looks like good news.
Alert · cannot fire -
Running and unreachable
The sign-in service, above. Active throughout; the proxy in front of it dead since boot; zero failed services on every physical machine.
79 min · 0 failed services -
Every value true, about the wrong machine
The estate moved onto a second, larger backup power unit and the first went to lab equipment. The power monitor kept reading the first. On mains, 100% charged, about 350 W. Every value was correct, and none of them was about the estate.
Service active · wrong unit
True about the wrong layer
A check answers for the layer it runs on. The sweep after the power event asked the physical machines about themselves, and they were fine. The question that mattered was one layer up, inside a machine, and further out still: does a request from outside get an answer?
The router is the same shape turned in time. Its configuration is read at boot. Applying a change checks the running system, which still holds whatever the last boot left. So every check was right about now and silent about the next boot, and those are two different states. A configuration that is only read at boot cannot be validated by applying it. Only a reboot can.
A check that cannot fail
The other shape is quieter. The endpoint answered, so it was up. The error count was absent, so nothing was wrong. The restore check found 0 failures, so the keys were right. Each check produced an answer. None could produce a different one.
The question that exposes them is simple: what would this check say if the thing were broken? If the answer is the same thing it says now, it is not a check. So we now prove a check against a deliberately broken input before trusting it. If it cannot fail there, it tells us nothing anywhere else.
That rule found its own next case. On 14 September we tested the alert we had since built for a failing log shipper. We cut one machine’s shipping for 2 minutes. The shipper’s own health stayed up, its error count stayed absent for 90 seconds, and the alert did not fire. Waiting for logs to stop would not work either: over 24 hours, the quietest machines went 410 minutes without logging anything on their own.
So we stopped watching for a failure and started requiring proof of life. Every shipper now sends a heartbeat every 5 minutes, and each machine the estate expects has its own alert for when that heartbeat stops arriving. We proved it with a real cut, not a reading of the rule: blocked at 18:00, firing at 18:22, and no other machine’s alert moved.
A signal that has to arrive can fail. A number that has to stay low is satisfied by silence.
What it cost
Three of the six cost nothing but a query, because someone asked for the outcome before trusting the status: the empty restore check, the endpoint with no readings, and the error count that did not exist.
The other three were found by accident.
- The sign-in service was unreachable for 79 minutes. It turned up while we were repairing something else, and the only outside symptom was a status code nobody was watching.
- The router came back broken from a reboot we had chosen for another reason. Its address was restored through the hypervisor, without a console. An unplanned reboot would have found the same fault at a moment nobody chose.
- The power monitor reported the wrong unit, and nothing was broken, so nothing could say so. The failure mode was not an outage. It was not being told. A scan run to answer a different question found it.
The rule it bought
Verify the outcome, never the syntax or the status. Not that the configuration applied: that the machine comes back from a reboot. Not that the service is active: that the request gets an answer. For the sign-in service, that means a real request through the proxy, at its public name, reading the status code.
And once a check is learned, it goes into the code, where the next rebuild keeps it.
- The proxy now looks names up per request instead of once at start. Proven on 14 September with name resolution blocked: it started, answered 502, and returned 200 when resolution came back, with no restart.
- The router’s code refuses to run when an interface it is about to configure does not exist, and if it cannot tell which interfaces exist, it fails rather than passing.
- The power monitor compares the serial number read off the device with the one the code declares. A monitor that compares its configuration with a device that agrees with it has checked nothing.
Writing every failure down with the check that would have caught it, and never with who caused it, is a blameless postmortem. The catalogue is what ours look like: no names, no session, just the trap and the command.
Why a catalogue, and not only the fixes? Because a fix protects one place, and the shape comes back somewhere new. An index of every entry is read at the start of each working session, since an agent that does not know a trap exists will never go looking for it. Neither will a person.
None of this is new
In 2016, Google’s Site Reliability Engineering drew the line these entries keep landing on. Black-box monitoring, it says, is “Testing externally visible behavior as a user would see it.” White-box monitoring reads the system’s insides: its logs, its internal statistics. And it warns that “one person’s symptom is another person’s cause.”
Every entry in our catalogue is an inside signal answering honestly. The service is active, the endpoint answered, the count is zero. The advice to check from where the user stands is not ours. What the catalogue adds is the evidence of how often, on one small estate, an honest signal from the inside was about the wrong thing.