Restore testing is taking a backup and actually restoring it, then checking that what came back is what went in. It is the only way to know a backup works. A backup job that reports success proves that data was written somewhere. It does not prove the data can be read back, decrypted, and put to use by the system that needs it.
Our operator puts it as a rule for calling anything production:
We have two rules for declaring that something is in production: 1. Is the code under source control? 2. Is the data being backed up, and have we tested the restore process?
The first rule is Infrastructure as Code. The second is this page. Both are the operator’s, “based on years of experience, but also because of SOX requirements”. And for anyone whose backups run green and have never been restored, the operator’s answer is short: “If it hasn’t been restored, then it is not in production.”
Disaster recovery
Disaster recovery (DR) is the plan for getting back after losing a machine, its data, or the place it runs. Restore testing is how you find out whether that plan works, and it only counts if the test uses the path a disaster would leave you. When we first restored the Git server, both of its inputs came from off the machine, not from the copies sitting beside it: if the server were gone, those copies would be gone with it, “so the easy sources test a path nobody will ever be on.”
Here, restoring is part of building
We do not treat restore as a drill beside the build. In the operator’s words, “it is required to restore when we build each vm from the start”. A machine that holds data nothing else can regenerate is rebuilt by restoring that data. So every rebuild of such a machine is also a restore test, and the second run of the build tests the restore a second time.
None of this is new
The compliance world wrote the same rule down as a control. The US government’s catalog of security controls has one for testing backups, and its guidance begins: “Organizations need assurance that backup information can be reliably retrieved.” It even describes the method we used, retrieving a sample and “comparing the information to the same information at the primary processing site can provide such assurance.”
What our restores found
- The Git server, 30 August 2026. Restored onto a throwaway machine: 133 merge requests and 428 notes, equal to the backup’s own rows. We compared against the backup, not against the live server, which had moved on to 135 and 435 by then and would have made a good restore look like data loss. An encrypted value was confirmed present and decrypting, because a restore can bring encrypted data back unreadable and still look as if it worked. And the test itself was a risk: the throwaway machine could have inherited the settings that prune the real backups, so the isolation was built into the code before anything ran.
- The secrets store, 31 August 2026. Restored into a throwaway copy. The proof was issuing a new certificate from the restored authority, because a certificate can come back while its private key does not, and reading the certificate looks the same either way. It was also the only test of the key that decrypts the backup, which nothing else ever exercises. And it found a trap: after a restore, the error on unlocking reads as if the unlock keys are wrong. In a real outage that misreading is worse than the fault.
- Both again, 4 September 2026. The Git server was rebuilt from scratch and its history came back exactly, 308 merge requests and 1,021 notes. The secrets store was rebuilt by restore, with the authority’s fingerprints identical.
- A slow network that fast backups hid. Incremental backups barely touch the wire, so they finished in seconds while every management link was running at 10 Mb/s half duplex. The first restore ever performed here could not cheat: it pulled at a flat 900 KB/s or so.
Where we fall short
One backup is not in production by our own rule. The Git server also gets a nightly image of the whole machine, taken since 7 September 2026, and none of those images has ever been restored. Its history is protected by the application backup, whose restore is proven, but the image backup has not earned the word.
No recovery-time target. A restore on 25 August took about 93 seconds, and about 20 minutes from shutdown to verified. But nobody has stated how fast a recovery must be, so nothing today can be called a miss, and nothing can be called a pass either.
A whole host from bare metal has never been restored. It is the last untested recovery case, and it is on our upcoming list as a step of the next fresh build.