Have You Actually Restored a Backup Recently?

Here's a question worth asking your current host: when was the last time they actually restored one of your backups?

Not scheduled a restore. Not had the capability to restore. Actually spun up a clean environment, loaded your backup, and verified that the site came back healthy.

For most managed hosts, the answer is: when a customer asked us to, and sometimes not even then.

The gap between taking backups and testing them

Backup systems fail silently. A misconfigured snapshot job can run on schedule and produce files that can't be restored. A backup that passes a checksum test might still fail when you actually try to boot from it. Database dumps that work on one version of MySQL might choke on another. The only way to know your backup works is to restore it.

This is not an obscure edge case. It's routine. Enterprise backup teams have a name for the discipline: backup testing, or restore verification. The principle is basic: you don't know if a backup works until you've used it.

The managed hosting industry mostly ignores this. Backup features are marketed by their frequency and retention period — daily backups, 30-day retention, one-click restore. Whether those backups have ever been verified to actually restore is a question that doesn't appear in the comparison table.

What the nightly canary drill does

Every night, our canary process takes a recent backup and restores it to a clean, isolated environment. The restored site is health-checked: does it respond, does it load without errors, are the database contents intact. The result is logged.

If the restore succeeds, we know the backup pipeline produced a working snapshot. If it fails, we know before a customer is in an emergency. That's the whole point: find out during the drill, not during the crisis.

A failed canary tells us something specific: which step in the backup pipeline broke, whether it's a one-time failure or a pattern, and what needs to be fixed before the backup becomes load-bearing. We'd much rather know this on a quiet Tuesday than at 2am when a customer's site is down.

Why this matters to you

When your site goes down — not if, when — the speed of recovery depends on how confident the team is in the restore path. If your host has been running untested backups, the moment a restore is needed is the first moment anyone finds out whether it works. That's a terrible time to discover the backup format changed three months ago.

With a nightly canary, we have recent evidence that the restore path works. The team isn't guessing. We're executing a procedure we've run automatically the night before.

Multi-tier backup storage

The canary drill tests the restore path, but the backup itself needs to be stored reliably first. We use multi-tier storage: a primary backup to Hetzner S3 object storage and a secondary mirror via restic to a separate storage system. Two independent copies, two independent failure modes.

Backups are stored in plaintext per-snapshot manifests with no proprietary format. If you ever want to take your backups with you — or verify what's in them without our tooling — you can. No lock-in to a format we control.

Customer self-service restore

If you need to restore a backup yourself, you don't open a support ticket and wait. You initiate the restore from the dashboard, verify your identity with a one-time code sent to your email, and the restore runs. No waiting for the support queue to move.

Support tickets for backup restores are an artifact of a model where restores are expensive operations that need to be manually provisioned. When the restore path is automated and verified nightly, self-service is the natural result.

The principle

Backups are not the feature. Reliable restoration is the feature. Taking daily backups is the table stakes. Verifying every night that those backups restore correctly is the actual work.

We do it quietly, automatically, and every night. You can read more about how we approach operational reliability at /security.