Backup Recovery Drills: Verify, Restore, and Test Before You Need To

Intermediate20 min6 steps

What you'll learn

Explain why an untested backup is not yet a backup, and what a recovery drill proves
Inspect and verify a backup's integrity before trusting it for a restore
Restore into a staging area and confirm completeness rather than overwriting production blind
Pick the correct point-in-time restore when the latest backup is bad

An untested backup is Schrödinger's backup
Understand that backups are only proven by restores, and what a recovery drill validates.
The uncomfortable truth of every incident post-mortem: a backup you have never restored is not a backup — it is a hope. Backups fail silently in a dozen ways: the job wrote a 0-byte file, the archive is truncated, it backed up the wrong directory, the tape/bucket was unreadable, or it faithfully captured already-corrupted data. You discover every one of these at the worst possible moment — during a real outage — unless you rehearse.
A recovery drill is that rehearsal: you deliberately restore from a backup, into a safe place, and verify you got back exactly what you expected. It answers the only question that matters — "can we actually recover?" — while the stakes are zero.
Why rehearse when the backup 'obviously' works?
Because "the job reported success" and "the data is restorable" are different claims. A backup job can exit 0 having archived an empty or corrupt source; a restore is the only thing that exercises the whole chain — media, format, permissions, and content. Two numbers make this concrete: RPO (recovery point objective — how much data you can afford to lose) and RTO (recovery time objective — how long recovery may take). You cannot honestly claim either number without having timed and verified a real restore.
Check yourself
Your nightly backup job has reported "success" for a year, but no one has ever restored from it. In one word, what is the status of your ability to recover?
Verify the backup before you trust it
Inspect an archive's contents and integrity before restoring from it.
The first move in any restore is not to extract — it is to look. Confirm the archive is readable, contains what you expect, and isn't truncated. For a tar archive, listing the contents (-t) reads and decompresses the whole stream without writing anything, so it doubles as a cheap integrity check: if the archive is corrupt or truncated, the list fails.
Example — not runnable on your lab box
```
$ tar -tzf app-backup.tar.gz
```
```
./
./app.conf
./db.conf
./cache.conf
./worker.conf
```
-t
List contents without extracting — reads the whole stream, so it also detects truncation/corruption.
-z
Decompress gzip on the fly (.tar.gz / .tgz).
-f app-backup.tar.gz
Operate on this archive file.
-t lists the archive; -z handles gzip; -f names the file. It walks the entire compressed stream, so a clean listing means the archive decompresses end-to-end. Eyeball the list too: are the files you need actually in here, or did the job back up the wrong directory?
```
$ tar -xzOf app-backup.tar.gz ./app.conf | head
```
```
# app config
key=9f2c1a...
```
-O
Write the extracted file to stdout instead of creating it on disk — preview without unpacking.
-O extracts a single named member to STDOUT instead of to disk, so you can preview one file's contents from inside the archive without unpacking anything. Great for confirming a specific config is the version you expect before committing to a full restore.
Assuming a backup is good because the file exists
A backup.tar.gz sitting in the right place with a recent timestamp feels reassuring and proves nothing. It may be 0 bytes, half-written, or a gzip stream that errors halfway. tar -tzf (or a checksum manifest verified with sha256sum -c) is the cheap check that catches this before you rely on it in an emergency.
Restore into staging, not over production
Recover into a separate area and verify completeness before swapping into place.
When you do restore, resist the urge to extract straight over the live directory. If the backup turns out to be partial or wrong, you have now destroyed whatever was left of production on top of it. The disciplined pattern is restore into a staging area, verify, then swap in.
Example — not runnable on your lab box
```
$ mkdir -p /srv/recovery && tar -xzf app-backup.tar.gz -C /srv/recovery
```
-x
Extract (unpack) the archive.
-C /srv/recovery
Extract INTO this directory — keeps the restore off production until you've verified it.
-x extracts; -C changes into the target directory first, so everything lands under /srv/recovery instead of the current directory or production. Now you can inspect the result in isolation — nothing in the live path was touched.
Runnable in the recovery-staged-restore room — that room’s setup plants the files it reads
```
$ ls /srv/recovery && diff -rq /srv/recovery /srv/app
```
```
app.conf  cache.conf  db.conf  worker.conf
```
diff -rq
Recursively (-r) and quietly (-q) report only which files differ — fast way to confirm a restore matches a known-good tree.
Confirm the file set is complete, then (if a reference exists) diff the recovered tree against it — clean output means they match. Only after this verification do you promote staging into production. Completeness is the whole point: a restore missing files is a failed drill, not a partial success.
Check yourself
Why restore a backup into a staging directory (e.g. /srv/recovery) instead of extracting straight over the live /srv/app?
Point-in-time: the latest backup isn't always the right one
Recognize when to restore an older generation instead of the newest.
Backups are generational — you keep several dated copies. The instinct in a crisis is to grab the newest one, but that is wrong precisely when it matters most: if a bad deploy, a botched migration, or ransomware ran before last night's backup, the latest backup faithfully captured the damage. The right restore point is the last known-good generation, which may be older.
Runnable in the recovery-point-in-time room — that room’s setup plants the files it reads
```
$ ls -1 /root/backups
```
```
backup-2026-08-11.tar.gz
backup-2026-08-12.tar.gz
backup-2026-08-13.tar.gz
```
-1
One entry per line — easy to read the generations and their dates.
Three dated generations. If you know corruption was introduced on the 13th, the newest backup (08-13) captured the broken state — so the correct restore point is 08-12, the last generation taken before the damage. Restore date drives the decision, not "most recent".
Example — not runnable on your lab box
```
$ tar -xzf /root/backups/backup-2026-08-12.tar.gz -C /srv/data
```
Deliberately restoring the 08-12 generation — the last one taken before the corruption — into production, rather than the newer 08-13 that captured the bad state. This is point-in-time recovery: choosing the restore point by what the data looked like, not by which file is newest.
Restoring the newest backup after a corruption
During an incident the reflex is "restore the latest." But if the corruption (bad deploy, ransomware, a fat-fingered DELETE) happened before the most recent backup ran, that backup contains the corruption. Identify when the data was last good, then restore the generation from before that point — even if it means accepting a little more data loss (a worse RPO) to get uncorrupted data back.
Full, incremental, and the 3-2-1 rule
See how backup strategy shapes the restore, and the durability rule behind it.
How backups are taken determines what a restore needs — so a drill has to match the strategy:
- Full — a complete copy every time. The restore is dead simple: one self-contained archive. The cost is space and time, since you re-copy everything on every run.
- Incremental — only what changed since the last backup (full or incremental). Cheap and fast to create, but the restore needs the last full plus every incremental since, applied in order. If any single link in that chain is missing or corrupt, the chain breaks and you can only recover up to the last good link — so an incremental scheme means verifying the whole chain, not one file.
- Differential — everything changed since the last full. The middle ground: differentials grow larger than incrementals over the week, but the restore needs only the full plus the single latest differential — two archives, not a chain.
GNU tar can do incrementals with a snapshot file (--listed-incremental): the first run records what exists, and each later run backs up only what changed since. That's less data per run — and more pieces to keep intact for a restore.
Underneath all of this is the durability rule worth memorizing: 3-2-1 — keep 3 copies of the data, on 2 different media/types, with 1 kept offsite. It's what survives a disk dying, a building fire, or ransomware reaching your only NAS. A 'backup' on the same disk as the data isn't a backup at all.
Example — not runnable on your lab box
```
$ ls -1 /backups/week
```
```
full-sun.tgz
inc-mon.tgz
inc-tue.tgz
inc-wed.tgz
```
-1
One entry per line — read the full + its incremental chain at a glance.
An incremental set: one weekly full plus a daily chain. To restore Wednesday's state you need full-sun THEN inc-mon, inc-tue, inc-wed applied in order — every link. Lose or corrupt inc-tue and you can only get back to Monday. That dependency is exactly why the drill must verify the whole chain, not just the newest archive.
Example — not runnable on your lab box
```
$ tar --listed-incremental=/root/backup.snar -czf /backups/inc-thu.tgz -C /srv/data .
```
GNU tar incremental: the snapshot file /root/backup.snar records the previous state, so this run captures only what changed since — a small daily archive. Restoring requires the level-0 full and each incremental in sequence, feeding the same kind of snapshot back to tar with -x. Small backups, a chain to protect.
The 3-2-1 rule
3 copies, 2 different media, 1 offsite. The offsite copy is the one that saves you when the site itself is the disaster (fire, flood, ransomware that reached every local share). Test each tier: an offsite copy you've never restored from is subject to the very same 'Schrödinger's backup' doubt as any other.
Check yourself
To restore from Wednesday's INCREMENTAL backup, is that one archive enough — or do you also need the earlier full and the incrementals in between?
Try a recovery drill in your live terminal
Run a small end-to-end drill: verify, restore to staging, and confirm.
Do a complete mini-drill on the box to your right. It creates a throwaway backup, restores it into a staging area, and verifies — the exact rhythm of a real recovery, at zero stakes.
Example — not runnable on your lab box
```
$ mkdir -p /tmp/src && echo data > /tmp/src/a.txt && tar -czf /tmp/bk.tgz -C /tmp/src .
```
Sets up a tiny source and backs it up into /tmp/bk.tgz — your archive to drill against.
Example — not runnable on your lab box
```
$ tar -tzf /tmp/bk.tgz
```
Verify first: list the archive. A clean listing proves it decompresses end-to-end and contains a.txt.
Example — not runnable on your lab box
```
$ mkdir -p /tmp/recovery && tar -xzf /tmp/bk.tgz -C /tmp/recovery && cat /tmp/recovery/a.txt
```
Restore into a staging dir and confirm the content came back ('data'). That confirmation — not the extract command — is what makes it a drill.

Recap

An untested backup is only a hope: a recovery drill — a real, verified restore — is the only proof you can actually recover, and the only honest basis for an RTO/RPO.
Verify before you trust: tar -tzf reads the whole archive (catching truncation/corruption) and lists contents; tar -xzOf <archive> <member> previews one file without unpacking.
Restore into a staging area with tar -x -C <dir>, confirm completeness (and diff against a reference), then promote — never extract blind over production.
Prefer the last KNOWN-GOOD generation, not the newest: if corruption preceded the latest backup, that backup captured it, so point-in-time recovery means restoring from before the damage.
Strategy shapes the restore: a full restores from one self-contained archive; an incremental needs the full plus every incremental since, IN ORDER (verify the whole chain); a differential needs the full plus one latest differential. Keep it durable with 3-2-1: 3 copies, 2 media, 1 offsite.

Recovery-drill step → command
Step	Command	Why
List / integrity-check	`tar -tzf backup.tgz`	Reads whole stream; catches truncation and shows contents
Preview one file	`tar -xzOf backup.tgz ./f`	Inspect a member without unpacking
Restore to staging	`tar -xzf backup.tgz -C /srv/recovery`	Keep the restore off production until verified
Confirm completeness	`ls` / `diff -rq staging ref`	A restore missing files is a failed drill
Pick the right generation	`ls -1 /backups` then restore pre-incident date	Newest may have captured the corruption
Know the chain	full + incrementals (in order) / full + differential	Incremental restore needs every link intact
Keep it durable	3 copies · 2 media · 1 offsite (3-2-1)	Survives disk death, fire, ransomware

Backup Recovery Drills: Verify, Restore, and Test Before You Need To

An untested backup is Schrödinger's backup

Verify the backup before you trust it

Restore into staging, not over production

Point-in-time: the latest backup isn't always the right one

Full, incremental, and the 3-2-1 rule

Try a recovery drill in your live terminal