How to automate backups you have actually tested restoring

Backups are among the easiest things to automate and among the easiest to get wrong, because the automation reports success in both cases. A job that runs nightly and writes an empty file is indistinguishable, from the outside, from one that works.

The only thing that distinguishes them is restoring. Everything else in this article is secondary to that, and a team that automates the backup without automating the restore test has automated the reporting rather than the protection.

What we'll cover

• The backup that contains nothing
• Restore drills, the only real verification
• What actually needs backing up
• Building it with CodeWords
• Retention, and what it is really for
• Separation, and why it matters more now
• The two numbers that define the policy
• Restoring under pressure
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

The backup that contains nothing

The failures are specific and they all report success.

An empty or truncated dump, because a credential expired partway or a table was locked.

A backup of the wrong thing — the old database after a migration, a directory that was renamed.

A partial backup that succeeded on nine tables and skipped the tenth silently.

A corrupted archive that writes fine and cannot be opened.

An encrypted backup nobody can decrypt, because the key was only in the environment that is now gone.

A backup that stopped months ago, where the job failed silently and no alert existed for its absence.

Check the size. A backup significantly smaller than yesterday's is the cheapest possible detection of most of these, and it is rarely configured.

Restore drills, the only real verification

Restore on a schedule, automatically, to a scratch environment.

Assert on the contents. Row counts within an expected range, the presence of a recent record, a checksum — because a restore that completes and produces an empty database has not verified anything.

Time it. How long a restore takes is the number that matters during an incident, and it is almost always longer than people assume.

Restore the oldest backup you retain, occasionally, not only the newest — since a format or key change breaks old backups in a way nobody notices.

Do a full drill periodically with a person: restore, bring the application up against it, confirm it works. The automated check verifies the data; the drill verifies the procedure.

Record the result. "Last successful restore" is a more meaningful status than "last successful backup" and is the one worth putting on a dashboard.

What actually needs backing up

Worth listing, because the gaps are consistent.

Databases, obviously, and usually the only thing anybody thinks of.

Uploaded files and media, which frequently live somewhere else and are frequently missed.

Configuration and secrets, without which a restored database is a database you cannot connect to.

Infrastructure definitions, which are in version control if you are fortunate and in a console if not.

Third-party state — what is configured in your payment provider, your email service, your identity provider — which is not backed up by anybody and which a real disaster would require recreating.

Not everything. Caches, derived data, and anything reconstructible are cost without benefit, and excluding them deliberately makes the restore faster.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the verification and the reporting can run across the storage, the database, and wherever the team is told.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: what to verify, how often, and who to tell when it fails.
3. Authorize the connections, with the narrowest access each step needs.
4. Describe the exceptions: a backup smaller than expected, a restore that fails, a job that did not run at all.
5. Break something deliberately — rename a source, revoke a credential — and confirm the automation notices.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Retention, and what it is really for

Most retention policies are set by instinct, and the instinct answers the wrong question.

Recent backups protect against failure — a bad deployment, a dropped table, hardware.

Older backups protect against discovery delay. Data corrupted two months ago and noticed today can only be recovered from a backup older than the corruption.

That second case is what sets your longest retention, not a round number of days.

Ask how long a subtle problem could go unnoticed. For most systems the answer is longer than the retention they have.

Balance against cost and obligation. Some data you are required to keep; some you are required not to keep longer than necessary, and backups are frequently forgotten in the second category.

Write the policy down, including the reasoning, because otherwise the next person changes it to a round number.

Separation, and why it matters more now

A backup reachable from the compromised system is not a backup. Ransomware encrypts what it can reach, and that has included backups routinely.

Separate credentials, so the account that writes backups cannot delete them and the production credentials cannot reach them at all.

Separate account or provider for the copy that matters, so a compromise of one does not reach both.

Immutable or write-once storage where available, which is the strongest practical protection and is now widely offered.

Offline or logically isolated for at least one copy, if your risk assessment warrants it.

Test that the separation holds — try to delete a backup using production credentials and confirm you cannot.

The two numbers that define the policy

How much data you can afford to lose, which sets how often backups run. Hourly and nightly are very different answers and the difference is usually a business decision rather than a technical one.

How long you can afford to be down, which sets how the restore works. A restore that takes eleven hours is incompatible with a two-hour tolerance regardless of how good the backup is.

Measure the second. Most teams have never timed a full restore and are surprised.

They differ per system. The customer database and the internal wiki do not need the same answer, and treating them alike overspends on one and underprotects the other.

Agree them with whoever owns the business risk, not within engineering alone, because both are trade-offs against cost.

Restoring under pressure

The procedure matters as much as the data, and it is exercised for the first time at the worst possible moment.

Write it down somewhere your systems are not. A runbook in the wiki that is hosted on the infrastructure you are restoring is not available.

Name who decides. Restoring is destructive — it replaces current state — and somebody has to authorise it rather than an engineer deciding alone at two in the morning.

Decide the restore target in advance. Restoring over production and restoring alongside it are different choices with different risks, and choosing during an incident goes badly.

Know the order. Database, then files, then configuration, then bringing services up — with the dependencies written down, because they are not obvious under pressure.

Expect partial recovery to be the realistic case, and have a view on what matters most if you cannot get everything back at once.

Rehearse with the people who would actually do it, not only the person who built it, because that person may be unavailable.

Making it survive

Alert on absence. A backup job that stopped produces nothing, and nothing looks exactly like a quiet night — this is the most common backup failure of all.

Check size against the previous run, which catches most silent failures cheaply.

Keep the keys somewhere the disaster does not reach, and test that somebody other than one person can retrieve them.

Document the restore procedure somewhere available when your systems are not, which is a genuinely common oversight.

Report the outcome. Backups taken, size, duration, last successful restore, and anything that did not run.

Limits worth knowing about

Backup and replication are different things. A replica propagates a dropped table immediately; it protects against hardware failure and not against mistakes.

Snapshots are tied to their provider, so a provider-level problem can take the snapshots with it.

Large restores take hours, and the time scales with size in ways that surprise people who have never measured it.

Third-party state is usually not backed up at all, which is the gap that makes a genuine disaster recovery longer than the plan says.

Backups contain personal data, which means retention and deletion obligations apply to them too — a deletion request that does not reach backups is incompletely honoured.

What to build first

An automated restore test: restore the most recent backup to a scratch environment on a schedule and assert on its contents, reporting the result. It converts "the backup job succeeded" into "the data is recoverable", which are different claims and only the second is worth anything.

Two habits make the difference. Assert on contents rather than on the restore completing, since an empty database restores perfectly. And alert when a backup does not run, not only when one fails — a job that silently stopped is the most common backup failure and the only one that produces no signal at all.

Frequently asked questions

How do I know a backup is any good?

Restore it, automatically, on a schedule, and assert on the contents — row counts in an expected range, a recent record present. A restore that completes and produces an empty database has verified nothing.

What is the most common backup failure?

A job that silently stopped running. It produces no output, no error, and no alert unless you specifically watch for absence — which is why "last successful restore" is a better status than "last successful backup".

Is a replica a backup?

No. A replica propagates a dropped table or a bad migration immediately. It protects against hardware failure; it does not protect against mistakes, which is what most recoveries are actually for.

How long should backups be kept?

Long enough to cover how long a subtle corruption could go unnoticed, which is usually longer than people's instinct. Recent backups handle failures; older ones handle discovery delay, and the second sets your longest retention.

What is usually missing from the backup set?

Uploaded files, configuration and secrets, and third-party state — what is configured in your payment provider, email service, and identity provider. The last is backed up by nobody and is why real disaster recovery takes longer than the plan.

Why does separation matter so much?

Because a backup reachable from a compromised system is not a backup. Use separate credentials that cannot delete, a separate account or provider, and immutable storage where it is offered — then test that production credentials genuinely cannot reach it.

Do deletion requests apply to backups?

Yes, which is frequently overlooked. A deletion honoured in the live system and not in backups is incompletely honoured, and it is worth agreeing the approach with whoever advises you rather than discovering it during an audit.

Where should the restore procedure live?

Somewhere that does not depend on the systems being restored. A runbook in a wiki hosted on the infrastructure you are recovering is not available at the moment you need it — which is a common and entirely avoidable discovery.

Who authorises a restore?

Somebody named, decided in advance. Restoring replaces current state, so it is destructive, and an engineer deciding alone at two in the morning under pressure is not the process anybody would design deliberately.

Related reading

• Automating API health checks
• Infrastructure as code
• PostgreSQL and CodeWords