Testing disaster recovery properly

A recovery plan that has not been tested is a document describing what somebody hopes will happen. The gap between the two is discovered at the worst possible time, and it is usually larger than anyone expected.

Testing is not complicated. It is mostly a matter of doing it regularly enough, at a level that matches what you are actually claiming.

Test at the level you are claiming

There is a hierarchy, and each level proves something the one below it does not.

Backup completed successfully. This proves a job ran. It does not prove the data is usable.

A file was restored. Proves the backup contains readable data. Does not prove the system can be rebuilt.

A system was restored and started. Proves the system can come back. Does not prove it works, because it may be missing dependencies it needs.

A system was restored and verified working. Someone logged in, ran a transaction, confirmed the data is correct and current to the expected point. This is the first level that means anything to the business.

A service was recovered end to end. All the components of a business service, in the right order, with the network paths and identity infrastructure they depend on. This is what an actual disaster requires.

Most organisations test at level two and describe themselves as being at level five.

The dependencies that break recoveries

Individual system recovery usually works. Service recovery fails, and it fails on the things that were not thought of as part of the system.

Identity. Almost everything requires authentication. If the directory service is not recovered first, nothing else can be logged into. It should be first in every recovery order and it frequently is not.

DNS. Systems find each other by name. A recovered environment where nothing can resolve anything is not recovered.

Certificates. Restored systems present certificates. If those are expired, or issued for hostnames that changed in recovery, connections fail in ways that look like network problems.

Licensing. Software tied to hardware identifiers, MAC addresses or activation servers may refuse to run on recovered infrastructure. This is a common and infuriating discovery mid-test.

Network paths. Firewall rules, routing and address assumptions that exist in production but were never replicated to the recovery environment.

Credentials. The recovery procedure requires administrative access to systems that are down. If those credentials live only in a system that is also down, the procedure cannot start. This is the single most common circular dependency and it is entirely avoidable.

External integrations. Partners, payment providers and suppliers that allow-list your production addresses. Traffic from a recovery environment may be rejected.

The documentation itself. If the runbook lives on the file share that is unavailable, it is not a runbook.

Types of test, and what each is for

Tabletop. Walk through the scenario in a room. Cheap, disruption-free, and genuinely useful for finding gaps in decision-making, communication and ownership. It will not find technical problems.

Component restore. Restore individual systems to an isolated environment and verify them. Low risk, and it should be routine rather than an event. Rotate through the estate so everything is covered over a year.

Service recovery test. Recover a complete business service, with its dependencies, into an isolated environment. This is the test that matters most and the one most often skipped because it takes real effort.

Full failover. Actually move production. The most convincing and the most disruptive. Worth doing if your recovery commitments are serious, and worth building up to rather than attempting first.

A workable cadence

• Monthly: automated restore verification of a sample. This should require no human effort.
• Quarterly: a service recovery test for one significant service, rotating so all are covered annually.
• Annually: a tabletop covering a scenario you have not tested, and a full test of your most critical service.
• On change: whenever a service changes materially, its recovery procedure is re-verified.

The last one prevents the common situation where the recovery plan describes an architecture that was replaced a year ago.

Measure the right things

Time to recover, measured honestly. From declaring the incident to the service being usable, including the time spent finding people, locating documentation and making decisions. Plans that record only the technical restore time understate reality substantially.

Data loss, measured. How much data was actually lost between the last recovery point and the failure. Compare it to what you committed to.

What was missing. Every test finds something. Record each item, assign it, fix it before the next test. A test that finds nothing was not a real test.

Common ways tests mislead

Testing with the people who wrote the plan. They know the undocumented parts. Have someone else execute it and see whether the document is sufficient.

Testing during working hours with everyone available. Real incidents happen at inconvenient times with half the team unreachable. At minimum, note who was involved and ask whether the recovery would have worked without them.

Skipping the parts that are difficult. Assuming the network is available, assuming identity is up, assuming the backup is current. These assumptions are exactly what fails.

Testing recovery to the same infrastructure. If the scenario is the loss of a site, recovering to that site proves nothing.

What to do with the results

The output of a test is not a pass or fail. It is a list of specific gaps, each with an owner and a date.

Write down the recovery time you actually achieved, next to the recovery time you have committed to. Where they differ, either close the gap or change the commitment. Continuing to publish a recovery objective that testing has shown to be unachievable is worse than having no objective, because people plan around it.