Inference Public

Reliability

What does a recovery test actually demonstrate?

A restored database is one result. A usable service is another. Choose the finish line, record the conditions and keep the limits beside the result.

Engineer using a laptop during a timed recovery exercise.

The restore job finishes at 09:42. The database is available, the dashboard is green and the recovery looks complete. At 10:07 a member of staff still cannot sign in: the recovery environment cannot reach the identity service.

Those are hypothetical times from an illustrative exercise. The point is the gap between restoring a component and recovering the task someone needs to perform. A useful recovery test records both. Otherwise the fastest part of the process can become the headline for the whole service.

Choose the scenario before the clock

Start with one service and one failure. A deleted table, encrypted workload account, unavailable Region and supplier outage create different recovery problems. State what remains available. If the exercise assumes working identity, network access and encryption keys, it cannot establish how recovery proceeds when those are missing too.

Agree the safe boundary with the service owner and operators. Is this an isolated restoration, a rehearsal using representative data, or an authorised failover of a live service? Each can provide useful evidence. The conclusions must match the exercise actually carried out.

AWS's disaster-recovery testing guidance recommends exercising recovery paths and using the runbook to identify problems. That is an engineering activity to plan, not permission to disrupt a production service without agreement.

Define the finish line in service terms

For a hypothetical casework service, a finish line might be: an authorised caseworker signs in, finds an agreed test case, records a permitted update and can retrieve that update afterwards. The exact task comes from the service owner.

Pair the task with recovery objectives. How long can the service be unavailable? How much data loss can the organisation tolerate? Record who agreed each objective. Where there is no agreed target, make that uncertainty visible; do not manufacture a pass by choosing the target after the result.

State where time starts. Incident occurrence, detection, declaration and the start of a restore job are distinct moments. A target measured from incident declaration must include subsequent decision and mobilisation time, not just the cloud provider's restore duration.

Keep more than one timestamp

Here is an illustrative timeline, not a measured Inference Public result:

TimeObservationWhat it establishes
09:00Exercise incident declaredStart of the agreed service-recovery clock
09:12Recovery point and destination approvedDecision and mobilisation consumed 12 minutes
09:42Database restore completesData component is available for validation
10:07Sign-in fails; DNS path to identity unavailableService task is still blocked
10:26Identity route corrected through the agreed procedureA dependency is now available
10:34Caseworker completes the agreed taskThe scoped service finish line is reached

If the objective was one hour from declaration, this exercise missed it. A 30-minute restore-job result is still useful, but it is not a one-hour service recovery. The report should say both things plainly.

Test the data, not only the connection

Identify the recovery point and its timestamp. Check the recovered data against a known, safely prepared reference, then compare the data currency with the recovery-point objective. A successful login cannot establish whether the last day's submissions survived.

For a database with dependent messages or files, consider consistency across those boundaries. Restoring a database to one time and an object store to another can leave missing attachments or actions that need reconciliation. Record what was checked and what was excluded rather than describing the entire estate as consistent.

Use automation for the part it can demonstrate

AWS Backup restore testing can schedule restores and report restore-job results and durations. Configure appropriate validation and understand the cleanup behaviour for the resource types involved. That automation can provide repeatable component evidence. Your application checks and user-task rehearsal establish the additional service behaviour.

Before relying on a cross-account or cross-Region recovery path, verify current support for the actual resource type, encryption arrangement and destination. An architecture label such as 'separate recovery account' does not settle those details.

Report the limit next to the result

An honest conclusion might say: 'Under the stated scenario, using the identified recovery point, the receiving team completed the agreed casework task in 94 minutes. Identity recovery required an undocumented DNS change. External reporting and peak-load behaviour were not tested.'

That conclusion is actionable. It identifies the demonstrated behaviour, the gap and the remaining uncertainty. It also gives the next exercise a clear purpose: fix the identity procedure and repeat the affected path.

Give actions an owner and a date, and decide which finding changes the service's accepted risk. Rehearse the updated runbook with someone other than its author. A report is most valuable when the next recovery is better because of it.

Use the recovery exercise record, read the backup introduction, or request a recovery readiness session.

The complimentary session is a conversation about existing evidence. It does not execute this exercise or certify readiness; any engineering or test work is scoped separately.

Further reading

AWS: test disaster-recovery implementation

AWS Backup: restore testing

All insights

Start a conversation

Let’s talk about what you need to build.

Bring the service, the constraint or the engineering question. We’ll work through a sensible starting point.

Discuss a project