Anyone who has tried to reach a concert venue during a heavy downpour in a big city knows the feeling: the route you planned is fine on paper, but the road you needed is blocked, the cab app is overloaded and the ticket QR code will not open without a signal. Systems behave the same way when something big goes wrong. This piece uses that city-travel mindset to diagnose why recovery plans fail, drawing on the discipline called DisasterRecoveryOps, and offers a fix for each symptom.
The framework comes from the Ops Atlas entry covering DisasterRecoveryOps, which treats recovery as continuous operational practice rather than a document. We will move through seven symptoms in the order a team is likely to meet them, and for each one separate what looks wrong from what is actually wrong.
Symptom 1: “We have a plan, but the clock still ran out”
The cause is usually hidden in the stopwatch, not in the servers. A recovery target is not only about how fast systems switch over; it also includes how long people take to notice a problem and agree to act. The Atlas illustrates this with a drill in which technical failover took thirty-three minutes and human steps took thirty-one, so the full sixty-four-minute run overshot a one-hour objective.
Think of a ticketing desk at a festival. The gates can reopen in minutes, but if the duty manager spends half an hour deciding whether to reopen them, the queue is already out of the building. The fix is a pre-agreed decision deadline, found by taking the target and removing the slowest recent drill and a safety margin. In the Atlas example the figure is twenty-two minutes, and it is written down before anything breaks.
Symptom 2: “The standby is up, but nobody can sign in”
This is the cab-app problem. The route exists, yet the tool you need to reach it relies on the same network that is down. The sheet points to a large regional outage in October 2025 where sign-in and identity services were among the systems that struggled. A team whose emergency access depended on those services would have been locked out of its own rescue.
The remedy is dull and effective: keep a separate set of recovery-only credentials in a vault that lives outside the zone of failure, and prove on a quiet day that the vault really opens. Rotate it when people leave. In event terms, this is the paper ticket in your bag when the phone dies.
Symptom 3: “We failed over, but both sides are still accepting work”
Here two copies of the system each believe they are in charge. Visitors booking tickets through the old copy keep creating records that the new one never sees. The Atlas describes the correct order as fence, promote, then redirect. Fencing deliberately shuts the old environment off from taking changes.
The sheet adds the awkward detail that turning routing off only steers traffic that arrives by name lookup. Scheduled jobs, queue consumers and clients that stored an address earlier can slip through. Walk through the venue after closing time and check every side door, not only the main entrance; the Atlas also admits that fencing for one managed database is best-effort.
Symptom 4: “The data is back, but the numbers look wrong”
When a failover happens suddenly, writes that had not yet reached the second copy are gone from it. Reconciling them needs care. The Atlas lays out a sequence: protect the old primary first, because an automatic snapshot may expire along with its retention period; compare using a restored copy instead of production; then replay the missing transactions through the application so that rules are applied once and only once.
Picture two clerks who both sold tickets for the same seat during a power cut. Matching their books afterwards is possible, but only if the books are kept safe first and nobody scribbles over them.
Symptom 5: “Our provider said it was covered”
Another frequent cause of surprise is assuming protection that was never purchased. The sheet quotes one cloud vendor’s documentation, which warns that recovery is not something the platform hands you automatically. It also separates high availability from recovery. Spreading a service over several zones helps with a single data-centre problem, but a region-wide failure or a bad deployment copied everywhere is another matter. A flawed content update in July 2024 made the point vividly: a second site cannot rescue you from a fault that every site shares.
Bangalore visitors choosing between theatre, comedy and live music already know that one wrong assumption about a venue’s access rules can undo a plan. For a refresher on picking wisely, see our guide on choosing a suitable event as a visitor, and treat your technology choices with the same scepticism.
Symptom 6: “The drill passed, so we assumed we were ready”
One good exercise is a data point, not a guarantee. The Atlas describes a maturity ladder, and its third rung, labelled exercised, wants every top-priority service drilled plus two drills in a row that land inside the target. For tracking it proposes four measures: how much was drilled, how closely the plan was followed, how long the decision took, and the margin left against the objective. Its worked example has drills covering 67% of what should be covered and only 25% of the plan’s steps followed as written, a very different picture from one green tick.
Evenings out follow the same logic. A smooth first visit to a venue does not tell you how it copes with a full house, and a team that has only drilled in quiet conditions has learned less than it thinks. If you want a feel for how varied city programmes can be, our article on the range of stage shows on offer around the city shows how many different formats organisers have to keep running.
Symptom 7: “We got back up, and then everything fell apart again”
Going back to the original site is a second move, and it often has no plan at all. In the sheet’s words, failback amounts to a second migration, and postponing its planning is listed among the anti-patterns. Decide early whether you will stay on the standby, how you will bring the old site up to date, and when you will cut across.
A compact diagnostic table in words
- Slow overall: measure the people steps, not just the machine steps.
- Locked out: add outside-the-zone credentials.
- Duplicate records: fence the old side and check every back door.
- Missing records: preserve, compare on a copy, replay once.
- Misplaced trust: read what the provider actually covers.
- False comfort: count coverage across all top-priority services.
- Messy return: plan the way back before you leave.
None of these fixes needs a large budget. They need an hour of honest discussion, a stopwatch and a willingness to find out the plan was weaker than hoped. That is the practical promise of treating recovery as a habit: you meet the surprises in rehearsal, on a calm day, rather than at the worst possible moment.