Picture a three-person team in Bangalore that keeps a live calendar of plays, gigs and weekend markets. Their listings feed a website, a newsletter and a WhatsApp broadcast. One Friday afternoon the database behind it all refuses to start. Whether that Friday becomes a story about a long evening or a story about a lost season depends on decisions the team made months earlier. It is a made-up scenario, yet it shows what BackupOps looks like when it is done properly.
The team and what they kept
Asha handled partnerships with venues, Dev wrote the listings and Rohini looked after the site. They stored venue contacts, ticket links, show timings and a few thousand subscriber addresses. None of them had a technology career in the strict sense, though Dev had dabbled in scripting. The part-time technical lead, Imran, set one rule after a scare the previous year: nothing counts as saved until someone has opened it from the copy.
That rule echoes the Ops Atlas, whose BackupOps entry describes the topic as restore engineering rather than copying. It adds that the label is informal, with no recognised standard behind it, which suited a team that wanted habits, not paperwork.
Friday, 4:10 pm: the outage
The database corrupted itself after a hosting-side disk fault. Rohini checked the dashboard and found the usual green status. Imran, called in from a train, asked a different question: “When did we last open the copy?” The log said three weeks ago, with a note that a sample of fifty venues had loaded correctly. That one line saved the evening, because it meant they knew the copy was readable, and roughly how stale it was.
Where the copies were
The team had arranged things so that no single event could reach everything. Production lived at the hosting company. A nightly export went to a storage bucket under a separate login that only Imran and Asha held. A weekly extra copy went onto an encrypted drive that Asha kept at home. The Atlas gives the reasoning: count the number of independent failure domains, not the number of copies, and keep at least one copy that everyday credentials cannot delete.
The Atlas points to a 2024 episode in which a pension fund’s cloud setup at a major provider was deleted by a provisioning error, along with its duplicate in the same service. Recovery was credited to copies held outside the deleted environment, though accounts differ on how decisive third-party copies were. Imran printed that paragraph for the team as a reminder that “two copies in one place” had failed someone far bigger.
The restore, hour by hour
- 4:30 pm – Imran confirms the nightly export from last night is present and not zero bytes.
- 4:50 pm – A fresh database is created at the host; the export is loaded.
- 5:40 pm – Row counts are compared against the figures noted in the last drill. Venues and shows match; the subscriber table is short by a handful of recent sign-ups.
- 6:15 pm – The missing sign-ups are re-added from the mailing tool’s own history.
- 6:45 pm – Site is back; Dev publishes the weekend list.
The loss window was about a day of edits, which matched what they had agreed was tolerable. The Atlas warns that the delay between a job starting and the copy landing in its safe location also counts toward that window, so Imran had measured the full run, not just its start time.
What the team learned afterwards
In the review, the team listed what went right and what was lucky. The key was an owner, a written restore card and a habit of drilling. The lucky parts were that the corruption struck at night’s end rather than mid-run and that the earlier drill happened to cover the subscriber table. They added more checks: an alert that triggers when no successful export appears within 26 hours, sent to a messaging channel outside the hosting company.
Tying it to the city we write about
Readers who follow our coverage of stage shows and theatre options around the city or the piece on where tourists can catch stand-up and other live acts are really reading the output of exactly this sort of small, careful data operation. A listing is only helpful if the show times are right, and they stay right only if the data behind them is recoverable.
The conversation they had with venues
After the incident Asha did something unexpected: she told two of the venues what had happened and what the team had changed. Both were unbothered, and one asked how to set up something similar for its own box-office sheet. Being open about it turned a near miss into a small piece of goodwill. It also pushed the team to write down, in plain sentences, which data they hold about venues and how long they keep it, because partners now asked.
A career angle
For anyone wondering how this connects to tech careers, restore skills are in short supply because they are unglamorous. Someone who can calmly load a copy, verify it and explain the gap to colleagues is valuable on any team. The Atlas lists KPIs such as restore-test currency and restore time against budget, which gives a beginner concrete numbers to track and talk about in an interview.
Read the source
If the story made you curious, the full text is in the Ops Atlas write-up on BackupOps, including a 30, 60 and 90 day plan a small team can scale down.
The team in our tale never became famous. They just kept their calendar alive through a bad Friday, and the following week they ran the drill again, because the log said it was due.