Anyone who has tried to book tickets for a big Bengaluru concert the minute sales open knows the specific heartbreak of a site that fails under pressure. Behind most of those failures sits a quieter one: a test pipeline that said everything was fine right up until it wasn’t. A few months ago I sat with the two-person engineering team behind a mid-sized events platform and helped them run a proper review of that hidden machinery — a TestOps audit, twelve checks covering whether their automated verdicts deserved the faith the business put in them. What we found, and fixed, is written up below as a checklist any small team can borrow.
How We Ran the TestOps Audit
We gave ourselves one afternoon and one rule: touch nothing, measure everything. Each check ended in a score of green, amber, or red, and honest ambers counted as reds with better manners. Before reading on, guess how your own pipeline would score. Most teams we talk to expect eight or nine greens and discover four. The gap between imagined and actual is the whole reason audits exist — the same reason a venue walk-through happens before doors open, not after.
Checks One to Four: The Paper Trail Behind Every Verdict
- 1. Can you find yesterday’s results? Every outcome needs a permanent home, filed under the test’s own classname and method label, stamped with the revision it judged and which attempt produced it. This team stored nothing older than the current run, which meant every failure investigation started from memory and ended in folklore.
- 2. Does the gate always run? Their release check was wired to skip entirely when earlier jobs failed, so branch protection happily waved through broken builds. The gate has to execute even on failed pipelines; on GitHub Actions that means guarding it with something like an if: !cancelled() condition rather than the default.
- 3. Is history kept long enough to spot patterns? A flaky test is only visible across runs. One week of retained results is the minimum for honest analysis.
- 4. Can you export the raw attempt-level record? Whoever holds your history holds your options. Vendors come and go; if results can’t leave, you’re renting your own memory.
The team scored one green out of four, and the green was generous.
Checks Five to Eight: Environments and Data
- 5. Is each dependency faked at the cheapest honest level? The ladder runs from an in-process stand-in, to a lightweight fake, to a throwaway container spun up per run — Testcontainers is the popular route — to fully separate per-change environments. They were running a full staging clone for tests that a stub would have served.
- 6. Do services prove their promises to each other? For the booking-to-payments boundary, contract tests answer in seconds what a staging deploy takes hours to suggest. If you use Pact, the can-i-deploy check belongs in the release path.
- 7. Does every test bring its own data? Records should be created through the application’s own setup routines, uniquely named for the run, and cleaned up afterwards. Half their failures traced to two tests sharing one “test-user” account.
- 8. Is any real customer data in the loop? Masked subsets, never full production copies. This one was green, mercifully.
Checks Nine to Twelve: The TestOps Ownership Test
- 9. Is the quarantine capped? Bench unreliable tests by all means, but set a hard limit. A workable pattern from Martin Fowler’s writing: around eight tests benched for at most one week. Beyond the cap, fixing beats shelving. Their quarantine held sixty-one tests with no expiry — a museum, not a waiting room.
- 10. Do you know what a verdict costs? Compute minutes spent divided by answers produced. Nobody had ever computed it; the figure, once on a whiteboard, funded its own cleanup project.
- 11. Does a developer own the tests, or does a “quality” queue? Whoever writes the change should run and fix its checks. Centralised test ownership reliably produces untestable code and resentful queues.
- 12. Is anyone answerable for the false-alarm rate? Not the failure rate — the rate of wrong answers. If no name sits beside that number, it is nobody’s number.
The Score, and What Changed in a Month
Final tally: three green, two amber, seven red. A month later, six of the reds had turned. Results got a proper home with revision tags; the gate learned to run on failures; the staging-for-everything habit gave way to containers; the sixty-one-test museum was reduced to five benched tests, each with an owner and a date. The pipeline got faster as a side effect, because the audit’s habit of asking “what is this actually for?” prunes as it goes. Just as valuable, the two-person team stopped arguing about whose fault a red build was — the paperwork settled those debates before they started.
If you want the reference material behind the twelve checks, the Ops Atlas page on TestOps covers the same territory — stage budgets, environment ladders, quarantine discipline — in careful detail.
The audit mindset will feel familiar to anyone who plans outings in this city. Choosing between competing events is its own audit — our piece on picking the right event in Bangalore walks the same compare-then-commit path. The stage-by-stage structure rhymes with what Bengaluru’s theatre scene offers: the cheap preview before the expensive opening night. And once the audit hands you a shortlist, where to track upcoming events keeps the calendar honest — which is, in the end, what a results store does for a pipeline.
Run the twelve checks this week. Score them honestly, fix the ugliest one first, and repeat next quarter — twelve honest answers, even ugly ones, beat one reassuring guess every single time.
