{"id":6389,"date":"2026-10-28T16:25:00","date_gmt":"2026-10-28T16:25:00","guid":{"rendered":"https:\/\/www.bangaloreorbit.com\/blog\/?p=6389"},"modified":"2026-10-28T16:25:00","modified_gmt":"2026-10-28T16:25:00","slug":"releaseops-weekend-listings-diagnosis","status":"publish","type":"post","link":"https:\/\/www.bangaloreorbit.com\/blog\/releaseops-weekend-listings-diagnosis\/","title":{"rendered":"ReleaseOps for Event Listings: Fixing Friday Nights"},"content":{"rendered":"

The Friday pattern was so regular the team could have sold tickets to it. Thursday night, the new build of the city listings platform went up; Friday morning, someone in the office opened the site, frowned, and typed “rollback” into the group chat. By the time a fix shipped, the weekend crowd \u2014 the site’s whole reason for existing \u2014 had spent hours looking at broken venue pages for shows like the theatre listings everyone visits on a Saturday<\/a>. The team did not lack effort or talent. What they lacked was a diagnosis, and it took an outside consultant one afternoon to name it: their installs and their audience exposure were the same event. ReleaseOps, the practice she introduced from the Ops Atlas guide to release operations<\/a>, begins exactly there.<\/p>\n

Symptom one: every ReleaseOps-less deploy is a bet on everyone<\/h2>\n

The platform pushed changes the way most small teams do: copy the new version over the old one, in one motion, for every visitor at once. Nothing in that motion is staged, measurable, or reversible on its own terms \u2014 reversing means pushing the old version back over the new, which is the same all-at-once bet in the other direction. The consultant drew the split on the whiteboard that changed how the team thought: getting the new version onto the servers is one action; letting specific people encounter it is a second action, and the second one deserves its own controls. Two dials \u2014 which build runs where, and who meets which behaviour \u2014 had been welded together since the first commit.<\/p>\n

Symptom two: rollbacks nobody had rehearsed<\/h2>\n

When the team looked honestly at their incident history, the rollbacks were theatre. The first attempt during any incident took three times longer than anyone predicted, because the steps existed in the head of one developer and a document nobody had opened since Diwali. The corrected habit came straight from the practice: rehearse the return journey when nothing is wrong, on a quiet Tuesday, and time it. A rollback drilled in calm conditions is a tool; a rollback attempted live for the first time is a wish. The team also learned the data caveat the hard way \u2014 a return to the old version only works if the database still speaks the old version’s language, which is why schema changes now travel in two steps, forward-compatible first, cleanup later.<\/p>\n

Symptom three: the freeze that froze nothing useful<\/h2>\n

The old policy was a blanket change freeze from mid-December to January. It felt prudent and did nothing: the delivery research on freeze behaviour found routine code freezes do not reduce failed changes, and the team’s own incident log confirmed the December halt simply moved the same failures to January. The ReleaseOps replacement was narrower and sharper. A short change freeze stays in place around genuinely fragile windows \u2014 New Year’s Eve listings, a major festival week. Beyond that, the platform adopted the error-budget halt: the site carries an allowance of failure it may spend each month, and when an incident or two exhausts it, every non-urgent change pauses automatically, security fixes excepted, until the allowance refills. The first time it triggered, the team complained. By the third time, they trusted it like a thermostat.<\/p>\n

The fix, staged the ReleaseOps way<\/h2>\n

What shipped over the following six weeks was unglamorous and effective:<\/p>\n

    \n
  1. New versions install everywhere but serve nobody \u2014 the internal crew sees them first, on staging URLs that mirror production.<\/li>\n
  2. A ring order was written down: staff accounts, then two hundred loyal users who had opted in as testers, then five percent of the city’s visitors, then everyone.<\/li>\n
  3. A waiting period sits between each hop \u2014 long enough that a bad venue import or broken filter surfaces while the blast radius is a few hundred people, not a few hundred thousand.<\/li>\n
  4. Behaviour-affecting settings, including the content imports for events like the comedy-and-live-entertainment calendar<\/a>, now travel the same staged road as code, because an import is a release whether or not anyone calls it one.<\/li>\n
  5. Every reversible change got a switch with an owner and an expiry date; the overdue list is reviewed monthly, and switches actually get removed.<\/li>\n<\/ol>\n

    What the season after looked like<\/h2>\n

    The numbers moved before the culture did. Problems once discovered by the weekend audience now surfaced among the tester ring on Thursday afternoon; the gap between a change going wrong and the site returning to safety shrank from hours to minutes; and the false alarms \u2014 panics over changes that turned out fine \u2014 dropped once the team set their alerts at the level the statistics demand rather than at the effect they feared. An unplanned bonus: because the platform finally had distinct readings for install health and audience experience, the editors stopped blaming infrastructure for problems that were really content imports, and the developers stopped blaming content for problems that were really infrastructure. A year later, the same staged approach quietly carried a partnership that syndicated Kerala’s events and travel experiences<\/a> onto the platform \u2014 launched ring by ring, no held breath anywhere. Friday nights, it turns out, were never cursed. They were just overexposed.<\/p>\n","protected":false},"excerpt":{"rendered":"

    A city events site kept breaking on weekend pushes. The diagnosis: installs and audience exposure were welded together \u2014 ReleaseOps split them.<\/p>\n","protected":false},"author":9,"featured_media":6390,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4896],"tags":[],"class_list":["post-6389","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-travel"],"_links":{"self":[{"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/posts\/6389","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/comments?post=6389"}],"version-history":[{"count":1,"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/posts\/6389\/revisions"}],"predecessor-version":[{"id":6393,"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/posts\/6389\/revisions\/6393"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/media\/6390"}],"wp:attachment":[{"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/media?parent=6389"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/categories?post=6389"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bangaloreorbit.com\/blog\/wp-json\/wp\/v2\/tags?post=6389"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}