{"id":6394,"date":"2026-10-29T16:25:00","date_gmt":"2026-10-29T16:25:00","guid":{"rendered":"https:\/\/www.bangaloreorbit.com\/blog\/?p=6394"},"modified":"2026-10-29T16:25:00","modified_gmt":"2026-10-29T16:25:00","slug":"automationops-plain-language-five-controls","status":"publish","type":"post","link":"https:\/\/www.bangaloreorbit.com\/blog\/automationops-plain-language-five-controls\/","title":{"rendered":"AutomationOps in Plain Language: Five Controls"},"content":{"rendered":"

Anyone who has lived in a big Indian city knows the feeling of watching a water tanker fill, valve open, hose aimed \u2014 and hoping the driver knows which gate he is filling. Automated jobs on servers create the same moment, a hundred times a day, except nobody stands by the valve. AutomationOps<\/strong> is the plain name for fixing that: it treats automated operations \u2014 restarts, clean-ups, failovers, rotations \u2014 as a system that needs rules, owners and brakes, the way the Atlas sets out on its AutomationOps reference page<\/a>. You do not need an engineering degree to follow the core of it. Five controls carry most of the weight, and each one maps to something you already understand from daily life. Here they are, explained without jargon first and with the technical names after, so you can talk to both your colleagues and your tools.<\/p>\n

Control One: AutomationOps Starts With a Rehearsal<\/h2>\n

Before the real run, the job performs everything except the final act and reports what it would have done. Weddings have rehearsals for the same reason: mistakes discovered at the mandap are expensive, mistakes discovered on Thursday are free. Engineers call this a dry run. One honest warning from practice: rehearsal tools do not always exercise everything. Some, including a popular option in the Ansible world, leave certain command-type steps unexecuted unless you switch them on \u2014 so a “successful” rehearsal may not have tested the exact line you feared. Ask one question of any rehearsal: did the dangerous part actually run? If the answer is no, the green tick you are looking at measures nothing that matters.<\/p>\n

Control Two: No Targets Means No Action<\/h2>\n

Every automated job takes instructions about what to act on \u2014 a list of servers, files, accounts, tickets. The rule: if that list arrives empty, the job must refuse to run and raise a complaint. It sounds obvious. It is also the rule whose absence deleted a company’s backups. Google has described how its Diskerase tool received a blank where a list of things to remove should have been and treated the blank as “everything”, wiping copies of data that were meant to be protected. A wrong filter in a spreadsheet, a renamed group, a query that returns nothing on a public holiday \u2014 any of these can hand a job an empty list. Refusing to act on nothing costs a few lines of code; not refusing has ended careers. Say it however you like internally \u2014 “empty never means all” is the version that stuck in the industry \u2014 but put the check in every job that accepts a list.<\/p>\n

Control Three: Small Batches, Checked in Between<\/h2>\n

No sensible person updates their phone, laptop and backup drive at the same second. You do one device, confirm it still works, then continue. Automated change deserves the same restraint, and the technique is batching: send the change to a small first wave \u2014 the canary \u2014 inspect it, then proceed in groups. Tools support this directly. In Ansible you can run hosts in series and set a limit for how high the failure percentage may climb before the run abandons the rest. The point is not perfection; it is containment. A bad change released to eight machines out of four hundred is an incident report. The same change released to all four hundred at once is a resignation letter. Health checks between waves turn a fleet-wide mistake back into a typo. Batching also buys information cheaply: the first wave tells you how the second will go, which is knowledge you cannot get any other way without spending real risk for it. And because each wave pauses for a look, a person stays genuinely in charge of the pace \u2014 the computer proposes, the timetable disposes.<\/p>\n

Control Four: A Budget of Failures, Spent Honestly<\/h2>\n

Decide in advance how many failures a run may accumulate before it stops itself \u2014 and then honour that decision. The failure budget exists because the natural instinct during a rollout is to push on: “most of them worked, the rest are probably a network blip”. Sometimes true. More often, a cluster of failures means the world changed under the job and every further action is a coin toss. Halting is not weakness; it is the whole reason you built the run to be resumable. Teams that write the number down before the rollout argue less during it, for the same reason a family that agrees on a budget in January argues less about money in June. The budget also protects the thing budgets always protect: attention. A halted run demands a look from a person, which is exactly right when the world has shifted, and exactly wrong as a way of life \u2014 if your jobs halt constantly, the problem is upstream in the targets or the job itself, not in the budget.<\/p>\n

Control Five: Every AutomationOps Job Has a Brake<\/h2>\n

Every job needs a stop control, and there should be two kinds: one brake for each job and one that halts everything. The person on call must be able to pull either without asking a manager first \u2014 at 3 a.m., permission chains are how small problems become long ones. And a stop control that has never been exercised is more hope than hardware. The Atlas recommends drilling the stop, twice a year for the riskier tiers of job, exactly the way buildings drill fire evacuation: the value is in everyone knowing the move before smoke appears. Read-only jobs earn lighter rules; jobs that can destroy data earn the strictest ones, including a second human approving before they act. The sorting of jobs into those levels \u2014 harmless-reading at the bottom, data-destroying at the top \u2014 is called tiering, and it is how a team decides which controls attach where without wasting effort.<\/p>\n

Where the five controls take you<\/h3>\n

Together, these five turn automation from a gamble into a service. Google’s own repair culture showed the payoff from the other side: once fixes were built to be safe \u2014 rehearsable, correctly targeted, batched, stoppable \u2014 they could be reapplied on a fifteen-minute schedule without anyone holding their breath. That is the real prize. Not “automation is fast” \u2014 everyone knows that \u2014 but “automation is fast and<\/em> boring”, which is what lets it run while you sleep. When a job earns that reputation, the time it returns becomes genuine: fewer emergency wake-ups, fewer rollbacks, more hours for the work only humans do.<\/p>\n

Condensed onto a single card, the whole discipline reads:<\/p>\n