Anyone who has lived in a big Indian city knows the feeling of watching a water tanker fill, valve open, hose aimed — and hoping the driver knows which gate he is filling. Automated jobs on servers create the same moment, a hundred times a day, except nobody stands by the valve. AutomationOps is the plain name for fixing that: it treats automated operations — restarts, clean-ups, failovers, rotations — as a system that needs rules, owners and brakes, the way the Atlas sets out on its AutomationOps reference page. You do not need an engineering degree to follow the core of it. Five controls carry most of the weight, and each one maps to something you already understand from daily life. Here they are, explained without jargon first and with the technical names after, so you can talk to both your colleagues and your tools.
Control One: AutomationOps Starts With a Rehearsal
Before the real run, the job performs everything except the final act and reports what it would have done. Weddings have rehearsals for the same reason: mistakes discovered at the mandap are expensive, mistakes discovered on Thursday are free. Engineers call this a dry run. One honest warning from practice: rehearsal tools do not always exercise everything. Some, including a popular option in the Ansible world, leave certain command-type steps unexecuted unless you switch them on — so a “successful” rehearsal may not have tested the exact line you feared. Ask one question of any rehearsal: did the dangerous part actually run? If the answer is no, the green tick you are looking at measures nothing that matters.
Control Two: No Targets Means No Action
Every automated job takes instructions about what to act on — a list of servers, files, accounts, tickets. The rule: if that list arrives empty, the job must refuse to run and raise a complaint. It sounds obvious. It is also the rule whose absence deleted a company’s backups. Google has described how its Diskerase tool received a blank where a list of things to remove should have been and treated the blank as “everything”, wiping copies of data that were meant to be protected. A wrong filter in a spreadsheet, a renamed group, a query that returns nothing on a public holiday — any of these can hand a job an empty list. Refusing to act on nothing costs a few lines of code; not refusing has ended careers. Say it however you like internally — “empty never means all” is the version that stuck in the industry — but put the check in every job that accepts a list.
Control Three: Small Batches, Checked in Between
No sensible person updates their phone, laptop and backup drive at the same second. You do one device, confirm it still works, then continue. Automated change deserves the same restraint, and the technique is batching: send the change to a small first wave — the canary — inspect it, then proceed in groups. Tools support this directly. In Ansible you can run hosts in series and set a limit for how high the failure percentage may climb before the run abandons the rest. The point is not perfection; it is containment. A bad change released to eight machines out of four hundred is an incident report. The same change released to all four hundred at once is a resignation letter. Health checks between waves turn a fleet-wide mistake back into a typo. Batching also buys information cheaply: the first wave tells you how the second will go, which is knowledge you cannot get any other way without spending real risk for it. And because each wave pauses for a look, a person stays genuinely in charge of the pace — the computer proposes, the timetable disposes.
Control Four: A Budget of Failures, Spent Honestly
Decide in advance how many failures a run may accumulate before it stops itself — and then honour that decision. The failure budget exists because the natural instinct during a rollout is to push on: “most of them worked, the rest are probably a network blip”. Sometimes true. More often, a cluster of failures means the world changed under the job and every further action is a coin toss. Halting is not weakness; it is the whole reason you built the run to be resumable. Teams that write the number down before the rollout argue less during it, for the same reason a family that agrees on a budget in January argues less about money in June. The budget also protects the thing budgets always protect: attention. A halted run demands a look from a person, which is exactly right when the world has shifted, and exactly wrong as a way of life — if your jobs halt constantly, the problem is upstream in the targets or the job itself, not in the budget.
Control Five: Every AutomationOps Job Has a Brake
Every job needs a stop control, and there should be two kinds: one brake for each job and one that halts everything. The person on call must be able to pull either without asking a manager first — at 3 a.m., permission chains are how small problems become long ones. And a stop control that has never been exercised is more hope than hardware. The Atlas recommends drilling the stop, twice a year for the riskier tiers of job, exactly the way buildings drill fire evacuation: the value is in everyone knowing the move before smoke appears. Read-only jobs earn lighter rules; jobs that can destroy data earn the strictest ones, including a second human approving before they act. The sorting of jobs into those levels — harmless-reading at the bottom, data-destroying at the top — is called tiering, and it is how a team decides which controls attach where without wasting effort.
Where the five controls take you
Together, these five turn automation from a gamble into a service. Google’s own repair culture showed the payoff from the other side: once fixes were built to be safe — rehearsable, correctly targeted, batched, stoppable — they could be reapplied on a fifteen-minute schedule without anyone holding their breath. That is the real prize. Not “automation is fast” — everyone knows that — but “automation is fast and boring”, which is what lets it run while you sleep. When a job earns that reputation, the time it returns becomes genuine: fewer emergency wake-ups, fewer rollbacks, more hours for the work only humans do.
Condensed onto a single card, the whole discipline reads:
- Rehearse — but confirm the risky steps actually ran under test.
- Refuse empty lists — nothing selected means nothing done.
- Batch and canary — small waves, health checks between them.
- Budget failures — agree the stopping number before the rollout starts.
- Keep a brake — per job and global, drilled, pullable at 3 a.m. without permission.
If this plain-language tour left you wanting the wider map — how these controls connect to monitoring, careers and certifications — two pieces on this site fit well. The beginner guide to AIOps training, tools and careers covers the monitoring side that feeds good candidates for automation, and the roundup of compliance automation platforms shows the same governance instincts applied to audit work. For the career path itself, this DevOps skills walkthrough explains how these controls read on a CV. Start with the five controls, though. Everything else is detail layered on a foundation that fits in one notebook page: rehearse, refuse empties, batch, budget failures, and keep the brake in reach.
