Automation: Robust Scripts & Ansible

Intermediate45 min6 steps

What you'll learn

Explain what separates automation from 'a script I run sometimes' — unattended operation, and what that demands
Write robust, safe shell scripts with strict mode and traps
Automate real tasks: backups with rotation, cleanup, and self-healing health checks
Explain idempotency and convert a fragile do-X script into a converge-to-X one
Move from scripts to Ansible for multi-host, declarative configuration management

What automation really means: nobody is watching
Build the mental model — an unattended script needs discipline a typed command doesn't
When you run a command by hand, you are the safety system. You read the error, notice the typo, press Ctrl+C when something looks wrong. The whole point of automation is to remove you from the loop — the backup runs at 2 a.m., the cleanup fires from cron, the health check loops every five minutes. Which means the safety system is gone, and the script inherits its responsibilities.
That's the honest definition: automation is code that must behave correctly with nobody watching. And it changes what "working" means. A script that usually works is fine when you're there to catch the unusual; unattended, it's a time bomb. The failure mode isn't a crash — it's silence: the backup script that's been failing for three weeks while the log rotation happily deleted the evidence, discovered the day you need the backup.
So this lesson is really about three disciplines, in escalating order:
1. Fail loudly — a script that hits trouble should stop and say so, not shrug and continue (strict mode, traps).
2. Converge, don't repeat — a script that runs on a schedule will run many times, so running twice must be as safe as running once (idempotency).
3. Declare, don't script — when the same state must hold on many machines, stop writing how and start declaring what (Ansible).
Each discipline builds on the previous one. Master the first two in Bash and the third feels inevitable.
Why 'worse than no script'
A missing backup script leaves a visible gap someone eventually notices. A silently-failing one produces false confidence — the cron entry exists, the log file grows, everyone relaxes. The disasters that make postmortems aren't usually caused by absent automation; they're caused by automation everyone trusted and nobody verified. Loud failure is the antidote.
Check yourself
What's more dangerous in production: a backup script that doesn't exist, or one that fails silently — and why?
Strict mode and traps: scripts that fail loudly and clean up
Give every script the safety header and exit hooks it needs to run unattended
Bash's defaults were designed for interactive typing, and they are shockingly forgiving: a failed command? Keep going. An unset variable? Expand it to empty text and continue. For a human at a prompt that's convenient; for an unattended script it's how small errors compound into disasters. Three set options — collectively "strict mode" — flip the defaults to stop and complain:
Example — not runnable on your lab box
```
$ head -3 backup.sh
```
```
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
```
set -e
exit immediately if any command fails — no silent continuing
set -u
using an unset variable is an error — catches typos before they act
set -o pipefail
a failure anywhere in a pipeline fails the whole pipeline
The safety header for every serious script — make typing it a reflex. Each option closes a real hole. Without -e, a failed cd is ignored and the next line runs in the WRONG DIRECTORY. Without -u, a typo'd variable expands to empty — and rm -rf "$BACKUP_DR"/ (note the missing I) becomes rm -rf /: the most famous class of shell disaster, caused by a one-letter typo the shell didn't flag. Without pipefail, grep whatever /nonexistent | wc -l exits 0 because only the LAST command in a pipe sets the exit code by default.
Strict mode handles stopping. The second half is leaving the scene clean — temp files removed, lock released, a note about where things broke — no matter how the script exits. That's what trap does: it registers code to run on an event, and the EXIT event fires on success, failure, or Ctrl+C alike.
```
$ cat > cleanup-trap.sh <<'EOF'
#!/bin/bash
set -euo pipefail
TMP=$(mktemp)
cleanup() { rm -f "$TMP"; }
trap cleanup EXIT
trap 'echo "failed at line $LINENO" >&2' ERR

cp /etc/passwd "$TMP"
sort "$TMP" | head -3
EOF
cat cleanup-trap.sh
```
```
#!/bin/bash
set -euo pipefail
TMP=$(mktemp)
cleanup() { rm -f "$TMP"; }
trap cleanup EXIT
trap 'echo "failed at line $LINENO" >&2' ERR

cp /etc/passwd "$TMP"
sort "$TMP" | head -3
```
EXIT
trap event: fires on ANY exit — normal, error, or interrupt
ERR
trap event: fires when a command fails (pairs with set -e)
trap cleanup EXIT guarantees the temp file dies with the script — success, failure, or kill. trap '...' ERR fires on the failing line and reports WHERE (that $LINENO), turning "it broke somewhere" into "it broke at line 9." mktemp creates a unique temp file safely (never hand-roll /tmp/myfile — two simultaneous runs would fight over it). This four-line skeleton is the difference between a script and a robust script.
The classic -u victim: the cleanup path
The deadly pattern is a destructive command consuming an unverified variable: rm -rf "$STAGING_DIR/"* where $STAGING_DIR came from a config file that didn't load. Empty variable → rm -rf /*. set -u turns that into STAGING_DIR: unbound variable and a dead script — annoying for ten seconds, versus a rebuilt server. When you review any script, find the rm/dd/mkfs lines first and trace where their variables come from.
Check yourself
Which set option makes a script error out when it uses an undefined variable, and what disaster class does it prevent?
Automate real tasks: backup, rotate, self-heal
Build the two workhorse patterns every ops environment runs on
With the safety discipline in place, automate something worth automating. Two patterns cover a remarkable share of real operations work, and both are two-liners once you see their shape.
Backup-and-rotate is a pair of obligations: create today's archive, and delete the ancient ones. Skip the second half and the backup job slowly fills the disk — then dies, silently, weeks later (you know from the scheduling lesson why nobody notices). Creation and rotation belong in the same script so they can't drift apart.
Example — not runnable on your lab box
```
$ DATE=$(date +%F)
tar czf "/backup/home_$DATE.tar.gz" /home
find /backup -name 'home_*.tar.gz' -mtime +7 -delete
```
czf
(tar) create, gzip-compress, to file
-mtime +7
(find) modified more than 7 days ago
-delete
(find) delete the matches — always test the find WITHOUT it first
Timestamped archive in, week-old archives out — the disk usage stays flat forever. The same skeleton adapts by swapping the middle line: rsync -a --delete for an incremental mirror, mysqldump | gzip for a database. The rotation line is doing quiet load-bearing work: -mtime +7 selects files older than 7 days, -delete removes them.
Self-healing is the second pattern: check a condition, repair it if it's wrong, and say what you did. Your systemctl knowledge supplies both halves — is-active --quiet is a silent health probe whose exit code is the answer, and restart is the repair.
```
$ for s in nginx sshd chronyd; do
  systemctl is-active --quiet "$s" || { echo "$(date -Is) restarting $s" >> /var/log/selfheal.log; systemctl restart "$s"; }
done
```
is-active --quiet
no output, just the exit code — built for scripting
-Is
(date) ISO timestamp with seconds — sortable, unambiguous log entries
For each critical service: probe, and on failure log-then-restart. The || chains the repair to a failed probe — exit codes as truth, exactly as you learned in conditionals. Run it from cron every 5 minutes and transient crashes fix themselves before a human notices. The log line matters: a self-healing system that doesn't record its interventions hides a service that's crash-looping every 5 minutes — the repair works, but the underlying disease goes untreated.
Schedule it — and prevent overlap
Automation only helps if it runs: install via cron (0 2 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1 — with the environment and redirect habits from the scheduling lesson) or a systemd timer with Persistent=true. One new hazard appears at this point: a job slower than its schedule interval starts overlapping itself (two backups writing at once). Guard with flock: flock -n /run/backup.lock /opt/scripts/backup.sh — the second copy finds the lock taken and exits instead of colliding.
Check yourself
Your nightly backup job has run fine for months, then the disk fills and everything breaks. Which half of the backup pattern was missing?
Idempotency — the key idea
Make automation safe to run again and again
Here's the concept that separates fragile scripts from real automation, and it has an unfriendly name for a friendly idea. An operation is idempotent if running it twice (or twenty times) leaves the system in the same state as running it once.
Why is this the property? Because scheduled scripts will run repeatedly — that's their job — and sooner or later one will also run at a weird time: re-run after a half-finished failure, replayed after a restore, triggered twice by an overlapping schedule. A script that assumes "this is the first time" breaks in all of those cases. A script that checks the current state and only changes what's wrong handles all of them for free.
Compare the mindsets: "do X" (imperative — appends, adds, creates blindly) versus "make sure X is true" (convergent — checks, then acts only if needed). The second phrasing is how reliable systems are built.
```
$ grep -q 'db01' /etc/hosts || echo '10.0.0.5 db01' >> /etc/hosts
cat /etc/hosts | grep db01
```
```
10.0.0.5 db01
```
q
(grep) quiet — exit code only, which is all the || needs
The idempotent append: check (grep -q), and only add if absent (||). Run it once, twice, a hundred times — exactly one db01 line exists. The naive version, echo >> without the guard, stacks a duplicate on every run: three runs, three lines, and eventually a confused resolver. The || pattern is exit-codes-as-truth again — your conditionals lesson, doing production work.
Append-without-checking is the classic bug
It has many faces: >> into a config file, useradd bob (errors on the second run because bob exists), iptables -A (stacks a duplicate rule per run), git clone (fails — directory exists). Before shipping any scheduled script, interrogate every line with one question: *what happens when this runs twice?* If the answer isn't "the same thing as once," wrap it in a check: id bob &>/dev/null || useradd bob.
Check yourself
Make this line idempotent: useradd deploy
From scripts to Ansible (infrastructure as code)
Scale automation across many hosts, declaratively
Now scale the idea. Your idempotent script converges one host. What about thirty web servers? Copying the script around and SSH-looping works — briefly — but you're now maintaining bespoke check-then-act logic for every kind of change, on every host, forever.
Ansible is what it looks like when idempotency becomes the platform instead of your responsibility. You write YAML that declares the desired state — "nginx is installed; nginx is running and enabled" — and Ansible's modules do the check-then-act dance for you, over SSH, across every host in an inventory. It's agentless (nothing to install on targets beyond Python) and the YAML lives in git, which earns the name infrastructure as code: the configuration of your fleet is versioned, reviewable, and reproducible, not folklore in an admin's head.
Example — not runnable on your lab box
```
$ cat site.yml
```
```
- hosts: webservers
  become: true
  tasks:
    - name: nginx is installed
      ansible.builtin.dnf:
        name: nginx
        state: present
    - name: nginx is running and enabled
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true
```
A playbook. Each task names a module (dnf, service) and a desired state (present, started) — note the language: not "install nginx" but "nginx IS installed." Run it with ansible-playbook -i inventory site.yml. The first run reports changed where it acted; run it again immediately and every task reports ok — already correct, nothing done. That ok is idempotency as a platform guarantee: the same property you hand-built with grep-guards, now free on every task. (Also recognize state: started + enabled: true — it's enable --now, declaratively.)
```
$ ansible -i inventory webservers -m ping
ansible -i inventory webservers -m command -a 'uptime'
```
```
web01 | SUCCESS => {"changed": false, "ping": "pong"}
web02 | SUCCESS => {"changed": false, "ping": "pong"}
web01 | CHANGED | rc=0 >>
 14:22:01 up 41 days,  2:11,  0 users,  load average: 0.08, 0.11, 0.09
```
-i inventory
the hosts file listing your fleet, grouped ([webservers], [dbs])
-m MODULE
which module to run (ping, command, dnf, service, copy…)
-a ARGS
arguments for the module
Ad-hoc mode: one module, every host, right now — no playbook needed. -m ping verifies Ansible can reach and run on each target (the first thing to try on a new inventory); -m command -a runs a one-off across the fleet. The mental upgrade: you stopped operating on A machine and started operating on a NAMED SET of machines.
Check yourself
You run the same playbook twice in a row. What does the second run report, and why is that the whole point?
Now do it — build and torture-test an idempotent script
Prove strict mode, traps, and convergence with your own hands
Three experiments, each proving one discipline: watch strict mode kill a typo, watch a trap clean up after a crash, and watch an idempotent script converge. Automation habits form when you've seen the failure each habit prevents.
```
$ cat > /tmp/strict.sh <<'EOF'
#!/bin/bash
set -euo pipefail
TARGET_DIR=/tmp/auto-lab
echo "cleaning $TARGT_DIR"
EOF
bash /tmp/strict.sh; echo "exit: $?"
```
A deliberate typo — $TARGT_DIR — in a line that (in a real script) would precede an rm. With set -u, the script dies instantly with 'TARGT_DIR: unbound variable', exit 1. Now delete the -u from the set line and re-run: it happily prints 'cleaning ' — an empty expansion, the seed of the rm -rf /* disaster. One flag, one disaster class, witnessed.
```
$ cat > /tmp/trapdemo.sh <<'EOF'
#!/bin/bash
set -euo pipefail
TMP=$(mktemp)
trap 'rm -f "$TMP"; echo "cleaned up $TMP" >&2' EXIT
echo "working in $TMP"
false   # simulate a mid-script failure
echo "never reached"
EOF
bash /tmp/trapdemo.sh; ls /tmp/tmp.* 2>/dev/null || echo "no temp files left behind"
```
The script fails halfway (that false) — but the EXIT trap still fires: you see 'cleaned up', and the ls confirms no orphaned temp file. Comment out the trap line and re-run: the temp file survives as litter. Multiply that by a cron job running every 5 minutes for a month, and you understand why traps are non-negotiable in scheduled scripts.
```
$ cat > /tmp/idem.sh <<'EOF'
#!/bin/bash
set -euo pipefail
grep -q demo /tmp/hosts.demo 2>/dev/null || echo "10.0.0.5 demo" >> /tmp/hosts.demo
id labuser &>/dev/null || echo "would create labuser"
EOF
bash /tmp/idem.sh; bash /tmp/idem.sh; bash /tmp/idem.sh
wc -l /tmp/hosts.demo
```
Three runs, then count: exactly 1 line. Both patterns converge — the grep-guard append and the id-guard user creation. Now break it on purpose: remove the 'grep -q ... ||' guard, run three more times, and count again (4 lines and climbing). The difference between those two counts is the entire concept of idempotency.
```
$ command -v ansible-playbook >/dev/null && ansible --version | head -1 || echo 'ansible not installed here — the concepts above are what transfer'
```
Check whether this sandbox carries Ansible. Where it exists, everything from the playbook step runs for real; where it doesn't, remember that Ansible is 'your idempotent guards, as a managed platform' — the Bash patterns you just proved are the concept the tool industrializes.

Recap

Automation is code that runs with nobody watching — so it must fail LOUDLY (silent failure breeds false confidence, the most dangerous state in operations).
Every serious script starts with set -euo pipefail (-e stop on failure, -u unset variables are errors, pipefail whole-pipe truth) and uses trap cleanup EXIT so temp files and locks never outlive the run.
The workhorse patterns: backup WITH rotation in the same script (tar + find -mtime +N -delete), and self-healing checks (is-active --quiet || log-and-restart) — scheduled via cron/timers, overlap-guarded with flock.
Idempotency is THE property of reliable automation: ask 'what happens if this runs twice?' of every line; convert do-X into make-sure-X (grep -q || append, id || useradd).
Ansible industrializes those guards: declarative YAML (state: present/started) over SSH across an inventory, idempotent by design — second run reports ok. Infrastructure as code = your fleet's config, versioned in git.

Concept → tool
Goal	How
Fail loudly	`set -euo pipefail`
Clean up on any exit	`trap cleanup EXIT` · `trap '...' ERR`
Backup + rotate	`tar/rsync` + `find -mtime +N -delete`
Self-heal a service	`is-active --quiet "$s" \|\| restart` (+ log it)
Run on schedule, once at a time	cron / timer (`Persistent=true`) + `flock`
Idempotent change	`grep -q X file \|\| echo X >> file` · `id u \|\| useradd u`
Many hosts, declarative	Ansible playbook (`state: present`), inventory
Quick multi-host check	`ansible group -m ping`

Automation: Robust Scripts & Ansible

What automation really means: nobody is watching

Strict mode and traps: scripts that fail loudly and clean up

Automate real tasks: backup, rotate, self-heal

Idempotency — the key idea

From scripts to Ansible (infrastructure as code)

Now do it — build and torture-test an idempotent script