Text Processing: grep, awk & sed

Intermediate45 min6 steps

What you'll learn

Pick the right tool by the verb: grep to find, sed to change, awk for fields and math
Search with grep and basic regular expressions (anchors, classes, alternation, context)
Transform text with sed substitution and deletion — safely, with the preview-then--i discipline
Extract and compute over columns with awk's pattern { action } model
Chain them into the filter → extract → count → rank pipeline that answers most log questions

Three tools, three jobs
Know which of grep/sed/awk to reach for before typing anything
Strip away the ceremony and most Linux administration is reading and reshaping text: logs, configs, the output of other commands. "Which IPs are attacking us?" is a text question. "Turn this setting off in 40 configs" is a text question. Unix's answer is three small tools with a clean division of labor — picture a text assembly line:
- grep — the filter: only lines matching a pattern get through. Its verb is find.
- sed — the stream editor: rewrites lines as they flow past. Its verb is change.
- awk — the field inspector: splits each line into columns, and can compute. Its verbs are extract and calculate.
Each reads text line by line and writes text — which is exactly why they compose: grep's output pipes into awk, awk's into sort. One tool, one verb, chained.
Why matching tool-to-verb is the actual skill
The time-sink isn't ignorance of flags — it's using the wrong tool: tortured grep-with-cut gymnastics to extract a column (that's awk's one job), or a bash loop to substitute text across files (that's sed, in one line). Before typing, name the verb — find? change? extract? count? — and the tool picks itself. That habit is worth more than any individual flag in this lesson.
Check yourself
Name the tool for each verb: (a) show me every line with 'timeout' in it, (b) replace every 'http:' with 'https:', (c) sum the 5th column.
grep — find the lines
Search files with patterns and regex, and read around the matches
grep's contract is minimal: give it a pattern and input, and it prints the lines that match. Everything else is flags shaping that contract — case-blind matching, counting instead of printing, inverting the match. The inversion one deserves a highlight before you meet it: -v prints lines that do NOT match, and it's how you remove noise — which turns out to be half of real-world grepping.
Example — not runnable on your lab box
```
$ grep -in 'error' /var/log/messages | head -2
grep -c 'Failed' /var/log/secure
grep -v '^#' /etc/ssh/sshd_config | head -3
```
```
112:Jul 11 03:14:22 web01 app[812]: ERROR: connection refused
245:Jul 11 06:02:10 web01 kernel: I/O error on sda2
17
Include /etc/ssh/sshd_config.d/*.conf
PermitRootLogin no
PasswordAuthentication no
```
-i
case-insensitive match
-n
prefix each match with its line number
-c
print a count of matching lines instead of the lines
-v
invert — print lines that do NOT match (noise removal)
-r
recurse through a directory of files
Three everyday shapes. -i catches ERROR and error alike, -n prefixes the line number (jump straight there in an editor). -c counts instead of printing — 17 failed logins, one number, script-ready. And the classic: -v '^#' drops comment lines, so a 200-line config reads as the ten lines that actually do something — this is how experienced admins read ANY config for the first time.
Plain text patterns get you far; regular expressions get you the rest. A regex is a pattern language: ^ anchors to line start, $ to line end, . is any one character, [0-9] a character class, [^0-9] its negation. Add -E (extended regex) and you get alternation |, +, ?, and () groups without backslash clutter:
Runnable in the sa04 room — that room’s setup plants the files it reads
```
$ grep -E 'error|warning|critical' app.log | head -2
grep '^root' /etc/passwd
grep 'bash$' /etc/passwd | head -2
```
```
14:31:02 warning: cache miss rate high
14:33:47 error: upstream timeout
root:x:0:0:root:/root:/bin/bash
root:x:0:0:root:/root:/bin/bash
alice:x:1000:1000::/home/alice:/bin/bash
```
-E
extended regex: | + ? () work unescaped
^ / $
anchor to line start / end — precision, not decoration
-A3 / -B3 / -C3
context lines after / before / around each match
-o
print only the matched text, not the whole line
Alternation matches any of three severities in one pass. The anchors do precision work: ^root means 'root at line START' (the root account itself — not chroot, not fakeroot buried mid-line), and bash$ means 'bash at line END' (accounts whose SHELL is bash, not lines merely mentioning it). Anchors are the difference between a pattern that works and one that works BY LUCK. Bonus for log reading: -A3/-B3/-C3 print 3 lines of context After/Before/around each match — the error line alone rarely tells the story.
Check yourself
You want the accounts that can log in — every /etc/passwd line NOT ending in nologin or false. Build the grep.
sed — transform the stream
Substitute and delete safely, with the preview-then--i discipline
grep decides which lines survive; sed changes what lines say. It's a stream editor: text flows through, sed applies editing commands to each line in flight, and the result flows out — the file itself untouched unless you explicitly say otherwise. That untouched-by-default design is a safety feature you'll lean on in a moment.
One editing command does 90% of sed's real work: `s/pattern/replacement/` — substitute. And it hides the most common sed surprise: by default it replaces only the first match on each line.
```
$ echo 'red fish red fish' | sed 's/red/blue/'
echo 'red fish red fish' | sed 's/red/blue/g'
```
```
blue fish red fish
blue fish blue fish
```
Same input, one-character difference, different results — the first-vs-all distinction demonstrated in two lines. Without g, only the first 'red' per line changes (the second survives!); with g (global), all of them. Nearly every real substitution wants g; forgetting it produces the eerie half-changed file where each line was fixed exactly once. Add i for case-insensitive matching: s/red/blue/gi.
Example — not runnable on your lab box
```
$ sed 's/Enabled/Disabled/g' /etc/myapp.conf | grep -n Disabled
sed -i.bak 's/Enabled/Disabled/g' /etc/myapp.conf
sed -n '5,10p' /etc/myapp.conf
```
```
3:FeatureX=Disabled
9:Telemetry=Disabled
MaxClients=64
Timeout=30
LogLevel=info
Telemetry=Disabled
KeepAlive=on
Port=8443
```
(no -i)
preview mode: result to stdout, file untouched — always first
-i.bak
edit in place, saving the original with a .bak suffix
/pat/d
delete matching lines
-n '5,10p'
quiet mode + print — output only the given line range
The professional rhythm, in order: PREVIEW first (no -i — output goes to stdout, the file is untouched; pipe through grep to eyeball exactly what changed), then commit with -i.bak (edit in place, but stash the original as .conf.bak). Two more daily moves: /pattern/d deletes matching lines ('/^$/d; /^#/d' strips blanks and comments), and -n '5,10p' prints only lines 5–10 — sed as a precision extractor.
Every sed command takes an optional address in front of it, naming which lines it applies to. Leave it off and the command runs on every line. Give it /pattern/ and it runs on the lines matching that pattern (/^#/d — delete comments). Give it two addresses separated by a comma and it runs on a span: 2,5 is lines 2 through 5, and — the one worth memorising — `/FROM/,/TO/` is every line from the first match of `FROM` through the next match of `TO`, inclusive. That last form is how you operate on a fenced block whose line numbers you don't know in advance: a stanza in a config, a chunk between markers, one host's entry in an inventory.
```
$ printf 'keep 1\nSTART\ndrop a\ndrop b\nEND\nkeep 2\n' | sed '/START/d; /END/d'
printf 'keep 1\nSTART\ndrop a\ndrop b\nEND\nkeep 2\n' | sed '/START/,/END/d'
```
```
keep 1
drop a
drop b
keep 2
keep 1
keep 2
```
/FROM/,/TO/
range address: the first FROM match through the next TO match, inclusive
d
delete the addressed lines
The same input, two addressing styles, and the difference is the whole point. The first line uses two SINGLE addresses — it deletes the lines that literally say START and END, and the block they were fencing survives. That is the wrong answer, and a convincing one, because the markers really did disappear. The second uses ONE range address, /START/,/END/, which selects the whole span including both markers, so d removes the block entirely. If you find yourself deleting a fence line by line, you want a range.
```
$ printf 'keep 1\nSTART\ndrop a\ndrop b\nEND\nkeep 2\n' | sed -n '/START/,/END/p'
```
```
START
drop a
drop b
END
```
-n
suppress the automatic print, so only p emits
p
print the addressed lines
The same range, inverted: -n silences the automatic print and p prints only the addressed lines, so you EXTRACT the block instead of removing it. One address, two opposite jobs, depending on whether you pair it with d or with -n ... p. This is the move for pulling one stanza out of a long config without counting lines — sed -n '/^\[database\]/,/^\[/p' grabs the [database] section. Mind the closing address there: the range ends at the NEXT section header, so that header comes along with it; pipe through sed '$d' if you want the stanza alone.
The other half of sed's substitution power is remembering part of what it matched. Wrap a piece of the pattern in a group and sed stores it; \1 in the replacement means "whatever group 1 matched", \2 group 2, and so on. These are backreferences, and they turn s/// from find-and-replace into genuine reformatting: reorder fields, wrap a value in something, keep a piece you can't spell out in advance.
One syntax trap costs everyone an afternoon once. In sed's default basic regex the grouping parentheses must be escaped —  — because a bare ( is a literal. Add `-E` for extended regex and they're bare ( ), the same syntax grep -E already uses. Pick -E and stay there.
```
$ printf 'alice:1000\nbob:1001\n' | sed -E 's/^([a-z]+):([0-9]+)$/\2 belongs to \1/'
```
```
1000 belongs to alice
1001 belongs to bob
```
-E
extended regex: groups are ( ) instead of 
([a-z]+)
capture group 1 — recalled as \1 in the replacement
\2
backreference to the second captured group
Two groups captured — the name and the UID — then written back in the opposite order with text between them. Nothing in the replacement names 'alice' or '1000': the pattern describes their SHAPE and the backreferences carry whatever was actually there, so the same one-liner reformats a file of ten thousand rows. The same rule in basic regex needs the parens escaped — sed 's/^$[a-z]$:$[0-9]$$/\2 belongs to \1/' — identical result, four more backslashes, which is the argument for -E.
sed -i has no undo
sed -i rewrites the file the instant you press Enter — a mistyped pattern silently corrupts a config with no ceremony and no recycle bin. The discipline is always the same two-step: run it without `-i` first and read the output, then re-run with `-i.bak` so the original survives until you've verified the result (and diff file file.bak shows precisely what changed). Editing a production config with bare -i, untested, is how five-minute tasks become incident reports.
Check yourself
sed 's/DEBUG/INFO/' ran across a log config, but some lines still contain DEBUG. The pattern is correct. What happened?
Check yourself
You need to swap two comma-separated columns in a file, but you don't know the values in advance. Which sed feature does that, and what does the replacement side look like?
awk — fields and computation
Pull columns and run totals with pattern { action }
grep sees lines; awk sees columns. It splits every line into fields — $1, $2, up to $NF (the last, however many there are) — on whitespace by default or on whatever -F names. That alone replaces most cut-and-paste column extraction. But awk's real identity is its program shape, worth learning as a sentence:
`pattern { action }` — for each line matching the pattern, do the action. Omit the pattern → the action runs on every line. Omit the action → matching lines print. Everything awk does is that sentence with different clauses.
```
$ awk -F: '$3 >= 1000 {print $1, $3}' /etc/passwd
```
```
alice 1000
bob 1001
deploy 1002
```
-F:
field separator — colon for /etc/passwd, ',' for CSV
$1 … $NF
fields by position; $0 = whole line, $NF = last field
pattern { action }
the core sentence: on matching lines, do this
The sentence in action, on real data: split /etc/passwd on colons (-F:), and for lines where field 3 (the UID) is at least 1000 — that's the pattern, a numeric comparison no grep could do — print fields 1 and 3. One line, and you have every regular user with their UID. Note what the pattern is comparing: not text, a NUMBER. That's the boundary grep can't cross and the reason awk exists.
Runnable in the sa03 room — that room’s setup plants the files it reads
```
$ awk '{sum += $3} END {print "total:", sum}' data.txt
awk '$2 ~ /^192\.168/ {print $1}' access.log | head -2
```
```
total: 4718
GET /index.html
GET /login
```
sum += $3 … END{}
accumulate across lines; END runs once after the last
$2 ~ /regex/
regex against a specific field — precision grep can't offer
NR / NF
current line number / number of fields on this line
Two abilities nothing else in the pipeline has. First: STATE across lines — sum accumulates as every line flows past, and the END block (which runs once, after the last line) prints the total; a column summed with no spreadsheet in sight. Second: per-FIELD regex — $2 ~ /^192\.168/ tests the pattern against field 2 only, so a 192.168 address in some other column can't cause a false match. BEGIN {} is END's mirror (runs before line one — headers, separators). NR (current line number) and NF (field count) round out the everyday variables.
sum += $3 keeps one running number. Change one character and awk keeps a running number per key: n[$1]++ is an associative array — a table whose subscripts are arbitrary strings, not integers. awk creates each entry the moment you touch it, so counting distinct things takes one expression, no setup, no sorting, and no second pass. Then for (k in arr) walks the keys in the END block to print what you accumulated.
This is the tool for every "how many per X" question: requests per IP, failures per user, files per extension. It is also the point where awk stops being a column-cutter and becomes a small program.
```
$ printf 'alice ok\nbob fail\nalice fail\nalice ok\n' | awk '{n[$1]++} END {for (u in n) print u, n[u]}' | sort
```
```
alice 3
bob 1
```
n[$1]++
associative array: count occurrences keyed by field 1
for (u in n)
iterate the keys in END — order is undefined, pipe to sort if you need one
n[$1]++ says 'increment the counter filed under this line's first field' — awk creates the entry on first sight and remembers it for the whole run. The END block then walks every key that was created and prints it with its count. Note the trailing sort: for (k in arr) hands back keys in an UNSPECIFIED order, so if you want them ordered, order them yourself. That surprise is the one thing about awk arrays worth writing on your hand.
Check yourself
df --output=used,target prints kilobytes used per mount. Sketch the awk that totals column 1.
Check yourself
awk '{n[$1]++} END {for (k in n) print k, n[k]}' printed its lines in a jumbled order. Is that a bug?
Combine them with pipes
Chain filter → extract → count → rank into the one-liner that answers log questions
Now the payoff for one-tool-one-verb: composition. Real questions are multi-verb — "who's brute-forcing us?" decomposes into find the failed logins (grep), extract the IP (awk), count per IP, rank. Each verb is a pipeline stage, and the result is the single most useful pattern in sysadmin text processing — filter → extract → count → rank:
Example — not runnable on your lab box
```
$ grep 'Failed password' /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -3
```
```
  312 203.0.113.42
   77 198.51.100.9
   12 192.0.2.200
```
Read left to right, one verb per stage: grep keeps only failed-password lines → awk pulls the IP (it sits 3 fields before the end on these lines, hence $(NF-3) — counting from the END survives lines of varying length) → sort groups duplicates together → uniq -c collapses them WITH a count → sort -rn ranks numerically, biggest first → head takes the top. Verdict: 203.0.113.42 has 312 failures — there's your brute-forcer, and your fail2ban candidate. Swap the grep pattern and the field, and this same chain ranks URLs, status codes, usernames — anything.
uniq without sort silently undercounts
uniq only collapses adjacent duplicates — it was built for sorted streams. Feed it unsorted data and the same IP scattered through the log counts as many separate runs of 1, and your "top attacker" ranking is quietly, confidently wrong. It's always the pair: `sort | uniq -c`. (And when you catch yourself building grep X | cut -d' ' -f3-style chains, remember awk '/X/{print $3}' is the filter and the extractor in one stage.)
Check yourself
Adapt the chain: top 5 most-requested URLs in access.log, where the URL is field 7.
Now do it — interrogate real files
Run each verb, then the full chain, against the sandbox's live /etc/passwd
Your sandbox has real files with real structure — /etc/passwd is the classic practice target: colon-separated, seven fields, guaranteed present. Run each tool's core move against it, then the full chain. All read-only; nothing here changes the system.
```
$ grep -c bash /etc/passwd; grep -vE '(nologin|false)$' /etc/passwd
```
grep both ways: count the bash mentions, then the login-capable accounts — the -v + alternation + $-anchor pattern from the knowledge check, now against live data. Compare the two outputs: is the count what the second list implies?
```
$ awk -F: '$3 >= 1000 {print $1, $3, $7}' /etc/passwd
```
The awk sentence on real input: UID ≥ 1000 with name, UID, and shell. Look closely at what comes back on this box — alongside the real login account you'll also see nobody at 65534, and that is the lesson. UID ≥ 1000 is the convention for human accounts, not a guarantee: nobody is the kernel's deliberately-powerless placeholder, pinned at the top of the range precisely so it can never collide with a real user. So the honest filter for humans is a RANGE — $3 >= 1000 && $3 < 65534 — and flipping to $3 < 1000 shows you most of the system accounts but not nobody. Run both and see the split for yourself: a pattern that's right about your data beats one that's merely close.
```
$ awk -F: '{print $7}' /etc/passwd | sort | uniq -c | sort -rn
```
The filter→extract→count→rank chain answering 'what shells does this box use?' — likely a pile of nologin, a few bash. Now re-aim it: swap $7 for $4 and you're ranking primary groups. One skeleton, endless questions.
```
$ sed 's/bash/BASH/g' /etc/passwd | grep BASH | head -3
```
sed in safe preview mode — no -i, so /etc/passwd is untouched while you watch the substitution work in the stream. That preview-first reflex, practiced here where stakes are zero, is exactly what you'll do before every real -i.bak on a config that matters.
```
$ awk -F: '{shells[$7]++} END {for (s in shells) print shells[s], s}' /etc/passwd | sort -rn
```
The same 'what shells does this box use?' ranking as the chain two blocks up, but awk does the counting itself — one pass, no sort | uniq -c. Run both and compare: identical numbers, and the array version keeps working when the thing you want to count isn't a whole field. Whichever you reach for, this is the shape behind every requests-per-IP and failures-per-user question you'll be asked.
```
$ sed -n '/^root:/,/^sys:/p' /etc/passwd
sed -E 's/^([a-z_-]+):x:([0-9]+):.*/\2 is \1/' /etc/passwd | head -3
```
Range addressing and backreferences against a real file. The first pulls the span from the root entry through the sys entry — a block whose line numbers you never had to know, and which still works on a box where those accounts sit somewhere else. The second captures the name and the UID out of each passwd line and writes them back reordered, proving the pattern describes shape rather than specific values. Both are read-only; /etc/passwd is untouched.

Recap

Name the verb, and the tool picks itself: grep finds lines, sed changes text, awk extracts fields and computes. Multi-verb questions become pipelines, not clever single commands.
grep: -i/-n/-c everyday, -v for noise removal (grep -v '^#' to read any config), -E for alternation, ^/$ anchors for precision-not-luck, -A/-B/-C for context around a match.
sed: s/old/new/ replaces the FIRST match per line — g makes it all (the most-hit surprise). The discipline: preview without -i, commit with -i.bak, diff against the .bak.
sed addresses pick the lines: /pattern/ for matches, and /FROM/,/TO/ for a whole fenced block, inclusive — pair it with d to delete the block or -n ... p to extract it. Deleting the two marker lines individually leaves the block behind, which is the convincing wrong answer.
Capture groups make s/// reformat rather than merely replace: group part of the pattern, recall it as \1 / \2 in the replacement. Use -E so the parens are bare ( ) instead of .
awk arrays count per key: n[$1]++ builds the table as it reads and END walks it with for (k in n) — that iteration order is undefined, so sort the output.
awk is one sentence — pattern { action } — with fields ($1…$NF, -F separator), numeric patterns grep can't do, per-field regex ($2 ~ /re/), and state across lines (sum += … END).
The chain that answers most log questions: filter → extract → sort | uniq -c → sort -rn | head. And it's ALWAYS sort before uniq — unsorted uniq silently undercounts.

Tool → core moves
Tool	Verb	Example
grep	find / de-noise	`grep -iv '^#' file`
grep -E	alternation + anchors	`grep -E '(err\|warn)$' log`
sed	substitute (preview → -i.bak)	`sed 's/old/new/g' f` → `sed -i.bak …`
sed	delete / slice	`sed '/^$/d' f` · `sed -n '5,10p' f`
awk	columns	`awk -F: '{print $1,$7}' /etc/passwd`
awk	compute	`awk '{s+=$3} END{print s}' f`
pipeline	rank by count	`… \| sort \| uniq -c \| sort -rn \| head`

Text Processing: grep, awk & sed

Three tools, three jobs

grep — find the lines

sed — transform the stream

awk — fields and computation

Combine them with pipes

Now do it — interrogate real files