The Core-Utility Tour: Composing ls, grep, cut, sort & Friends

Beginner40 min5 steps

What you'll learn

Know which everyday utility is the right tool for search, transform, and summarize jobs
Chain small tools with pipes (|) so each does one thing and passes its output to the next
Build a real analysis pipeline: extract a field, sort it, count it, rank it
Use awk's built-ins ($NF, NF, NR, length, END) and sed's -n and -i for the jobs cut and a plain s/// can't do
Recognize the composition patterns that the One-Liner Dojo grades for efficiency

The Unix philosophy: small tools, one pipe
See why chaining tiny single-purpose tools beats one giant command — and picture the stream flowing through them.
Most operating systems hand you big, do-everything applications. Linux does the opposite: it gives you a drawer full of small, sharp tools — each one does exactly one thing and does it well — plus a way to connect them. That connector is the pipe, written |. A pipe takes whatever the tool on its left prints and hands it, unchanged, to the tool on its right as input.
Here's the mental model to keep for the rest of this lesson: think of your data as water flowing left to right through a series of filters. Each tool sits in the stream, lets through what you want, reshapes it, or measures it — then passes the result downstream. You don't look for one command that does the whole job. You look for the chain of little steps that, snapped together, produce the answer:
`
extract a column | sort it | count duplicates | keep the top few
`
That habit — decompose the task into a chain of one-step transforms — is the whole game. It's also exactly what the One-Liner Dojo grades you on later: the tightest correct chain wins.
```
$ cut -d: -f1 /etc/passwd | sort | head -3
```
```
_apt
backup
bin
```
-d:
use ':' as the field delimiter (the character that separates columns)
-f1
keep field number 1 (the first column)
Read it left to right as three stages of the stream. cut pulls out field 1 (the username) from every line of the password file, splitting on the : character. sort puts those usernames in order. head keeps only the first three. Each tool is completely blind to the others — it just reads from its input and writes to its output. The pipes do all the wiring.
Every tool speaks the same two channels: stdin and stdout
The reason any tool can pipe into any other is that they all agree on a common language: read text from standard input (stdin), write text to standard output (stdout). A pipe simply connects one tool's stdout to the next tool's stdin. Because the interface is universal, the tools compose in any order that makes sense — that's what turns a dozen little utilities into billions of possible pipelines.
Why compose instead of memorizing a mega-command
There is no single command for 'the top 3 IPs in this log' — but there's a chain for it, and the same building blocks (sort, uniq -c, head) solve a hundred other problems. Learn the pieces and the pipe, and you can assemble an answer to tasks nobody ever wrote a dedicated command for. Memorizing commands doesn't scale; composing them does.
Cluster 1 — Search & filter (find the rows you care about)
Narrow a stream down to just the lines and fields you want, throwing the rest away.
The first family of tools reduces a stream — they let some lines through and drop the rest. Think of them as sieves. The four you'll reach for daily:
- `grep PATTERN` — keep only lines that match a pattern.
- `head -n N` / `tail -n N` — keep the first / last N lines.
- `cut -d X -f N` — keep field N from each line, splitting on delimiter X.
- `find` — locate files (from the previous lesson); pipe its paths onward with xargs.
Let's actually use them, because a bullet list teaches nothing until you see the output.
Example — not runnable on your lab box
```
$ grep -i error /var/log/app.log | tail -n 2
```
```
2026-07-01 08:12 ERROR db timeout
2026-07-01 08:19 Error retry failed
```
-i
case-insensitive match — Error, ERROR, and error all count
-n 2 (tail)
keep the last 2 lines of the input
Two stages. grep -i error keeps only the lines mentioning 'error' in any capitalization, dropping the thousands of normal lines. tail -n 2 then keeps the last two of those. The order matters: filter first to shrink the stream, then slice — a pattern you'll use constantly.
grep has three more flags worth committing to muscle memory, because they turn it from a search tool into a mini-analysis tool: -v inverts the match (keep lines that DON'T match), -c counts matches instead of printing them, and -n prefixes each hit with its line number.
Example — not runnable on your lab box
```
$ grep -c ssh /var/log/auth.log
```
```
147
```
-c
print the count of matching lines rather than the lines themselves
grep -c prints just the number of matching lines — 147 lines mention ssh. Notice you did NOT write grep ssh … | wc -l: grep can count itself, so piping into wc -l is a redundant stage. Reaching for the built-in flag instead of an extra tool is exactly the efficiency habit the Dojo rewards.
Check yourself
You want every line of a config file EXCEPT the comments (lines starting with #). Which single grep flag does that?
Cluster 2 — Transform (reshape each line)
Change the content of a stream: substitute text, translate characters, pull fields.
The second family rewrites the stream, line by line — same number of lines (usually), different content. Three tools, in increasing order of power:
- `tr SET1 SET2` — translate or delete characters. The bluntest tool; character-level only.
- `sed 's/old/new/'` — a stream editor: find-and-replace, delete lines, edit in place. Its everyday job is substitution.
- `awk '{print $2}'` — field-aware processing: it splits each line into $1 $2 $3 … and lets you print, compute, or test per field. The heavy machinery for when the others aren't enough.
```
$ echo 'user=alice role=admin' | sed 's/admin/AUDITOR/'
```
```
user=alice role=AUDITOR
```
sed's s/old/new/ means 'substitute old with new' on each line. Here it swaps admin for AUDITOR. Sed reads from stdin (piped in from echo) and writes the transformed line to stdout — a pure filter you can drop into the middle of any pipeline. Add a trailing g (s/old/new/g) to replace every match on a line, not just the first.
```
$ echo 'alice bob carol' | tr ' ' '\n' | sort
```
```
alice
bob
carol
```
SET1 → SET2
tr maps each character in SET1 to the matching one in SET2 (here space → newline)
tr ' ' '\n' translates every space character into a newline — turning one line of three names into three lines of one name each — and then sort orders them. tr is the right, lightweight tool for pure character swaps; only reach past it to sed or awk when you need patterns or fields.
cut vs awk — don't overreach
For simple fixed-delimiter extraction, cut -d, -f2 is clearer and faster than awk -F, '{print $2}' — they do the same thing, but cut says what it means with less ceremony. Reach for awk only when you actually need its power: math, conditions, several fields at once, or uneven spacing that cut can't handle. Grabbing a bulldozer to plant a seed is a code smell — and yes, the Dojo scores for it.
So when is awk the light tool? When the answer depends on something only awk knows about each line. It hands you a few built-in names inside the { }:
- `$1`, `$2`, … are the fields, and `$NF` is the last field, however many the line has. -F, sets the separator, like cut -d,.
- `NF` is the number of fields on the current line.
- `NR` is the current line (record) number, so a bare NR==1 in front of the braces means 'only on line 1'.
- `length` is the current line's length in characters.
sed has two everyday moves beyond s/// too. `sed -n '5p'` prints only line 5: -n switches off sed's habit of printing every line, and the 5p address prints just the line you named. `sed -i` edits the file in place instead of printing the result. Preview without -i first: it rewrites the file the moment you press Enter.
```
$ printf 'a,b,c\nd,e\nf,g,h,i\n' | awk -F, '{print NF, $NF}'
```
```
3 c
2 e
4 i
```
-F,
split fields on commas instead of whitespace
NF / $NF
the number of fields / the last field itself
Each line has a different number of comma-separated fields, which is exactly where cut gets stuck: there is no single -f for 'the last one'. awk prints NF (how many fields this line has) and $NF (the field at that position, the last one), whatever the count.
```
$ printf 'short\na much longer line\nmid line\n' | awk '{print length, $0}'
```
```
5 short
18 a much longer line
8 mid line
```
length is the line's character count and $0 is the whole line, so every line comes out labelled with its length. Pipe that into sort -rn | head -1 and you have the longest line, with the number still in front for cut to peel off: no 'longest line' command needed.
```
$ printf 'one\ntwo\nthree\n' | sed -n '2p'
```
```
two
```
-n
don't print lines automatically; print only what a p command asks for
2p
address line 2, and print it
-n stops sed printing every line it reads, and 2p prints line 2 only. One process for one line, where head -2 | tail -1 takes two.
```
$ printf 'level=debug\n# debug on, debug verbose\n' > app.conf && sed -i 's/debug/info/g' app.conf && cat app.conf
```
```
level=info
# info on, info verbose
```
-i
edit the file in place instead of printing the result
s/old/new/g
substitute every match on each line, not just the first
sed -i rewrote app.conf itself: nothing printed until cat read the file back. The trailing g mattered on line 2, which had two matches: without it only the first debug on each line would change.
Cluster 3 — Summarize (turn many lines into an answer)
Collapse a whole stream into counts, ranks, and totals.
The last family aggregates — it takes many lines in and produces a small result out. This is where a pipeline stops describing data and starts answering questions:
- `sort` — order lines (-n numeric, -r reverse, -u unique-and-sorted, -k by a key column).
- `uniq` — collapse adjacent duplicate lines (so you almost always sort first). uniq -c prefixes each line with how many times it occurred.
- `wc` — count lines (-l), words (-w), or characters (-c).
- `xargs` — turn a stream of items into arguments for another command (e.g. find … | xargs rm).
Runnable in the dojo-unique-sorted-ips room — that room’s setup plants the files it reads
```
$ cut -d/ -f1 requests.log | sort | uniq -c | sort -rn | head -3
```
```
     42 10.0.0.7
     31 10.0.0.4
     19 10.0.0.9
```
-c (uniq)
prefix each unique line with the number of times it occurred
-rn (sort)
reverse + numeric — so the biggest counts come first, not '10' before '9'
This is the canonical 'top talkers' pipeline, and it's worth memorizing as a shape. cut extracts the field (here, the client before the first /), sort groups identical values so they sit together, uniq -c counts each group, sort -rn ranks those counts biggest-first, and head -3 keeps the podium. Five tiny tools answer a question none of them could alone — that's composition paying off.
The classic bug: uniq without sort
Run uniq -c on unsorted data and you'll get wrong counts and duplicate groups — because uniq only collapses lines that are adjacent. If the same value appears on line 2 and line 900 with other lines between them, uniq treats them as two separate runs. Always sort first so every identical line is neighbours with its twins. This is the single most common pipeline mistake beginners make.
awk can summarize too, because it remembers across lines. A variable you add to in the main { } block keeps its value from one line to the next, and an `END { }` block runs once, after the last line, to print the result. And since NR counts every line awk has read, across all the files you name, END {print NR} is a line count with no cat and no wc.
```
$ printf '5\n10\n7\n' | awk '{s += $1} END {print s}'
```
```
22
```
END { … }
runs once, after the last input line
For every line, add field 1 to the running total s. After the last line, END prints it once: 5 + 10 + 7 = 22. That is a column sum in one command, with no spreadsheet and no paste | bc gymnastics.
```
$ printf 'a b c d\ne f\n' > two.txt && printf 'g\n' > one.txt && awk 'NR==1 {print NF}' two.txt && awk 'END {print NR}' two.txt one.txt
```
```
4
3
```
NR==1
a pattern: run the action only where the line number is 1
Two questions, two one-liners. NR==1 {print NF} runs the action on line 1 only, and line 1 has 4 fields. END {print NR} waits until both files are read: NR kept counting from two.txt into one.txt, so it reports the grand total of 3 lines.
Check yourself
Why must you almost always run sort before uniq -c?
Now do it — build a top-talkers pipeline in your live terminal
Assemble the count-and-rank pattern yourself, one stage at a time, on a file you can see.
Reading pipelines is not the same as building them. The trick pros use is to build a pipe one stage at a time, checking the output after each | before adding the next — so you always know exactly what's flowing downstream. Try it now. First, make a tiny data file:
```
$ printf '10.0.0.7\n10.0.0.4\n10.0.0.7\n10.0.0.9\n10.0.0.7\n10.0.0.4\n' > /tmp/ips.txt; cat /tmp/ips.txt
```
This writes six lines of IPs to a scratch file and prints them back so you can see your raw input. (Your terminal supplies the output — that's the point of a practice step.)
```
$ sort /tmp/ips.txt | uniq -c | sort -rn
```
Now build the summarizer. Run it once as written and read the result. Then take it apart: run just sort /tmp/ips.txt, then add | uniq -c, then add | sort -rn, checking the stream after each stage. Watch the raw list become grouped, then counted, then ranked. When the top IP (10.0.0.7, three hits) lands at the top, you've built the pattern that solves the top-talkers Dojo — and a hundred real log questions besides.
Stuck building a pipe? Chop it at the last pipe.
When a long pipeline gives a surprising result, delete everything after the last | and look at what's actually flowing into that final stage. Repeat, walking backwards, until the output matches what you expected — the broken stage is the next one. Debugging a pipe is just reading the water at each joint.

Recap

Compose, don't memorize: decompose a task into a chain of one-step transforms joined by pipes (|), each tool reading stdin and writing stdout.
Search/filter tools REDUCE a stream (grep, head/tail, cut); transform tools REWRITE it (tr, sed, awk); summarize tools AGGREGATE it (sort, uniq -c, wc).
sort before uniq — uniq only collapses adjacent duplicates; sort | uniq -c | sort -rn is the universal 'count and rank' pattern.
Pick the lightest tool that does the job: cut over awk for fixed delimiters, grep -c over grep | wc -l — the exact habits the One-Liner Dojo rewards.
Reach for awk when only awk knows the answer: $NF (last field), NF (field count), NR (line number, so NR==1 means line 1), length, and END { } for totals across lines. sed -n '5p' prints one line; sed -i edits the file in place.
Build long pipes one stage at a time, reading the stream after each pipe, and debug them by chopping at the last | to see what's really flowing in.

The core-utility drawer, by job
Job	Reach for	Example
Keep matching lines	grep	grep -i error log
Drop matching lines	grep -v	grep -v '^#' config
First/last N lines	head / tail	tail -n +2 data.csv
Pull a delimited field	cut	cut -d, -f2 users.csv
Field-aware / math	awk	awk '{sum+=$1} END{print sum}'
Last field, whatever the count	awk $NF	awk -F, '{print $NF}' data.csv
Find-and-replace	sed	sed 's/old/new/g'
Print one line by number	sed -n	sed -n '10p' file
Swap/strip characters	tr	tr a-z A-Z
Order lines	sort	sort -rn
Count each distinct value	sort \| uniq -c	sort \| uniq -c \| sort -rn
Count lines/words	wc	wc -l

The Core-Utility Tour: Composing ls, grep, cut, sort & Friends

The Unix philosophy: small tools, one pipe

Cluster 1 — Search & filter (find the rows you care about)

Cluster 2 — Transform (reshape each line)

Cluster 3 — Summarize (turn many lines into an answer)

Now do it — build a top-talkers pipeline in your live terminal