Text Filters: Reshaping Lines with cut, sort, uniq, wc & tr

You can now locate files (find) and search inside them (grep). The third everyday skill is reshaping the lines you get back — pulling out a column, sorting, counting, deduping. The tools that do this are filters.

A filter follows one simple contract: it reads lines on standard input, does one small transformation, and writes lines on standard output. Because input and output are both just streams of lines, filters snap together with pipes — the output of one becomes the input of the next. This is the Unix philosophy in action: instead of one giant do-everything command, you chain a few tools that each do one job well.

The filters worth knowing:

• `cut` — pull out columns (fields or character ranges) from each line.
• `sort` — put the lines in order.
• `uniq` — collapse adjacent duplicate lines (and count them).
• `wc` — count lines, words, or characters.
• `tr` — translate or delete individual characters.
• `head` / `tail` — take just the first or last few lines.

Each reads lines and writes lines, so any of them drops into a pipeline. The art is picking the right small tools and ordering them.

Lots of data is columnar — each line is fields separated by a delimiter (a comma in a CSV, a colon in /etc/passwd, spaces in a log). `cut` pulls out the columns you want:

• `-d` sets the delimiter (the separator character). -d',' for CSV, -d':' for /etc/passwd. The default is a tab.
• `-f` picks the fields by number, starting at 1. -f1 is the first field; -f1,3 is fields 1 and 3; -f2-4 is a range.
• `-c` picks character positions instead of fields: -c1-8 is the first eight characters of each line.

So cut -d':' -f1 /etc/passwd prints just the username from each line — the text before the first colon. The mental model: cut slices every line at the delimiter and keeps the pieces you name.

`sort` puts lines in order. By default it sorts lexically (as text), which surprises people with numbers: as text, 10 comes before 2 because it compares character by character. The flags fix this:

• `-n` sorts numerically (so 2 before 10).
• `-r` reverses the order (largest/last first).
• `-k N` sorts by field N, not the whole line (pair with -t to set the field separator, like cut's -d).

`uniq` collapses duplicate lines — but with a catch that trips up everyone: it only removes duplicates that are adjacent. It compares each line to the one right before it. So uniq on unsorted data misses duplicates that aren't next to each other. The fix is a rule: `sort` first, then `uniq`. Once the data is sorted, all copies of a line are adjacent, and uniq can collapse them.

uniq's star flag is `-c`: it prefixes each line with a count of how many times it occurred. sort | uniq -c is the canonical "tally up how many of each" move.

When all you want is the distinct values, `sort -u` (unique) sorts and drops the duplicates in one step: the same lines as sort | uniq, with one less process. Keep uniq for when you need what only it can do, like -c counts.

Three more filters finish the everyday set:

• `wc` counts. `wc -l` counts lines (the one you'll use most — 'how many matches?'), -w counts words, -c counts bytes. grep … | wc -l answers 'how many lines matched?'.
• `tr` translates or deletes individual characters. tr 'a-z' 'A-Z' upper-cases; tr -d '\r' deletes carriage returns (handy for cleaning Windows line endings); tr -s ' ' squeezes repeated spaces into one. Note tr works on characters from stdin only — it takes no filename.
• `head` and `tail` take a slice. `head -n 5` prints the first 5 lines; `tail -n 5` the last 5. tail has a superpower: `tail -f` follows a file, printing new lines as they're appended — the standard way to watch a live log.

These are the 'ends and edges' tools: count how much, tidy the characters, or grab just the top or bottom of a stream.

Every filter so far reads one stream. Two classic tools read two sorted files side by side, which answers a family of questions nothing else here can:

• `comm A B` compares the files line by line and prints three columns: lines only in A, lines only in B (indented one tab), and lines in both (indented two tabs). You almost never want all three, so you suppress the ones you don't: -1, -2 and -3 hide columns 1, 2 and 3. comm -23 A B leaves only column 1, the lines only in A (a set difference). comm -12 A B leaves only column 3, the lines in both (an intersection). comm -13 A B is what B has that A lacks.
• `join A B` is a relational join: for each key (the first field, by default) that appears in both files, it prints the key followed by the rest of A's line and the rest of B's. Keys found in only one file are dropped.

Both tools walk the two files in step, which only works if both are sorted on the part they compare: the same rule as uniq, for the same reason. Sort first if you aren't sure.

One continuous experiment building the most useful pipeline in all of text processing: cut a field, sort it, count each value, then rank them. This 'top-N' shape answers 'which IP hit us most?', 'which status code is most common?', 'who ran the most commands?' — all the same pattern.