jq: JSON on the Command Line — Filter, Select, and Aggregate

Intermediate20 min6 steps

What you'll learn

Explain why jq (not grep/awk) is the right tool for structured JSON
Pull fields out of objects and arrays with jq filters and the pipe
Filter records with select() and print raw values with -r
Aggregate JSON with length, add, unique, and max_by

Why jq instead of grep or awk
Understand that JSON is structured data and needs a structure-aware tool.
APIs, config files, container inspects, cloud CLIs — they all speak JSON. The temptation is to grep it, but JSON is not line-oriented text: a value can span lines, nesting matters, and the same key appears in many objects. grep/awk see characters; they don't understand that .users[2].name is a path into a structure.
`jq` is grep/sed/awk for JSON. You hand it a filter — a little expression describing what to pull out or compute — and it applies that filter to the parsed structure. Because it actually parses the JSON, it can index arrays, follow nested keys, and emit valid JSON (or raw text) back out.
```
$ echo '{"name":"auth","port":8080}' | jq '.port'
```
```
8080
```
.port
A field access — the value at key 'port' of the input object.
The filter .port is a path: 'from the input object, give me the value at key port'. jq parsed the JSON and returned the number 8080 — something grep could only fish out by fragile pattern-matching. . alone is the identity filter (the whole input); .key descends into it.
grep'ing JSON and getting burned
grep '"active": true' data.json feels fine until the file is pretty-printed differently, the key order changes, or active appears in a nested object you didn't mean. JSON has no guaranteed whitespace or key order, so text matching is brittle by design. jq '.[] | select(.active)' asks the structure the question and is immune to formatting.
Descending into arrays and objects
Use .[], field access, and the pipe to reach the values you want.
Most real data is an array of objects. Two constructs get you through it:
- `.[]` — the array/object iterator: it streams each element out, one at a time.
- `|` — the pipe, exactly like the shell's: it feeds each value on the left into the filter on the right.
So .[] | .name means 'for each element, give me its .name'.
Example — not runnable on your lab box
```
$ jq '.[] | .host' inventory.json
```
```
"node00"
"node01"
"node02"
```
.[]
Iterate: emit each element of an array (or each value of an object) separately.
.[] streams each object out of the array; | .host pulls the host field from each. You get one result per element. Note the quotes — jq prints JSON strings by default, quotes and all.
Example — not runnable on your lab box
```
$ jq -r '.[].host' inventory.json
```
```
node00
node01
node02
```
-r
Raw output: emit string values without surrounding quotes (and without JSON escaping).
Two changes: .[].host is shorthand for .[] | .host, and -r (raw) prints strings WITHOUT the JSON quotes — essential when you're piping the result into another shell command or writing a plain list to a file.
Check yourself
You want the bare host names (no quotes) from an array of objects, one per line. Which flag makes jq drop the surrounding quotes?
Filtering records with select()
Keep only the elements that match a condition.
`select(<condition>)` passes an element through only if the condition is true, and drops it otherwise — the JSON equivalent of grep. Combine it with .[] to filter an array, and wrap the result in [ ... ] to collect the survivors back into an array (for example, to count them).
Example — not runnable on your lab box
```
$ jq -r '.[] | select(.active) | .host' inventory.json
```
```
node01
node03
node04
```
select(.active)
Keep the element only if the expression is truthy; drop it otherwise.
For each element, select(.active) keeps only those whose .active is true, and .host prints that survivor's name. Read the pipeline left to right: iterate → filter → project. Conditions can be richer too: select(.cpus >= 4), select(.role == "db").
Example — not runnable on your lab box
```
$ jq '[.[] | select(.active)] | length' inventory.json
```
```
5
```
[ ... ]
Array construction: gather a stream of results back into a single JSON array.
length
The number of elements in an array (or characters in a string).
Wrapping the filtered stream in [ ... ] collects the matches into an array, and length counts it. This 'filter then count' pattern — [ .[] | select(cond) ] | length — is one you'll use constantly.
Aggregating: add, unique, max_by
Compute sums, distinct sets, and extremes over a JSON array.
jq isn't just extraction — it aggregates. Three workhorses:
- `add` — sum an array of numbers (or concatenate an array of strings).
- `unique` — sort an array and drop duplicates.
- `max_by(.k)` / `min_by(.k)` — return the whole element with the largest / smallest .k.
Example — not runnable on your lab box
```
$ jq '[.[].requests] | add' metrics.json
```
```
3921
```
add
Reduce an array of numbers to their sum (empty array → null).
[.[].requests] collects every request count into an array, then add sums it. The bracket step matters: add works on an array, so you build the array first, then fold it.
Example — not runnable on your lab box
```
$ jq -r 'max_by(.errors) | .service' metrics.json
```
```
billing
```
max_by(.errors)
The element whose .errors is largest — returns the object, not the value.
max_by(.errors) returns the ENTIRE object with the highest .errors (not just the number), so | .service then names the worst offender. min_by is the mirror image. This 'which record is the extreme?' question is exactly what max_by answers in one step.
Example — not runnable on your lab box
```
$ jq -r '[.[].role] | unique | .[]' inventory.json
```
```
cache
db
web
```
unique
Sort an array and remove duplicate values.
Collect all roles, unique sorts and de-duplicates them, and .[] streams the distinct values out (with -r, unquoted). The distinct-values idiom — [ .[].field ] | unique — is the JSON version of sort -u.
Why build an array before add/unique?
.[] produces a stream of separate values, but add and unique operate on a single array. The [ ... ] around .[].field is what turns the stream back into one array so the aggregator has something to fold over. Forget the brackets and you'll get an error or a per-element result instead of one aggregate — the brackets are the join between 'stream of things' and 'one collection'.
Reshaping JSON: build new objects with {} and map
Emit JSON in a new shape — the transform half of jq, not just extraction.
So far you've pulled values out. The other half of jq is putting them back in a new shape — turning one program's JSON into what another tool expects. Two constructs do it:
- `{ }` builds a new object. {host: .host, up: .active} makes an object with your chosen keys from the input. The shorthand `{host, active}` copies same-named fields, so {host, active} means {host: .host, active: .active}.
- `map(f)` applies filter f to each element of an array and collects the results back into an array. So map(.host) is the same as [.[].host], and map({host, up: .active}) reshapes every element in one step. map is the array-in, array-out shorthand for [ .[] | f ].
Two more everyday touches:
- `-c` (compact) prints each result on one line instead of pretty-printed — exactly the JSON Lines format you pipe into another program or append to a log.
- `.field // "default"` supplies a fallback when a key is missing or null, so a reshape doesn't silently emit null. // is jq's 'or-else' operator.
Example — not runnable on your lab box
```
$ jq -c 'map({host, up: .active})' inventory.json
```
```
[{"host":"node00","up":false},{"host":"node01","up":true},{"host":"node02","up":true}]
```
map(f)
Apply f to each array element and collect the results into a new array.
-c
Compact output: one line per result (JSON Lines), instead of pretty-printed.
map({host, up: .active}) rebuilds each element as a new object: it copies host (shorthand for host: .host) and renames active to up. -c prints the whole array on one compact line. You've transformed the input's shape, not just read from it — the core of gluing one JSON tool's output into another's input.
Example — not runnable on your lab box
```
$ jq -r '.[] | .region // "unknown"' inventory.json
```
```
us-east
unknown
eu-west
```
// "default"
Or-else: use the right-hand value when the left is missing, null, or false.
The // operator gives a fallback: when an element has no region (or it's null), jq emits "unknown" instead of null. This keeps a reshape robust against missing fields — real-world JSON is rarely uniform, and // is how you tolerate the gaps.
Check yourself
In jq, which brackets construct a NEW object with keys you choose — { } or [ ]?
Try jq in your live terminal
Run a few filters on real JSON, read-only.
Make a tiny JSON file and query it on the box to your right. Nothing here changes the system — you're just parsing and computing.
```
$ echo '[{"s":"a","n":3},{"s":"b","n":7}]' > /tmp/d.json && jq 'length' /tmp/d.json
```
Creates a two-element array and counts it with length → 2.
```
$ jq '[.[].n] | add' /tmp/d.json
```
Collects the n values into an array and sums them → 10. Watch the [ ... ] that builds the array for add.
```
$ jq -r 'max_by(.n) | .s' /tmp/d.json
```
Returns the object with the largest n and prints its s (raw) → b. That's max_by picking a whole record by a key.

Recap

JSON is structured, not line-oriented: use jq (a filter language that parses JSON) instead of grep/awk, which see only characters.
Reach values with field access (.key), the iterator (.[]), and the pipe (|); add -r for raw, unquoted string output.
Filter records with select(<cond>); the 'filter then count' idiom is [ .[] | select(cond) ] | length.
Aggregate with add (sum), unique (distinct, sorted), and max_by/min_by (the record with the extreme key) — building an array with [ ... ] first when an aggregator needs one.
Reshape, don't just read: { } builds a new object ({host, up: .active}), map(f) transforms each array element, -c prints compact JSON Lines, and .field // "default" supplies a fallback for missing keys.

jq filter → what it does
Filter	Does	Note
`.key` / `.a.b`	Field access (and nested)	A path into the structure
`.[]`	Iterate array/object	Emits a stream of values
`-r`	Raw string output	Drops the JSON quotes
`select(cond)`	Keep matching elements	JSON's grep
`[ .[] \| select(cond) ] \| length`	Count a filtered subset	Bracket to build an array
`[.[].k] \| add`	Sum a field	add needs an array
`[.[].k] \| unique`	Distinct, sorted values	JSON's sort -u
`max_by(.k)` / `min_by(.k)`	Record with extreme key	Returns the whole element
`{a, b: .c}` / `map(f)`	Build a new object / transform each	The reshape half of jq
`-c` / `.k // "def"`	Compact JSON Lines / fallback value	For piping onward / missing keys

jq: JSON on the Command Line — Filter, Select, and Aggregate

Why jq instead of grep or awk

Descending into arrays and objects

Filtering records with select()

Aggregating: add, unique, max_by

Reshaping JSON: build new objects with {} and map

Try jq in your live terminal