APT Investigation: Reconstruct the Kill Chain, Then Eradicate It

Advanced45 min6 steps

What you'll learn

Frame a multi-stage intrusion as a kill chain and investigate it in stages, not as scattered alerts
Reconstruct the chain by correlation across sources, defeating volume and single-source decoys
Follow the attacker across hosts (lateral movement) and out (exfiltration)
Assemble an evidence-anchored timeline mapped to MITRE ATT&CK
Eradicate completely: enumerate every independent persistence foothold, then write the containment/eradication/recovery report

Think in kill chains, not alerts
Frame an APT investigation as reconstructing a staged campaign.
A single alert — a failed-login spike, one odd outbound connection — is a dot. An advanced intrusion is a chain of dots: initial access → foothold/execution → persistence → command-and-control → lateral movement → collection → exfiltration. The investigator's job is to reconstruct that chain end to end, because containment is only as good as the last stage you found.
The discipline that makes this tractable is MITRE ATT&CK: a shared catalogue of adversary techniques (T-numbers). Naming each stage — T1110 brute force, T1505.003 web shell, T1021.004 SSH lateral movement, T1041 exfiltration — turns a pile of logs into a story with named acts, and tells you which acts you haven't yet accounted for. If you can name the C2 but have no exfiltration technique in your timeline, you're not done.
Why 'find one thing and clean it' fails against an APT
Amateur intrusions have one foothold; capable ones have several, deliberately independent. If you find the webshell, kill it, and move on, the cron beacon re-drops it tonight and the backdoor SSH key lets them straight back in. Reconstruct the whole chain first — you cannot eradicate what you haven't mapped.
Reconstruct by correlation, not by volume
Identify the intruder as the entity that ties stages together.
The first hard problem is which IP is the attacker. Volume lies: the loudest brute-forcer is usually a botnet that never got in, and a noisy web scanner never touches a shell. The real intruder is the one entity present in more than one stage — the IP that both logged in successfully and drove the webshell. That intersection is the anchor everything else hangs from.
Runnable in the apt-reconstruct room — that room’s setup plants the files it reads
```
$ comm -12 <(grep 'Accepted password' /var/log/auth.log | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -u) <(grep -E '/uploads/[^ "]+\.php' /var/log/nginx/access.log | grep -oE '^([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -u)
```
```
203.0.113.42
```
comm -12
Print only lines common to both sorted inputs — the set intersection.
The left set is IPs that authenticated successfully; the right set is IPs that requested the webshell. comm -12 prints only what's in BOTH. The brute-force decoy is absent from the login-success set; the scanner is absent from the webshell set — so each drops out, leaving the one correlated intruder. This is the whole game: intersect stages, don't rank noise.
Check yourself
One IP has 5,000 failed SSH logins and no success; another has 12 failures, one success, and later webshell POSTs. Which is the APT actor?
Follow the attacker across hosts and out the door
Trace lateral movement to a second host and the exfiltration channel.
Once inside, an APT rarely stays put. Lateral movement (T1021) uses the stolen credentials to reach deeper hosts — and the evidence is on the second host's logs, which is why responders pull auth logs from neighbours, not just the entry point. Then comes the point of the whole operation: collection (T1560, staging an archive) and exfiltration (T1041).
Runnable in the apt-reconstruct room — that room’s setup plants the files it reads
```
$ grep -E 'Accepted publickey for deploy|COMMAND=.*tar' /root/case/*/auth.log
```
```
/root/case/gitlab01/auth.log:Jul 14 06:22:14 gitlab01 sshd[8801]: Accepted publickey for deploy from 10.0.12.7 port 40122 ssh2
/root/case/gitlab01/auth.log:Jul 14 06:24:51 gitlab01 sudo[8850]:   deploy : TTY=pts/1 ; PWD=/var/opt/gitlab ; USER=root ; COMMAND=/usr/bin/tar czf /tmp/.cache/archive-d1b195.tar.gz /var/opt/gitlab/backups
```
Accepted publickey
A key-based login — here the attacker reusing access on the next host inward.
Several hosts' logs were pulled; one of them shows the stolen deploy account logging in from web01's internal IP (10.0.12.7) — lateral movement confirmed — then tar-ing that host's data into a hidden staging file. The hostname field (gitlab01 in this example; your box's pivot differs) is your pivot target; the tar is the collection step. Neither is visible from web01 alone. Another host may show deploy from the same IP being rejected — a failed attempt is not a pivot.
Runnable in the apt-reconstruct room — that room’s setup plants the files it reads
```
$ awk -F'[ =]' '{for(i=1;i<=NF;i++){if($i=="dst")d=$(i+1);if($i=="bytes")b=$(i+1)}print b" "d}' /var/log/conn.log | sort -rn | head -1
```
```
8123456 203.0.113.212
```
sort -rn | head -1
Largest-first by bytes — surface the single heavy transfer among many small beacons.
C2 and exfil both look like 'outbound to an external IP', but they differ by SIZE: C2 is a small beacon (~288 bytes) repeated on a timer; exfil is one big transfer (megabytes). Sorting connections by byte count puts the exfil destination on top — 203.0.113.212 received 8 MB, the stolen archive leaving. Don't confuse the chatty beacon with the heavy transfer.
Reporting the C2 as the exfil destination
They're different techniques (T1071 vs T1041) and usually different hosts. The beacon is small and periodic; the exfil is a byte spike. If your report lists the C2 IP as where the data went, you've mis-scoped the breach — block the beacon and the data still walked out a different door. Separate them by volume every time.
Build the timeline, mapped to ATT&CK
Assemble an ordered, evidence-anchored narrative.
A timeline is the deliverable that turns findings into an incident. Order every confirmed event by timestamp and tag it with its technique, so a reader sees the campaign unfold and can check nothing is missing between stages.
Build it from evidence you can point at, not inference: each entry should trace back to a specific log line, file ctime, or connection record. Anchor on one trustworthy timestamp — the successful login, or a payload's change-time — and pull everything in the surrounding window; timestamps that cluster are usually one operator working one session. Where two clocks disagree (the web server in UTC, an endpoint in local time), normalise to one zone before you order events, or the story scrambles. A good timeline is also falsifiable: if a stage has no evidence line, you mark it a hypothesis, not a fact, and go looking — the gap between 'C2 established' and 'data exfiltrated' is exactly where an unfound collection or staging step is hiding.
The target format: an evidence-anchored, ATT&CK-tagged timeline
You build this yourself as you work — there is no pre-written timeline.txt to cat; the deliverable is the table you assemble from the evidence. The shape to aim for (placeholders stand in for the values you derive from this box's telemetry — fill them from your own findings, don't copy these):
`
time ATT&CK event (evidence)
05:22 T1110 brute force from <attacker_ip> (web01 sshd)
05:23 T1078 valid login accepted for deploy@web01 from <attacker_ip>
06:10 T1505.003 webshell POST <webshell_path>
06:15 T1053 cron <malicious_cron> -> curl|bash
06:16 T1071 C2 beacon web01 -> <c2_ip>:8080 (288B/60s)
06:22 T1021.004 lateral: deploy web01 -> <pivot_host> (publickey)
06:24 T1560 tar <data dir> -> staging
06:33 T1041 exfil web01 -> <exfil_ip> (large transfer)
`
Each line is one confirmed event: time, ATT&CK ID, and the evidence it traces to. Read top to bottom and the campaign is legible — and the gaps are obvious. No T1041 line? You haven't found the exfil yet. This ordered, technique-tagged narrative is what reconstruction produces and what the report is built on.
Eradicate completely — find every foothold
Enumerate all independent persistence, not just the obvious one.
Eradication is where completeness wins or loses. An APT plants several independent persistence mechanisms so removing one leaves the rest. Sweep every classic surface, and treat each as if it's the only thing standing between the attacker and re-entry:
- `cron.d` / timers (T1053) — a job that re-downloads the payload.
- `authorized_keys` (T1098.004) — a backdoor SSH key.
- shell init scripts — profile.d, rc.local, bashrc (T1546) — code that runs on login/boot.
- accounts (T1136 / T1078) — a new user, or an extra UID-0 (root-equivalent) account.
```
$ grep -rlE 'curl|wget|/bin/bash' /etc/cron.d /etc/profile.d; awk -F: '$3==0 && $1!="root"{print $1}' /etc/passwd
```
```
/etc/cron.d/systemd-timesync
/etc/profile.d/colorls.sh
systemd-net
```
grep -rlE
Recursively list files whose contents match — surface phone-home scripts across surfaces at once.
$3==0 && $1!="root"
UID-0 (root-equivalent) accounts that aren't root — backdoor admin logins.
Two INDEPENDENT sweeps, one per line — the content grep across cron.d and profile.d, then the UID-0 account check. They are joined with ;, not &&, on purpose: with && the account sweep would only run if the grep already matched, so on a host with no phone-home script the UID-0 backdoor would silently never get checked. Independent findings must report independently. (The output above is a REFERENCE incident that populated every surface — a downloader in cron.d AND in profile.d, plus an extra root-equivalent account systemd-net. Your per-seed box seeds fewer: expect one malicious /etc/cron.d/<job> under a randomized name and, unless the box planted more, root as the only UID-0 account. The lesson is the technique — sweep every surface and read each result on its own.) Miss the profile.d script and the box re-infects on the next login; miss the UID-0 account and they log straight back in. Enumerate the whole set before you remove anything.
Order of operations: scope, then contain, then eradicate
Don't delete the first foothold you find — you tip off the attacker and lose evidence, while the other footholds survive. Fully scope the persistence set first, contain (isolate the host / block the C2 and exfil IPs), then eradicate everything at once, and finally recover and verify. Piecemeal cleanup is how boxes get re-owned the same night.
Write the report, and try it live
Turn evidence into a containment/eradication/recovery record, and practise the moves.
The report is the product. It states, with evidence: what happened (the ATT&CK-tagged timeline), the IoCs (attacker IP, C2, exfil, webshell, staging path, backdoor accounts/keys), containment (what you isolated/blocked), eradication (every foothold removed), and recovery (rebuild/restore + verification). Anyone should be able to act on it without re-deriving your work. Now practise the enumeration moves on the box to your right — read-only, nothing here is malicious.
```
$ awk -F: '$3==0{print $1" is UID 0"}' /etc/passwd
```
Lists every UID-0 account on this box (should be just root). On a real host, a second name here is a backdoor to add to your accounts manifest.
```
$ sudo grep -rlE 'curl|wget' /etc/cron.d /etc/profile.d 2>/dev/null || echo 'no phone-home scripts found'
```
Sweeps two persistence surfaces for scripts that fetch and run code. A clean box prints the fallback; a compromised one lists the files to eradicate.
```
$ last -a 2>/dev/null | head -5
```
Recent logins with source — the starting point for spotting the initial-access and lateral-movement stages on a real investigation.

Recap

Investigate an APT as a kill chain (initial access → foothold → persistence → C2 → lateral movement → collection → exfil), tagging each stage with MITRE ATT&CK so missing stages are obvious.
Find the actor by correlation across stages (successful login ∩ webshell), not by volume — the loudest source is usually a decoy.
Follow the attacker across hosts (lateral movement shows on the second host's logs) and out (exfil is the big byte transfer, distinct from the small periodic C2 beacon).
Eradicate completely: enumerate every independent foothold (cron, authorized_keys, profile.d/init scripts, UID-0 accounts) before removing anything — one missed foothold re-owns the box.
The deliverable is a report: ATT&CK timeline, IoCs, and containment / eradication / recovery with verification.

Stage → MITRE → where the evidence is
Stage	ATT&CK	Evidence
Initial access	T1110 / T1078	auth.log: fails then Accepted
Foothold	T1505.003	web log: POST to /uploads/*.php
Persistence	T1053 / T1098.004 / T1546	cron.d, authorized_keys, profile.d
C2	T1071	conn.log: small periodic beacon
Lateral movement	T1021.004	second host's auth log (login from web01)
Collection / Exfil	T1560 / T1041	sudo tar; conn.log: large byte transfer

APT Investigation: Reconstruct the Kill Chain, Then Eradicate It

Think in kill chains, not alerts

Reconstruct by correlation, not by volume

Follow the attacker across hosts and out the door

Build the timeline, mapped to ATT&CK

Eradicate completely — find every foothold

Write the report, and try it live