Automated Triage: Scripting Live Evidence Collection

Advanced40 min6 steps

What you'll learn

Explain why a scripted collector beats manual triage: speed, consistency, and a smaller footprint on the live system
Order a collection set by volatility — memory-adjacent process and network state first, persistence and logs last
Build a real bash collector that writes each artifact to its own file under a timestamped output directory
Prove the integrity of a collection with a sha256 manifest and a collection log for chain-of-custody

Why automate triage at all
Frame scripted collection as the answer to slow, inconsistent, footprint-heavy manual triage.
You get the call: a host is behaving strangely and you have authorization to investigate it live, while it's still running. Your instinct is to start typing — ps here, ss there, cat a log file. That instinct is a trap.
Three problems make manual triage on a live box dangerous:
- It's slow. While you type one command and read its output, volatile evidence is decaying — sockets close, processes exit, the next attacker action overwrites a buffer.
- It's inconsistent. Run the same triage on Monday and on Friday and you'll gather a slightly different set, in a different order. When the findings end up in a report (or a courtroom), "I think I checked that" is worthless.
- It alters the system. Every command you run is itself an event on the host — it spawns a process, touches atimes, writes to your shell history. The more you poke around by hand, the more you contaminate the very scene you're trying to preserve.
The fix is a triage collector: a single script that captures a complete, repeatable evidence set in the order of volatility, then timestamps and hashes everything for chain-of-custody.
Think of a paramedic's go-bag. When someone collapses, the medic doesn't improvise — they reach for the same kit, packed in the same pockets, used in the same order, every single time. Under pressure, nothing gets forgotten because the procedure is baked into the bag, not into the medic's memory. A triage script is your go-bag: the same artifacts, gathered in the same order, so the panic of a live incident can't make you skip the volatile evidence that won't be there in five minutes.
Why it matters
This is why mature DFIR teams ship collectors, not checklists. Open-source examples like UAC (Unix-like Artifacts Collector), CyLR, and Linux-CatScale exist precisely to make collection one command, identical on every host. We're not going to copy them — we're going to re-implement their principles so you understand what a good collector does and why, and could build or audit one yourself.
Check yourself
Name one reason manual, type-it-yourself triage is risky on a live compromised host (speed, consistency, or footprint — give one).
Order of volatility: what to collect, and in what order
Sequence the collection set so the most perishable evidence is captured first.
The single most important design decision in a collector is order. Evidence has a shelf life, and the rule is order of volatility: capture the thing that will disappear soonest, first. Memory-resident state (running processes, live sockets, who's logged in) can vanish between one second and the next. A log file on disk will still be there in an hour. So you collect inward-out, volatile to stable:
- Volatile, memory-adjacent first — running processes (ps), live network state (ss, lsof, /proc), and logged-in users (w, last). This is the evidence an attacker's next move could erase.
- System context next — hostname, kernel, uptime, mounted filesystems, installed packages. Slow-changing, but it frames everything else.
- Persistence locations — cron jobs, systemd units, ~/.ssh/authorized_keys. How the attacker survives a reboot. Stable, but the point of triage.
- Logs last — auth logs, syslog, shell history. Largest and most stable; copy them after the perishable stuff is safe.
Collecting in the wrong order destroys evidence
The classic blunder: you start by cat-ing log files and ls-ing through /tmp to "have a look first." Meanwhile the malicious process you would have caught with ps has exited, and its socket is gone from ss. You captured the stable evidence (which wasn't going anywhere) and lost the volatile evidence forever. Worse, your ls and cat updated access times and shell history — you altered the disk before imaging the volatile state. Volatile first. Always.
```
$ w -h
```
```
root     pts/0    10.0.0.31        11:48    0.00s  0.04s  0.00s w -h
webadmin pts/1    198.51.100.9     11:31    9:42   0.01s  0.01s -bash
```
-h
Omit the header line, so the output is clean to redirect into an evidence file.
w shows who is logged in right now and what they're running — the most perishable evidence there is, because a session ends the moment the user disconnects. The webadmin session from 198.51.100.9 (a remote IP, idle 9m42s) is exactly the kind of live foothold you must capture before it logs off and vanishes from this view.
Check yourself
A triage collector should gather one category of evidence FIRST, before anything else. What kind?
The collection set: one artifact, one file
Map each triage question to a command and its own output file.
A good collector treats each artifact as a named, self-contained file. Don't dump everything into one giant log — when you (or a teammate, or an analyst three weeks later) open the bundle, processes.txt, connections.txt, and logins.txt are instantly findable, diffable, and individually hashable. One question → one command → one file. Here are the workhorses of the volatile and persistence tiers.
```
$ ss -tanp 2>/dev/null | head -5
```
```
State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port  Process
LISTEN  0       128      0.0.0.0:22            0.0.0.0:*          users:(("sshd",pid=912,fd=3))
LISTEN  0       1        0.0.0.0:31337        0.0.0.0:*          users:(("kworkerd",pid=2207,fd=4))
ESTAB   0       0        10.0.0.7:54122       198.51.100.55:443  users:(("kworkerd",pid=2207,fd=5))
```
-t
TCP sockets.
-a
All sockets — both LISTEN and ESTABLISHED, so you capture inbound doors and outbound beacons in one shot.
-n
Numeric ports/addresses — no name resolution, which is faster and can't be spoofed by a friendly service name.
-p
Show the owning process (name + PID) for each socket.
This one command answers "what is the network doing?" — every listening port and every live connection, each tied to a PID. 2>/dev/null discards the harmless "could not open netlink" warnings so they don't pollute the evidence file. In a collector this whole line redirects to connections.txt. Notice it captures both the inbound backdoor (:31337) and the outbound C2 (->198.51.100.55:443) in a single artifact.
```
$ (crontab -l 2>/dev/null; echo '--- /etc/cron.d ---'; ls -la /etc/cron.d 2>/dev/null)
```
```
*/5 * * * * /tmp/.cache/kworkerd --quiet
--- /etc/cron.d ---
total 12
drwxr-xr-x  2 root root 4096 Jun 23 11:50 .
-rw-r--r--  1 root root   42 Jun 23 11:50 0hourly
```
Persistence lives in many places; a collector grabs them all into cron.txt. The parentheses run the commands in a subshell, so all their combined output redirects together as one artifact. The echo separator labels where each source begins. Here the user crontab reveals the attacker's persistence: a job re-launching /tmp/.cache/kworkerd every 5 minutes — survive-a-kill and survive-a-reboot in one line.
Redirect, don't print
In an interactive demo we head and eyeball output. In the actual collector, every command's stdout goes to its own file with > "$OUT/connections.txt", and stderr is either captured (2>>collection.log) or suppressed (2>/dev/null) — never left to scroll past on screen. The script's job is to file evidence, not display it.
Check yourself
Why give each collected artifact its OWN file instead of appending everything to one combined log?
Build the collector: output dir, redirection, log
Assemble a minimal bash collector that files each artifact and logs what it did.
Now we wire the pieces into a real script. The skeleton of every collector is the same three moves: (1) make a uniquely-named, timestamped output directory; (2) run each collector with its output redirected to its own file, in order of volatility; (3) keep a running collection log that records what ran and when. The log is half the value — it's your contemporaneous record of exactly what the tool did to the host.
Example — not runnable on your lab box
```
$ cat collect.sh
```
```
#!/bin/bash
set -u
HOST=$(hostname)
OUT="/tmp/triage_${HOST}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$OUT"
LOG="$OUT/collection.log"

run() {            # run() label "command..."
  local file="$1"; shift
  echo "[$(date -u +%FT%TZ)] collecting $file: $*" >> "$LOG"
  eval "$@" > "$OUT/$file" 2>> "$LOG"
}

# --- volatile first ---
run processes.txt   'ps auxf'
run connections.txt 'ss -tanp'
run openfiles.txt   'lsof -i -P -n'
run logins.txt      'w -h; last -a'
# --- persistence ---
run cron.txt        'crontab -l; ls -la /etc/cron.d'
run authkeys.txt    'cat /home/*/.ssh/authorized_keys /root/.ssh/authorized_keys'
echo "done: $OUT"
```
set -u
Error on any unset variable — a typo'd path becomes a loud failure, not a silent write to the wrong place.
$(date +%Y%m%d_%H%M%S)
Timestamp in the dir name: every run is a distinct, sortable bundle that never overwrites a previous collection.
eval "$@"
Run the collector string as a command so a single helper handles every artifact uniformly.
2>> "$LOG"
Append each command's stderr to the collection log, so errors are evidence too, not noise on screen.
The run() helper is the heart of it: for each artifact it logs a UTC timestamp + the exact command to collection.log, then redirects stdout to that artifact's file and stderr to the log. Add the artifacts in order of volatility (processes → connections → logins, then cron → authkeys), and you've encoded the entire triage procedure once. To gather a different artifact, you add one run line — the consistency is structural, not something you have to remember.
Why it matters
The collection log is what turns a pile of text files into defensible evidence. It answers "what did this tool actually do to the host, and when?" with a UTC-timestamped line per command. If anyone later asks whether your collector touched something it shouldn't have, the log is the answer — written by the script, at the moment it ran, not reconstructed from memory afterward.
Never write the bundle onto the suspect filesystem
We write to /tmp here so the lesson is safe and reversible — but on a real investigation /tmp is part of the evidence and writing there overwrites unallocated space you may need to recover deleted files from. A real collector writes to removable or network storage (a mounted USB, an SMB share, stdout piped off-box over SSH) so the collection never lands on the suspect disk. Collecting evidence onto the drive you're investigating can destroy the very artifacts you came for.
Prove integrity: the sha256 manifest
Hash every collected file so the bundle's integrity can be proven later.
A bundle of evidence is worthless if you can't prove it wasn't altered after collection. The final, non-negotiable step is a manifest: a sha256sum of every file in the output directory, written to manifest.sha256. Anyone — a colleague, an analyst, opposing counsel — can later run sha256sum -c manifest.sha256 and get a per-file PASS/FAIL. A matching hash is mathematical proof the artifact is byte-for-byte what you collected.
```
$ ( cd "$OUT" && sha256sum * > manifest.sha256 )
```
The subshell `( cd ... && ... )` changes directory only inside the parentheses, so the parent script's working directory is untouched. Running sha256sum from inside $OUT makes the manifest store relative filenames (processes.txt, not /tmp/triage_.../processes.txt), so the bundle stays verifiable even after you copy it off-host to a different path. The && ensures we only hash if the cd succeeded — you never want to accidentally hash the wrong directory.
```
$ cat "$OUT/manifest.sha256"; echo '---'; sha256sum -c "$OUT/manifest.sha256"
```
```
0f5e...a91c  connections.txt
7b2d...4e88  cron.txt
3a90...c1f2  logins.txt
d44e...0b73  processes.txt
---
connections.txt: OK
cron.txt: OK
logins.txt: OK
processes.txt: OK
```
-c
Check mode: read the manifest and verify each listed file still matches its recorded hash, reporting OK or FAILED per file.
The manifest pairs each artifact with its hash; sha256sum -c re-verifies them. Every line reading OK is proof that file is unchanged since collection. If even one byte of processes.txt were altered, its line would read FAILED and the tampering would be undeniable. This is chain-of-custody in two commands — generate once at collection, verify any time after.
No manifest = unprovable evidence
The most common rookie mistake is skipping the hash because "I'm just doing a quick look." Then a finding turns serious, and you cannot prove the file you're presenting is the file you collected — maybe it was edited, maybe truncated, you have no way to show otherwise. Hash at the moment of collection or your evidence is just text files. The manifest costs one line and converts a pile of output into something defensible.
Check yourself
What command verifies that every file in a bundle still matches the hashes recorded at collection time?
Try it in your live terminal
Build and run a minimal triage collector end-to-end on the real sandbox.
You have a real root shell. Build a minimal collector by hand, run it, and verify its manifest — the same shape as a production tool, scaled down. It writes only under `/tmp`, so it's completely safe and reversible. Run the lines in order: make the output dir, collect volatile-first, then persistence, then hash the lot.
```
$ OUT=/tmp/triage_$(date +%s); mkdir -p "$OUT"; echo "collecting into $OUT"
```
Create a unique, timestamped output directory ($(date +%s) is the Unix epoch, guaranteeing no collision) and stash its path in $OUT so every later command writes to the same bundle.
```
$ ps auxf > "$OUT/processes.txt"; ss -tanp > "$OUT/connections.txt" 2>/dev/null; last -a > "$OUT/logins.txt"
```
The volatile tier, captured first: the full process tree, every socket with its owning PID, and the login history — each redirected to its own named file. 2>/dev/null on ss drops harmless netlink warnings so they don't land in the evidence file.
```
$ (crontab -l; ls -la /etc/cron.d) > "$OUT/cron.txt" 2>/dev/null
```
The persistence tier, collected after the volatile state is safe. The subshell ( ... ) combines the user crontab and the /etc/cron.d listing into one artifact, cron.txt.
```
$ ( cd "$OUT" && sha256sum * > manifest.sha256 )
```
Generate the integrity manifest from inside the bundle so filenames are relative. This is the step that makes the collection provable — run it last, once every artifact exists.
```
$ ls -la "$OUT"; cat "$OUT/manifest.sha256"
```
Confirm the result: a directory of named artifacts plus a manifest.sha256 pairing each file with its hash. That's a complete, ordered, integrity-proven triage bundle — built by hand, exactly as a collector script would. Clean up afterward with rm -rf "$OUT" since it's all under /tmp.

Recap

Automate triage because manual collection is slow, inconsistent, and contaminating — a script captures a complete, repeatable set before volatile evidence decays. It's your paramedic's go-bag.
Collect in order of volatility: memory-resident process/network/login state first, then system context, then persistence, then logs. Wrong order loses the perishable evidence forever.
Structure the bundle as one artifact per named file under a timestamped output dir, with a contemporaneous collection log of exactly what ran and when.
Hash everything into a sha256 manifest at collection time, and write the bundle OFF the suspect host — without those two disciplines your evidence isn't defensible.

Collection stage → what → command
Stage (volatility)	What it captures	Command → file
Volatile: processes	Running process tree + lineage	`ps auxf > processes.txt`
Volatile: network	Listening + established sockets w/ PIDs	`ss -tanp > connections.txt`
Volatile: open files	Per-process network/file handles	`lsof -i -P -n > openfiles.txt`
Volatile: sessions	Who's logged in + login history	`w -h; last -a > logins.txt`
Persistence	Cron, systemd, authorized_keys	`crontab -l; ls /etc/cron.d > cron.txt`
Integrity	Per-file hashes for chain-of-custody	`cd $OUT && sha256sum * > manifest.sha256`

Automated Triage: Scripting Live Evidence Collection

Why automate triage at all

Order of volatility: what to collect, and in what order

The collection set: one artifact, one file

Build the collector: output dir, redirection, log

Prove integrity: the sha256 manifest

Try it in your live terminal