Detection & Triage: Finding Indicators of Compromise

You get the call: "this box is acting weird, can you look?" You don't yet know if it's a misconfiguration, a noisy neighbour, or a live intruder. Triage is the fast first pass that answers one question — is something wrong here, and if so, where do I dig?

Think like an ER doctor. A patient rolls in and you don't start with their family history; you check vitals in a fixed order, treat the life-threatening thing first, and chart as you go so the next clinician can pick up where you left off. Detection works the same way: a fixed sweep, worst-first treatment, and notes the next responder can act on.

The sweep has four pillars. Check each, every time:

• Logs — who logged in, who failed, what the system itself reported
• Processes — what is running that shouldn't be
• Network — what is listening or phoning home
• Files — what was created or changed, and when

The pillars are not four separate investigations. They are four windows onto one event.

The reason the four pillars work as a system is that they are linked by the same underlying activity. A real intrusion leaves a thread you can pull, and pulling it in one pillar exposes the others.

Say the process pillar turns up something odd — a process called kworkerd eating CPU from /tmp. That single indicator is a thread:

• Process → Network: what port is it holding? ss -tlnp ties the PID to a listening socket and any outbound C2.
• Network → Files: where does its binary live? /proc/PID/exe gives the real path on disk.
• Files → Logs: when did that file appear? Its ctime is a timestamp — and that timestamp tells you which log lines to read to find the initial access.

One suspicious process just handed you a port, a file, and a moment in time. That moment is the anchor for the timeline you'll build in step 5.

Logs are where access leaves its first fingerprint. Before an attacker can run a process or open a port, they usually have to get in — and getting in is logged. Three quick reads cover most of it:

• last — successful logins (who, from where, when)
• lastb — failed logins, the signature of a brute-force or password-spray
• journalctl / /var/log/auth.log — the detailed auth events behind those summaries

An attacker who only has a stolen password is one password-reset away from being locked out. So a common next move is persistence: create a new account, or quietly hand an existing one root. Two files hold the answer, and both are plain text you can read in seconds.

The rule that matters: on Linux, root is not a name, it's a number — UID 0. Any account with UID 0 is root, whatever it's called. So an attacker's support user with 0 in the third field of /etc/passwd is a root account hiding behind an innocent name.

Individual indicators are dots. A timeline connects them into a story: spray, login, drop binary, add account, beacon out. You build it by anchoring on one known-bad timestamp and asking the filesystem what else changed around then.

You already have the anchor: the binary's ctime was 02:14, and the login was 02:11. So the question becomes: what was created or modified between 02:00 and 02:30? find answers it directly with -newermt (newer than a time) bracketed by ! -newermt (but not newer than a later time).

Triage isn't just finding things — it's deciding what to handle first. Back to the ER: you treat the arterial bleed before the sprained wrist, even if the wrist patient is shouting louder. Rank findings by two questions:

• Impact — how bad is this if left alone? (active root C2 > dormant file)
• Spread potential — can it reach other hosts? (lateral movement, planted SSH keys, shared credentials > a local-only artifact)

For our box: the live C2 connection and the UID-0 `toolkit` account come first — active control and root-level persistence. The cron jobs and authorized_keys are next — persistence that re-arms the threat. The cryptominer's CPU burn, annoying as it is, is last: it's loud but contained.

A usable IoC list from this investigation looks like:

• IP 198.51.100.23 — brute-force source + successful login (block at firewall)
• IP 192.0.2.66 — outbound C2 destination (block egress)
• File /tmp/.cache/kworkerd — cryptominer payload (preserve, then remove)
• File /etc/cron.d/.sysupdate, /var/spool/cron/crontabs/root — persistence (remove)
• File /home/webadmin/.ssh/authorized_keys — planted key (audit, remove attacker entry)
• Account toolkit (UID 0) — backdoor root account (disable, then remove)
• Timeline 02:08 spray → 02:11 login (webadmin) → 02:14 payload → persistence + C2

That list is the deliverable. Anyone can act on it without re-deriving it.

You have a real root shell. Run this sweep in order — it's the exact first-pass triage you'd do on a "this box feels off" call. There may be nothing malicious here; the goal is to make the moves reflexive so they're automatic when it counts. Note anything that breaks the pattern as you go — that's your IoC list starting.

One caveat as you run these: `journalctl` may come back empty on this box — it logs to flat files under /var/log, not systemd's journal. That's not a miss, it's the signal that this host doesn't use journald, so /var/log (and the last/lastb reads above) is where the answer lives. The rest of the sweep runs in full.