The Investigation Playbook: How to Hunt a Compromised Linux Host

A pager goes off: "this server is making thousands of connections to an IP nobody recognizes." The instinct of a beginner is to reach for a checklist — run these ten commands, paste the output, hope the answer falls out. That's not how a responder thinks, and it's the single biggest thing separating someone who finds the intruder from someone who just collects logs.

Here's the model to hang everything on. An investigation is a loop:

1. Observe — what does the evidence in front of you actually say?

2. Hypothesize — what's the simplest story that would explain it? ("A process on this box is beaconing to a command-and-control server.")

3. Test — go find evidence that would prove or kill that theory. Not evidence that flatters it — evidence that could destroy it.

4. Pivot — the test points you somewhere new. Follow it. Loop.

You exit the loop when you can tell a single, evidence-backed story from first signal to root cause — and only then do you contain.

One rule sits above the loop and governs every step of it: prefer ground truth over telemetry. Telemetry is what a tool reports — an EDR log line, a process's name, a dashboard. Ground truth is what's actually on the machine — the live process, the bytes of the binary on disk, the socket the kernel is holding open. Tools lie, by omission or by being fooled. The kernel doesn't. When they disagree, ground truth wins, every time. Keep that sentence in your head for the rest of this lesson.

Your first hypothesis is almost always "some process is doing this." So list them all — everything, with full command lines, because the arguments are where the truth usually hides.

Two processes stand out. kworkerd looks like a kernel worker thread — the kernel really does run things called kworker. That's the disguise. But here is the tell, and it's a rule you can rely on: real kernel threads have no program on disk. They live entirely inside the kernel and show up in ps wrapped in square brackets, like [kworker/0:1]. Anything with a real path like /usr/local/sbin/kworkerd and a parent shell is a user-space program wearing a kernel's name. Impostor.

Don't take my word for it — or the process's. Ask the package manager who owns that file. Every legitimate system binary was installed by a package; a dropped payload was not.

You suspect this process is beaconing to a command-and-control (C2) server. To prove where, don't lean on one artifact — a single log line can be forged, misread, or stale. A destination you can see from three independent angles is a destination you can write in the report with confidence. Look for agreement.

Source one, the process's own arguments — you already have them from ps: --c2 198.51.100.7:443. Source two, the binary itself. Read it:

Source three, the network record. This host runs a flow logger, and its log is the third witness:

Here's the part that trips people up, and it's the most important idea in this lesson. The endpoint security agent (EDR) on this box logged about three events all night. The network sensor logged about twenty thousand flows to the C2. Same activity, same hours. How can two tools on the same machine disagree by four orders of magnitude?

The answer is in how each tool watches. This EDR hooks process creation — every time the kernel runs a new program (an execve system call), the agent records one event. That's a cheap, common way to build endpoint telemetry. Now look at the beacon: it's one long-lived process that opens a socket, fails, sleeps, and opens another socket — in a loop, forever. It never launches a new program to do it. So the execve hook fired once, when the beacon started, and then went silent for the entire attack. Three events: the launch and a couple of neighbors.

The network sensor, meanwhile, watches at a completely different layer — it counts flows on the wire. It doesn't care how many processes exist; every connection attempt is one record. Twenty thousand beacons, twenty thousand rows. Neither tool is broken. They're watching different things, and the attacker's technique — reuse one process, reconnect in a loop — happens to fall squarely in the EDR's blind spot while lighting up the network log.

You've found it, proven it, and understood it. Now stop it — carefully. There's a legitimate service running right next to the beacon: /usr/local/bin/webapp, the thing customers are actually using. Containment that kills the beacon and the storefront isn't a win; you've just traded a security incident for an outage. Precision is the whole job here.

The tempting move is a broad pkill. Resist it. Watch what a lazy pattern does:

Then — always — verify both halves of the outcome. The threat is gone and the business is still up:

Killing the process is not the end. Any attacker worth worrying about installs persistence — a mechanism that relaunches the payload after a crash, a logout, or a reboot. Kill the beacon, walk away, and if you left a cron job or a systemd unit behind, it's beaconing again before you've closed your laptop. So hunt the respawn mechanisms the same way you hunt everything: search for the one string you know appears in all of them — the beacon's name.

Reading the method is not the same as running it under a little pressure. Open the Beacon Hunt lab (ir-beacon-hunt) — it's the exact scenario you just walked through, seeded live on a real host, waiting for you to work it end to end. Don't copy the commands above blindly; run the loop. Observe, hypothesize, test, pivot.

Your objectives in the lab, in order: