Containment: Stopping the Spread

You've detected an intruder. The process forensics lesson taught you how to find the malicious process, the port it opened, and the C2 server it's calling home to. Containment is what comes next — and it comes before eradication (the clean-up).

The goal of containment is narrow and urgent: stop the bleeding. Cut the attacker's control and halt the spread to other hosts. But there's a trap built into the phase, because almost every containment action is destructive — killing a process, blocking a connection, or pulling a host offline all destroy the very volatile evidence you'd want later to understand what happened.

Think of a ship taking on water. When a compartment floods, the crew closes the watertight doors to seal it off — they don't scuttle the whole vessel, and they don't wait until they've fully surveyed the damage. They seal the compartment while the inspection continues. Containment is closing watertight doors: isolate the damage so it can't spread, but keep the ship afloat and investigable.

Before you cut a single connection, take a snapshot. Volatile evidence is the data that exists only while the system is in its current state: the running process table, the live socket list, the established connections, and the malware's binary still resident in memory. All of it is gone once you contain.

This is the cross-reference to your forensics toolkit — but here the purpose is different. In forensics you read these tables to find the threat. In containment you capture them to freeze the scene before you destroy it. Save everything to a separate evidence directory.

Now capture the one piece of evidence that disappears most completely: the malware binary itself. Recall that an attacker often deletes the file from disk ((deleted) in /proc/<pid>/exe) while the process keeps running from memory. The kernel still holds that binary open, so you can copy it straight out of /proc before you kill the process.

With evidence frozen, your first containment action targets the attacker's lifeline: their command-and-control (C2) channel. As long as the implant can reach its C2 server, the attacker can issue commands, pull more tools down, pivot to other hosts, and exfiltrate data. Cut that channel and the attacker goes blind — the running process is now a beached fish.

Network isolation is the highest-leverage containment because it's fast (one firewall rule), reversible (you can lift it), and it severs control without you having to touch the malicious process yet. It buys time to investigate calmly.

There's an art to how much you cut. The crude move is to yank the network cable — but that takes the host fully offline, alerts the attacker that they've been caught (their session just died hard), and severs your own remote access to investigate. The surgical move is to block only the C2 address at the firewall. To the attacker it looks like an ordinary network blip; you keep your shell and every other connection for analysis.

From your captured lsof/ss snapshot you have the C2 address. Here we'll use the reserved documentation address 203.0.113.5 as the C2. On a modern system the firewall is nftables; the rule blocks all outbound traffic to that one host.

C2 is severed. Now stop the malicious process on the host itself. Here learners reach for the wrong tool out of habit: systemctl stop.

A planted process is not a managed service. systemctl only knows about units — services that systemd was told to manage, with a unit file under /usr/lib/systemd/system/ or /etc/systemd/system/. An attacker's process that someone launched from a shell, or that a web exploit spawned, has no unit. systemd has never heard of it. Running systemctl stop kworkerd against it does literally nothing — there's no unit by that name to stop.

Think of systemctl as the building's official staff directory: it can page anyone the company hired. An intruder who climbed through a window isn't in the directory, so paging the directory never reaches them. You have to go to them directly. That direct tool is kill, which sends a signal straight to a process by PID — no unit required.

Blocking the C2 and killing the process stop the malware. They do nothing about the credentials. If the attacker came in with a stolen or guessed password, they can simply log in again, so when threat intel or the auth log says an account is being used by someone else, contain the account too.

Lock it, don't delete it. usermod -L <user> (or the equivalent passwd -l <user>) puts a ! in front of the password hash in /etc/shadow. No password can ever hash to a value starting with !, so password authentication fails, but the account, its home directory, its files and its UID all stay exactly as they were. That matters: the home directory, shell history and owned files are evidence, and userdel would throw them away (and free the UID for reuse, so the next account created inherits ownership of anything left behind). Locking is also reversible: usermod -U restores the original hash once the password has been reset.

Everything so far was surgical: block one IP, kill one process, keep the host running and investigable. That's the right default — minimal collateral damage, maximum information retained. But sometimes surgery isn't enough, and you escalate to full host isolation.

The deciding question is spread risk: how likely is it that the compromise has gone beyond what you can see, or is actively reaching other systems right now? Back to the ship: you seal one compartment when the leak is contained to it, but you seal off a whole deck when water is pouring between compartments faster than you can patch.

Stay surgical when: you've identified the specific process and its single C2, there's no sign of lateral movement, and you need to keep watching the attacker or keep the host's services running.

Escalate to full host isolation when: the host is actively attacking other machines, you suspect a rootkit or unknown additional implants you can't fully enumerate, the blast radius is unclear, or the host is too critical to risk leaving even partially connected. Isolation means cutting the host off from the network entirely while leaving it powered on (so volatile evidence and your forensic access via an out-of-band path survive).

Some files must not change while you work: authorized_keys (so a re-planted key can't land), /etc/passwd and /etc/shadow, and the logs that record what happened. File permissions can't guarantee that on a host where the attacker may hold root, because root bypasses permission bits. Inode attributes are a separate layer the kernel checks with no root override:

• chattr +i FILE, immutable: no write, delete, rename, hard link or mode/owner change, for root too.
• chattr +a FILE, append-only: new lines can be added at the end, and nothing already written can be overwritten, truncated or removed. That makes a log tamper-evident.

Root is not powerless against them. Clearing a flag (chattr -i) needs the CAP_LINUX_IMMUTABLE capability, which a normal root shell holds, so the flag won't stop a determined intruder with a full root shell. It does stop scripts, careless commands and any process that dropped that capability. Treat it as a speed bump that also leaves a trace. Clearing it is a deliberate act.

Attackers use the same flag in the other direction. A dropped cron job, systemd unit or SSH key pinned with +i survives your cleanup: rm as root answers Operation not permitted, which looks like a permissions problem until you run lsattr. So sweep for attributes before eradication, while you are still recording the scene, not after a removal mysteriously fails:

You have a real root shell. Walk the inspection half of the containment workflow — the part you'd do before any destructive action. Read each output and ask: what would I capture, and what would I contain?

The destructive commands you'd add on a live incident — and which you should not run here without having captured evidence first — are:

• sudo cp /proc/<pid>/exe /evidence/sample.bin — recover the binary from memory
• sudo nft add rule inet filter output ip daddr <C2-IP> drop — sever C2
• sudo kill -9 <pid> — stop the malicious process

Rehearse the order, not just the keystrokes: capture, cut C2, kill, then decide isolation scope.