Rootkits & Hidden Processes

Hard40 min7 steps

What you'll learn

Distinguish userland rootkits (trojaned binaries that lie) from kernel rootkits (hooked syscalls that make every tool lie)
Apply discrepancy detection — compare two views that should agree to expose what's being hidden
Run rkhunter and chkrootkit and read their findings without over-trusting a clean result
Inspect loaded kernel modules with lsmod and /proc/modules for a hidden hook
Know the escalation boundary: when to stop trusting the box, boot clean media, and reimage

Why this is the hardest IR problem
Frame rootkit hunting as the moment you can no longer trust the box's own tools.
Every forensics skill you've learned so far rests on a quiet assumption: that ps, ls, and ss tell you the truth. A rootkit is malware built to break exactly that assumption. Its entire job is to hide its own presence — the files, processes, ports, and logins of the attacker — so your normal tools report a clean system that is anything but.
Think of it as asking a suspect to investigate their own crime. You hand the prime suspect your notebook and ask, "Did anyone break in here?" Of course the answer is no. You cannot trust a witness who is also the perpetrator. To get the truth you need an outside witness — something the rootkit doesn't control.
That one idea drives this entire lesson. We'll separate rootkits into two classes by how deep they sit, learn the single technique that catches both, run the standard scanners, check the kernel's module list, and finish at the hard boundary where the only honest witness is a machine booted from clean media.
Why it matters
This is where incident response stops being a checklist and becomes epistemology: how do I know what I know? On a possibly-rooted box, the answer to "is this output real?" is never automatically yes. Internalizing that distrust is the whole skill — the commands are just how you act on it.
The two classes: userland vs. kernel
Tell apart a rootkit that lies through trojaned binaries from one that lies through the kernel itself.
Rootkits hide at one of two depths, and the depth decides how hard they are to catch.
Userland rootkits replace honest programs with trojaned binaries. The attacker swaps your real /bin/ls, /bin/ps, or /bin/netstat for doctored copies that work normally — except they silently filter out anything matching the attacker's secret list. A trojaned ps simply omits PID 6613 from its output; a trojaned ls skips the .hidden directory. The kernel still knows the truth; only the program you're asking has been corrupted.
Kernel rootkits go a layer deeper. Loaded as a kernel module (LKM), they hook syscalls — they rewrite the kernel's own answer to getdents() (directory reads) or readdir, so the filtering happens below every program. Now even a pristine, freshly-compiled ps gets a lie, because the lie is baked into the data the kernel hands it.
Why depth is everything
Against a userland rootkit you can win by bringing your own trusted binaries (a known-good ps from clean media). Against a kernel rootkit that trick fails — your trusted ps makes an honest syscall and gets a poisoned answer. The deeper the hook, the more of your toolbox is compromised, and the less you can verify from inside the running system.
An analogy ties them together. A userland rootkit is a bribed clerk: ask the clerk to read the ledger and he skips the incriminating line, but the ledger itself is intact. A kernel rootkit has forged the ledger — every honest clerk who reads it now repeats the same lie, because the source is corrupt.
Check yourself
You boot a known-good statically-linked ps binary from a USB stick and it still hides a process you can prove exists. Userland or kernel rootkit?
Discrepancy detection: the core technique
Catch a hidden process or file by comparing two views that should agree but won't.
Here is the one technique that catches both classes: compare two independent views of the same fact. If they should agree but don't, the gap is the rootkit's lie made visible.
The reason this works: a rootkit author has to patch every path that could reveal them, and that's hard. Maybe they trojaned ps but forgot that ls /proc also enumerates running processes — each running PID has a /proc/<pid> directory. So you ask the same question two ways and watch for a number that doesn't match.
A running process must have a /proc/<pid> directory (the kernel creates it). So every PID under /proc should also appear in ps. List them both and diff:
```
$ comm -23 <(ls /proc | grep -E '^[0-9]+$' | sort) <(ps -e --no-headers -o pid | tr -d ' ' | sort)
```
```
6613
```
comm -23
Show lines unique to the FIRST input — PIDs in /proc that ps never listed. Suppresses columns 2 (right-only) and 3 (shared).
<(...)
Process substitution: feed each command's output to comm as if it were a file, so you diff two live views inline.
grep -E '^[0-9]+$'
Keep only the numeric directory names under /proc — i.e. the PIDs — and drop entries like 'cpuinfo' or 'modules'.
An empty result means the two views agree — every /proc PID is also in ps. Here PID 6613 exists in /proc but ps refuses to show it. The kernel created its directory (it's really running) yet ps denies it: a hidden process. You found it not by trusting either tool, but by catching them disagreeing.
The same move works for hidden files. The shell's * glob and the ls binary build their listings differently — if a rootkit trojaned ls but not your shell, they'll disagree on directory contents. A blunt cross-check is a raw count: compare how many entries two methods see.
```
$ echo /var/tmp/*; ls -a /var/tmp
```
```
/var/tmp/systemd-private-aa /var/tmp/.ICE-unix
.  ..  systemd-private-aa  .ICE-unix  ...quota.user
```
The shell glob (echo *) and a trojaned ls can be made to lie differently. Here ls -a reveals ...quota.user — a name using leading dots to blend in — that wouldn't show in a naive listing. When two enumerations of the same directory disagree, the extra entry is what someone wanted hidden. (A clean box agrees on both, allowing for * not matching dotfiles — that's expected, not a discrepancy.)
Trusting a single tool on a suspect box
The classic fatal error: you run ps, see nothing weird, and declare the box clean. On a rooted host that's exactly the result the attacker engineered. One tool's word is never evidence of innocence here. Evidence comes from agreement between independent views — or the lack of it. No discrepancy found is reassuring only if you checked from more than one angle.
Check yourself
A PID appears as a directory under /proc but ps -e never lists it. What single word best describes that PID?
Known-signature scanners: rkhunter & chkrootkit
Run the standard scanners and read their output without over-trusting a clean pass.
Discrepancy detection finds unknown hiding. The other half of the job is checking for known rootkits cheaply, and two tools do that: `rkhunter` (Rootkit Hunter) and `chkrootkit`. Both ship a database of signatures — known malicious filenames, trojaned-binary fingerprints, telltale strings — and scan your system against it, the way antivirus matches known malware.
Example — not runnable on your lab box
```
$ sudo rkhunter --check --sk
```
```
[ Rootkit Hunter version 1.4.6 ]

Checking system commands...
  /usr/bin/ls                  [ OK ]
  /usr/bin/ps                  [ Warning ]
Checking for rootkits...
  Suckit Rootkit               [ Not found ]
  Possible rootkit strings     [ Warning ]

System checks summary
  Possible rootkits: 1
  Files checked: 142    Suspect files: 2
```
--check
Run the full scan: hash-compare system binaries, hunt known rootkit artifacts, scan for suspicious strings.
--sk
(--skip-keypress) Don't pause between sections — lets the scan run unattended in a script or capture.
Read the Warnings, not the OKs. /usr/bin/ps flagged as [ Warning ] means its hash no longer matches rkhunter's known-good record — a strong hint the binary was trojaned. "Possible rootkits: 1" is your cue to investigate, not panic: rkhunter has false positives (a legit package update also changes a hash). It points you where to look; it doesn't close the case.
Example — not runnable on your lab box
```
$ sudo chkrootkit | grep -v 'not found\|not infected'
```
```
Checking `ps'... INFECTED
Checking `lkm'... You have    2 process hidden for ps command
Searching for suspicious files... /usr/lib/.libfk found
```
| grep -v 'not found...'
Filter out the long list of clean results so only INFECTED / suspicious lines remain — the signal, not the noise.
chkrootkit agrees from a second angle: `ps` INFECTED, and its own lkm check reports 2 processes hidden — chkrootkit ran its own enumeration and compared it to ps, the same discrepancy logic you applied by hand. Two independent scanners flagging ps is far stronger than one.
A clean scan does NOT prove innocence
The most dangerous misread in this lesson: rkhunter and chkrootkit only know published rootkits. A custom or fresh kernel rootkit produces a clean report while fully owning the box — and worse, a kernel rootkit can hide its own files from the scanner. "rkhunter found nothing" means "no known rootkit was detected," never "this system is safe." Treat a clean pass as one weak data point, not a verdict.
Check yourself
rkhunter --check finishes with zero warnings. Is it safe to conclude the host has no rootkit?
Check the kernel's own module list
Inspect loaded modules for the LKM a kernel rootkit must load to hook syscalls.
A kernel rootkit usually arrives as a loadable kernel module (LKM) — code injected into the running kernel to install its syscall hooks. So the loaded-module list is a natural place to look. lsmod is the friendly view; /proc/modules is the raw kernel source it reads from.
```
$ lsmod | head
```
```
Module                  Size  Used by
nf_conntrack          172032  2 nf_nat
bridge                307200  1 br_netfilter
intel_rapl_msr         20480  0
xt_MASQUERADE          20480  1
diamorphine            16384  0
```
lsmod
Pretty-print the currently loaded kernel modules — name, memory size, and reference count — by reading /proc/modules.
| head
Show only the first lines; a real list is long, and you're scanning for a name that doesn't belong.
Most entries are recognizable kernel subsystems (nf_conntrack, bridge, networking). The outlier is `diamorphine` — the name of a well-known open-source LKM rootkit, with a Used by count of 0 (nothing legitimate depends on it). An unfamiliar module is your prompt to research what it is before trusting the system. Caveat: a sophisticated kernel rootkit can also unlink itself from this list so lsmod won't show it at all — another reason no single check is conclusive.
```
$ grep -iE 'diamorphine|reptile|knark|adore' /proc/modules || echo 'no known-rootkit module name matched'
```
```
diamorphine 16384 0 - Live 0x0000000000000000 (OE)
```
grep -iE
Case-insensitive extended-regex match for several notorious LKM-rootkit names in one pass.
(OE)
Module taint flags in the output: O = out-of-tree (not shipped with the kernel), E = unsigned. Legit third-party modules can show these too, but on a rootkit hunt they raise the priority.
Reading /proc/modules directly (rather than via lsmod) is the same trust-the-kernel-source instinct from earlier. The (OE) taint flags — out-of-tree and unsigned — fit a module nobody intentionally installed. Combined with the rootkit-family name, this is strong corroboration of a kernel-level compromise.
There is a catch: a capable LKM rootkit unlinks itself from the kernel's module list right after it loads, so it vanishes from lsmod and /proc/modules alike. What it cannot take back is what the kernel said while loading it. Loading an out-of-tree module, or one whose signature can't be verified, taints the kernel, and the kernel announces that in its ring buffer, which dmesg prints. The module name is right there in the message. The taint is also sticky: the bitmask in /proc/sys/kernel/tainted stays non-zero until reboot, however hard the module hides. So dmesg is the second, independent view: a name that tainted the kernel at load time but is missing from the module list now is a module that hid itself.
Example — not runnable on your lab box
```
$ sudo dmesg | grep -iE 'taint|verification failed'
```
```
[ 4211.889102] diamorphine: loading out-of-tree module taints kernel.
[ 4211.889215] diamorphine: module verification failed: signature and/or required key missing - tainting kernel
```
dmesg
Print the kernel ring buffer: boot messages, driver and module loads, and kernel warnings such as taints.
taint|verification failed
The two kernel messages logged when an out-of-tree or unverifiable module loads.
What a rooted host's ring buffer looks like (it needs a real kernel with the module loaded, so this one doesn't run on the lab box). Two lines, both naming the module: it was built outside the kernel tree, and its signature couldn't be verified. A vendor driver that is out-of-tree but properly signed logs only the first line, so the verification failure is the sharper tell. Cross-check every name these lines give you against lsmod: tainted at load and absent from the list is the self-hiding rootkit. On a captured case the same grep works on a saved copy (dmesg > dmesg.txt at collection time), which is also why you capture dmesg early: the ring buffer is fixed-size and old lines scroll out.
Check yourself
What kind of object does a kernel rootkit typically load into the running kernel to install its syscall hooks?
The hard truth: when to stop trusting the box
Recognize the escalation boundary — clean-media examination and reimaging.
Now the uncomfortable conclusion. A kernel rootkit subverts everything in user space. Every command in this lesson runs through the kernel — so on a fully compromised box, even your discrepancy checks can be fed consistent lies, and the rootkit can hide its module, its files, and its processes from all of them at once. You're back to the suspect investigating their own crime, and this time the suspect controls the room.
High-assurance detection therefore has to come from outside the running system — the outside witness from step one, made literal:
- Boot from clean external media (a trusted live USB/CD). Now the suspect kernel isn't running, so it can't lie.
- Examine the disk offline — mount it read-only from the clean environment and inspect the real files on disk, with the rootkit dormant and unable to filter what you see.
- Compare against known-good hashes/images to find what was altered.
This is an escalation boundary, not a step in the same workflow
When evidence points to a kernel rootkit, in-band investigation has reached its limit and you escalate: isolate the host from the network, preserve memory/disk for forensics, and move to offline analysis. Continuing to poke at the live box risks tipping the attacker and trusting poisoned output. Know where your live-system tools stop being reliable — that knowledge is itself the deliverable.
When in doubt, reimage
Proving a box is clean after kernel compromise is often impossible and rarely worth the effort versus the risk of a missed implant. The pragmatic answer is reimage from trusted media and restore from known-good backups. It feels like giving up; it's actually the only way to regain the trust a rootkit destroyed. Cheaper, faster, and more certain than chasing a ghost you can't fully see.
Check yourself
Why must high-assurance kernel-rootkit detection be done by booting from clean external media instead of from the installed OS?
Try it in your live terminal
Run the safe discrepancy and module checks yourself on the real sandbox.
You have a real root shell. These commands are safe and read-only — they inspect, they don't change anything. This sandbox is almost certainly clean, so expect agreement (empty discrepancies, recognizable modules). The point is to make the moves reflex, so on a real suspect box you reach for two views by habit, never one.
```
$ ls -d /proc/[0-9]* | wc -l; ps -e --no-headers | wc -l
```
Count running processes two independent ways: by counting numeric directories under /proc, then by counting ps lines. On a clean box these are equal (or differ by 1-2 as processes start/exit between the two reads). A persistent, larger gap — /proc seeing more than ps — is the hidden-process signature.
```
$ comm -23 <(ls /proc | grep -E '^[0-9]+$' | sort) <(ps -e --no-headers -o pid | tr -d ' ' | sort)
```
The precise version: list the actual PIDs each view sees and print those in /proc but missing from ps. Empty output = the two witnesses agree. Any PID printed here is one to investigate.
```
$ echo /etc/*; ls /etc | wc -l
```
Cross-check a directory two ways — shell glob versus ls. On a clean box they describe the same contents. The habit you're building: never accept one enumeration of a directory as the whole truth on a suspect host.
Example — not runnable on your lab box
```
$ lsmod | head; grep -ciE 'diamorphine|reptile|knark|adore' /proc/modules
```
Skim the loaded modules and scan /proc/modules for known LKM-rootkit names. A 0 from the grep means no known-rootkit module name matched. Remember the caveat: a clean module list is reassuring but not conclusive — a kernel rootkit can hide itself from it.

Recap

A rootkit's purpose is to make your own tools lie. On a suspect box, no single tool's output is evidence of innocence — only agreement (or disagreement) between independent views.
Userland rootkits trojan binaries (ls/ps/netstat); a trusted binary defeats them. Kernel rootkits hook syscalls, so even clean tools are lied to — depth decides difficulty.
Discrepancy detection is the core skill: ask the same question two ways (/proc vs ps, glob vs ls) and treat any gap as the hidden thing made visible.
rkhunter/chkrootkit catch known rootkits; lsmod//proc/modules reveal a suspect LKM. A clean scan or clean module list is one weak data point, never a verdict.
A module can unlink itself from lsmod, but not from the taint it caused: dmesg keeps the 'taints kernel' / 'verification failed' lines naming it, so tainted-but-unlisted is the self-hiding rootkit.
A kernel rootkit subverts all of user space — escalate to clean-media boot, offline disk analysis, and reimage when in doubt. Knowing where your live tools stop being trustworthy is the deliverable.

Rootkit type → how it hides → how you catch it
Rootkit type	How it hides	How you catch it
Userland (trojaned binaries)	Doctored `ls`/`ps`/`netstat` filter out the attacker's files, processes, ports	Discrepancy vs an honest view (`/proc` vs `ps`, glob vs `ls`); known-good binary from clean media; rkhunter hash mismatch
Kernel (LKM / syscall hooks)	Hooks `getdents`/readdir etc. so even honest tools get poisoned answers	Suspect module in `lsmod`/`/proc/modules`; chkrootkit `lkm` count; ultimately clean-media boot + offline disk exam
Hidden process	Omitted from `ps` though `/proc/<pid>` exists and it's really running	`comm -23` of /proc PIDs vs `ps` PIDs; process counts that disagree
Hidden file	Leading-dot/odd names filtered from directory listings	Shell glob (`echo *`) vs `ls -a`; entry/count mismatch on the same directory
Fully compromised kernel	Subverts every user-space tool at once; may unlink its own module	Stop trusting the box: boot clean media, examine disk offline, reimage when in doubt

Rootkits & Hidden Processes

Why this is the hardest IR problem

The two classes: userland vs. kernel

Discrepancy detection: the core technique

Known-signature scanners: rkhunter & chkrootkit

Check the kernel's own module list

The hard truth: when to stop trusting the box

Try it in your live terminal