Linux Forensics Fundamentals

Intermediate45 min7 steps

What you'll learn

Explain the order of volatility and why volatile evidence is collected before disk
Adopt the first-responder mindset: preserve, then investigate — never the reverse
Hunt the highest-value quick wins: rogue UID 0 accounts and deleted-but-open files
Map where persistence hides so you know which deeper IR lesson to reach for next

You're the first responder
Frame incident response as evidence preservation, and meet the order of volatility.
A box looks compromised and you're the one who got the call. Your instinct is to start fixing — kill the bad process, delete the malware, reboot to be safe. Resist it. The moment you act like a janitor instead of a detective, you destroy the very evidence that tells you what happened, how far it spread, and whether the attacker is still inside.
The first-responder mindset has one rule above all others: preserve before you investigate, investigate before you remediate. Your first job is not to win the fight — it's to take the photographs before the scene changes.
And the scene is changing right now. Different kinds of evidence decay at wildly different speeds. The plan that decides what you grab first is called the order of volatility: collect the things that vanish soonest, soonest.
Why order matters more than thoroughness
A perfect disk image taken after a reboot is worthless for catching a memory-only implant — the implant is already gone. A clumsy-but-fast capture of running processes and live connections, taken before anyone touches the box, can break the whole case open. Sequence beats polish. You only get one shot at the volatile stuff.
Think of a footprint in fresh snow. The footprint is perfect evidence — but the sun is out and it's melting by the minute. A good investigator photographs the footprint first, while it's crisp, and only later measures the fence it leads to. The fence will still be there tomorrow; the footprint won't. Memory, running processes, network connections, and logged-in users are the footprints in the snow. The disk is the fence.
The order of volatility, top to bottom
Learn the canonical evidence hierarchy and read it as a collection plan.
Evidence ranks by how quickly it disappears. From most volatile (grab first) to least (grab last):
- CPU registers & caches — gone in nanoseconds; specialist territory.
- RAM / memory — the running malware lives here; a reboot wipes it completely.
- Process & kernel state — what's running, who's connected, the routing table.
- Network connections & logged-in users — live sessions drop the instant a socket closes or someone logs out.
- Temporary filesystems — /tmp, /run; often tmpfs, meaning they live in RAM and evaporate on reboot.
- Disk — files, logs, configs. Relatively stable; the fence that's still there tomorrow.
- Backups & archives — the most durable, least volatile of all.
Read top to bottom, that list is your collection order.
```
$ uptime; who; date -u
```
```
 14:09:52 up 6 days,  2:14,  2 users,  load average: 0.41, 0.22, 0.09
webadmin pts/0        2026-06-23 09:01 (198.51.100.23)
root     pts/1        2026-06-23 14:07 (203.0.113.77)
Thu Jun 23 14:09:52 UTC 2026
```
uptime
How long since the last boot — tells you whether volatile evidence still survives.
who
Currently logged-in users and the source IP of each session — a live attacker shows up here.
date -u
Timestamp the moment of collection in UTC, so every later artifact lines up on one clock.
Three free seconds of pure volatile evidence. `uptime` proves the box has not rebooted (6 days) — so the volatile layer is still intact and worth collecting. `who` shows live sessions: webadmin is normal, but a root shell from 203.0.113.77 that you didn't open is an active intruder. `date -u` stamps your collection in UTC so your timeline is unambiguous later. Run these before anything else.
The reboot reflex
"Turn it off and on again" is muscle memory for sysadmins and the single most destructive thing you can do to a live investigation. A reboot flushes RAM, kills every process and socket, and wipes tmpfs mounts like /tmp. If the malware never wrote itself to disk, the reboot just deleted your only copy of it. When in doubt, isolate the network, don't power-cycle.
Check yourself
Two pieces of evidence are at risk: a suspicious process's live network connection, and a config file on disk. Which do you collect first, and why?
Quick win #1: rogue root accounts
Find any account that secretly has root power, beyond the real root.
Once you've snapshotted the volatile layer, go for high-value quick wins — checks that take one line and frequently catch the attacker red-handed. The first: hidden privilege.
On Linux, what makes an account root is not the name root. It's the UID (user ID) being 0. The kernel grants total power to UID 0 — the name beside it is just a label. So a classic backdoor is to add a second account, give it UID 0, and name it something boring like webupdate or systemd-sync. It logs in as an ordinary-looking user and is root.
```
$ awk -F: '($3==0){print $1}' /etc/passwd
```
```
root
toor
```
-F:
Split each line on the colon — /etc/passwd is colon-separated fields.
$3==0
Field 3 is the UID; match only lines where it equals 0 (root-level).
print $1
Print field 1, the username, of each matching line.
This prints the name of every account whose UID is 0 — every account with root power. On a clean system the answer is exactly one line: root. Here a second name, `toor`, also has UID 0. That's a rogue root account: a hidden administrator the attacker planted so they keep their access even if the real root password is changed.
Why UID, not name
Beginners scan /etc/passwd for the word root and stop. The kernel doesn't care about the word — it checks the number. UID 0 is the privilege; the name is cosmetic. Always audit by UID. The same logic applies to GID 0 and to sudo/wheel group membership, which are the next places to look once you've cleared the passwd file.
Check yourself
An account named backup has UID 0 in /etc/passwd. Does it have root privileges?
Quick win #2: deleted-but-still-open files
Catch malware that unlinked its own file from disk but is still running.
The second quick win exploits a Unix quirk attackers love. When you rm a file that a running program still has open, the directory entry disappears — but the data stays alive on disk until the last open handle closes. So malware launches itself, then deletes its own binary or its log file. To anyone browsing the filesystem, it's gone. To the kernel, it's still right there, held open by a live process.
lsof ("list open files") can find these ghosts. The +L1 selector means "show files whose link count is less than 1" — i.e. files with zero names left on disk, but still open.
```
$ sudo lsof +L1
```
```
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
nginx     1001 root   8w   REG  253,0   145220     1  39021 /var/log/nginx/access.log
systemd-j 410  root  mem   REG  253,0  8388608     0 131884 /tmp/.x11-unix/.sess (deleted)
systemd-j 410  root   3u   REG  253,0   204800     0 131902 /dev/shm/.payload (deleted)
```
+L1
Show only open files whose on-disk link count is below 1 — i.e. deleted but still held open.
sudo
Required to see files opened by other users' processes; without root the malware's handles stay invisible.
Read the NLINK column: it's the number of names the file still has on disk. The first line (NLINK 1) is a normal open log file — fine. The next two have NLINK `0` and say `(deleted)`: a process calling itself systemd-j (PID 410) is holding open two files with no name left on disk — one in /tmp, one in /dev/shm. Legitimate software does not run from deleted binaries in shared memory. That's your implant, hiding in plain sight.
You can still recover it
A deleted-but-open file isn't lost to you either. The kernel still serves its contents through /proc/<PID>/fd/. Copy it out for analysis — sudo cp /proc/410/fd/3 /evidence/payload.bin — and you've recovered the live malware straight from memory, even though ls swears the file doesn't exist. The deeper Process & Network Forensics lesson drills this /proc recovery in full.
Check yourself
In lsof +L1 output, what does a value of 0 in the NLINK column tell you about the file?
Quick win #3: the attacker's cleanup, in their own history
Spot anti-forensics log wiping in shell history, and tell destroying a log from reading one.
Attackers know the logs will name them, so a common last move is anti-forensics: destroy the evidence before leaving. The crude version is everywhere, because it needs no tooling: empty the log file in place. The file still exists, with the right owner and permissions, so a glance at ls /var/log looks normal; it just holds nothing. Three spellings do it, and all of them are worth recognising on sight:
- > /var/log/auth.log or : > /var/log/auth.log: redirect nothing into the file, truncating it to zero bytes.
- truncate -s 0 /var/log/auth.log: the same thing with a dedicated tool (-s 0 sets the size to zero).
- cat /dev/null > /var/log/auth.log: the old-school spelling of the same truncation.
The irony is that the attacker typed these into a shell, and unless they also disabled history, `~/.bash_history` recorded the wipe. The command names the exact log that was destroyed, which tells you which evidence to go recover from elsewhere (the journal, a remote syslog server, a backup).
```
$ printf '%s\n' 'tail -n 50 /var/log/auth.log' 'truncate -s 0 /var/log/wtmp' 'grep -c Failed /var/log/auth.log' ': > /var/log/lastlog' > history.txt
grep -nE 'truncate|^ *:? *>|/dev/null *>' history.txt
```
```
2:truncate -s 0 /var/log/wtmp
4:: > /var/log/lastlog
```
truncate
In the pattern: matches the truncate tool, as in truncate -s 0 FILE.
^ *:? *>
A line that starts with a redirection (optionally after the : no-op): > file or : > file, which empties the file.
/dev/null *>
Catches cat /dev/null > file, the same truncation spelled the long way.
A four-line stand-in for a root history file. The grep keeps the lines that destroy a file and skips the ones that only read one: tail and grep -c against auth.log are the attacker looking around, which leaves the log intact. What it found are two wipes, wtmp emptied with truncate and lastlog with a bare : >. Those are the logs you can no longer trust on this box. On a real case, point it at /root/.bash_history and every /home/*/.bash_history.
Reading a log is not wiping it
An attacker's history is full of log paths, because they read logs to see whether they were noticed. tail /var/log/auth.log and grep sshd /var/log/syslog mention a log without touching it. Only a command that writes to the file (a redirection into it, truncate, shred) destroys evidence. Report the log that was wiped, not every log that was named, or your finding sends the next responder chasing logs that are intact.
Check yourself
A root history contains tail -f /var/log/syslog and, later, cat /dev/null > /var/log/auth.log. Which log was tampered with?
Where persistence hides: the checklist
Map the common persistence locations so you know what to sweep and which lesson goes deep.
An attacker's last worry is surviving a reboot or a password change — that's called persistence. The good news for you: there are only so many places Linux will automatically run something, so persistence hides in a predictable, finite set of locations. This is your sweep checklist:
- Cron — /etc/crontab, /etc/cron.d/, /var/spool/cron/, and per-user crontab -l. A job that re-launches a backdoor every minute.
- systemd units — /etc/systemd/system/, ~/.config/systemd/user/, and timers. A rogue .service masquerading as a system daemon.
- SSH keys — ~/.ssh/authorized_keys for every user. One added public key is permanent password-free access.
- rc.local & init scripts — /etc/rc.local, /etc/init.d/. The old-school boot hook, still honored.
- Shell rc files — ~/.bashrc, ~/.bash_profile, /etc/profile.d/. Code that runs every time someone opens a shell.
You don't have to memorize the exact commands today — you have to know that these five doors exist, so you check all of them and don't declare victory after closing one.
```
$ for u in $(cut -d: -f1 /etc/passwd); do k=/home/$u/.ssh/authorized_keys; [ -f "$k" ] && echo "== $u ==" && cat "$k"; done
```
```
== webadmin ==
ssh-ed25519 AAAAC3Nz...realkey webadmin@laptop
ssh-rsa AAAAB3Nz...c2hlbGw= mailer@localhost
```
cut -d: -f1
Pull field 1 (the username) from each colon-separated /etc/passwd line, to iterate every account.
[ -f "$k" ]
Only act when that user actually has an authorized_keys file, so the loop stays quiet otherwise.
This walks every user and prints their authorized_keys. webadmin's first key (their laptop) is expected. The second — labeled `mailer@localhost` — is the tell: a key the user never added, granting silent, password-free SSH to whoever holds the matching private half. Comparing the comment and count of keys against what each user expects is how you spot the plant. This is one door of five — the same loop logic applies to cron and the rest.
One door closed is not the building secured
The expensive mistake in IR is finding a persistence mechanism, removing it, and declaring the box clean. Competent attackers plant two or three — a cron job and an SSH key and a systemd timer — precisely so that cleaning one leaves the others to re-establish access. Sweep every door on the checklist, every time, before you call it remediated.
This lesson is the map, not the territory
Each door here is a full lesson of its own: the systemctl and cron lessons cover service and scheduled-task persistence; Process & Network Forensics goes deep on running implants and /proc recovery; the access-control and hardening lessons cover users, keys, and sudo. Today you learned the method and the order. Those lessons are where you drill each step until it's reflex.
Try it in your live terminal
Run the first-responder sweep yourself, in order, on the real sandbox.
You have a real root shell to your right. Run this sequence in order — volatile first, persistence last — exactly the discipline you'd follow on a real call. The box may be perfectly clean; the goal is to make the moves reflex so the order is automatic when it counts.
```
$ uptime; who; date -u
```
Step one, always: confirm the volatile layer survives (no recent reboot), capture live sessions, and stamp the clock in UTC.
```
$ awk -F: '($3==0){print $1}' /etc/passwd
```
Quick win #1: list every UID 0 account. Anything beyond root is a rogue administrator to investigate.
```
$ sudo lsof +L1
```
Quick win #2: surface deleted-but-open files. Watch the NLINK column for 0 and the (deleted) tag — a process running from a file with no name on disk.
```
$ for u in $(cut -d: -f1 /etc/passwd); do k=/home/$u/.ssh/authorized_keys; [ -f "$k" ] && echo "== $u ==" && cat "$k"; done
```
Persistence sweep, door one of five: read every user's authorized_keys and question any key the user can't account for. Then continue down the checklist — cron, systemd, rc.local, shell rc files.

Recap

Preserve before you investigate, investigate before you remediate — you're a detective first, a janitor last.
The order of volatility decides collection order: memory, processes, connections, and logged-in users are footprints in the snow; the disk is the fence. Grab the footprints first.
A reboot is the most destructive reflex in IR — it flushes RAM, kills sockets, and wipes tmpfs. Isolate the network instead.
Privilege is the UID, not the name: any UID 0 account is root. A deleted-but-open file (NLINK 0) is a process hiding from the filesystem.
Anti-forensics often means emptying a log in place (> file, : > file, truncate -s 0, cat /dev/null >); shell history usually records the wipe and names the log. Reading a log is not tampering with it.
Persistence lives in a finite set of doors — cron, systemd, SSH keys, rc.local, shell rc files. Sweep all of them; attackers plant more than one.

Command → what it reveals
Command	Reveals	Watch for
`uptime; who; date -u`	Whether volatile evidence survives, and who's logged in	No recent reboot; an unexpected root session from a strange IP
`awk -F: '($3==0){print $1}' /etc/passwd`	Every account with root power (UID 0)	Any name besides `root`
`sudo lsof +L1`	Files deleted from disk but still held open	NLINK 0 and `(deleted)`, especially in /tmp or /dev/shm
`grep -nE 'truncate\|^ :? >' ~/.bash_history`	Commands that emptied a file	A log path after `>`, `: >` or `truncate -s 0`
`crontab -l` / `/etc/cron.d/`	Scheduled persistence	Jobs re-launching unknown binaries
`cat ~/.ssh/authorized_keys`	Password-free SSH access per user	Keys the user never added
`~/.bashrc`, `/etc/rc.local`	Code that runs at shell start or boot	Appended one-liners calling out to the network

Linux Forensics Fundamentals

You're the first responder

The order of volatility, top to bottom

Quick win #1: rogue root accounts

Quick win #2: deleted-but-still-open files

Quick win #3: the attacker's cleanup, in their own history

Where persistence hides: the checklist

Try it in your live terminal