Process & Network Forensics

Intermediate40 min6 steps

What you'll learn

Read the full process table as a tree and spot a process that doesn't belong
Tie a suspicious process to the port it's listening on with ss
Map a process to its live network connections (including outbound C2) with lsof
Interrogate a process through /proc to recover its real binary, arguments, and the deleted-file evasion trick

Why hunt processes at all
Frame process forensics as triage: a live threat has to run, and running leaves marks.
An attacker who lands on a box almost always needs something running — a backdoor listening for them to connect, a miner burning your CPU, an agent beaconing out to their server. To do its job, that program must exist as a process, and a process can't hide from the kernel that schedules it.
Think of yourself as an ER doctor doing triage. You don't start with the patient's life story; you check vitals — what's consuming resources, what's listening, what's phoning home. Three questions find almost everything:
- What is running that shouldn't be? (ps)
- What is listening for an inbound connection? (ss)
- What is connected out to the internet? (lsof)
Then you confirm your suspect's identity through /proc.
Why it matters
Malware can rename itself, delete its own file, and fake a kernel-thread name — but it cannot stop being a process while it runs. The process table, the socket table, and /proc are the kernel's own bookkeeping. That's why these three tools beat any tool the attacker dropped on the box: you're asking the kernel, not the intruder.
See everything: ps auxf
Read the whole process table as a parent/child tree and pick out the anomaly.
ps aux lists every process on the system; adding f ("forest") draws the parent → child tree so you can see who launched what. Lineage is the tell: a shell spawned by sshd is normal; a high-CPU process whose binary lives in /tmp is not.
```
$ ps auxf
```
```
USER       PID %CPU %MEM    VSZ   RSS TTY   STAT START   TIME COMMAND
root       912  0.0  0.2  15860  9112 ?     Ss   10:21   0:00 /usr/sbin/sshd -D
root      1340  0.0  0.1   6816  3020 ?     Ss   10:22   0:00  \_ sshd: webadmin [priv]
webadmin  1402  0.0  0.1   6816  2110 ?     S    10:22   0:00      \_ sshd: webadmin@pts/0
webadmin  1403  0.0  0.1   8084  4220 pts/0 Ss   10:22   0:00          \_ -bash
root      1001  0.0  0.4  55180 17200 ?     Ss   10:21   0:00 nginx: master process
root      2207 98.7  1.4 712044 58220 ?     Rsl  11:48   7:33 /tmp/.cache/kworkerd
```
a
Show processes for all users, not just your own.
u
User-oriented columns — USER, %CPU, %MEM, START, the full command.
x
Include processes with no controlling terminal (TTY = ?) — daemons and, often, malware.
f
"Forest": draw the process tree so you can see parent/child lineage.
PID 2207 is the outlier. Three red flags at once: 98.7% CPU, a binary under `/tmp/.cache/` (no legitimate service lives there), and a name — kworkerd — chosen to look like the real kernel thread kworker. Note it has no parent in this tree: it wasn't launched by a login shell or a service manager, the way the sshd → bash chain above it was.
Real kernel threads have no path
Genuine kernel workers show up in brackets with no path, like [kworker/0:1], and use ~0% CPU. A process named to imitate one but showing a real executable path (/tmp/.cache/kworkerd) and burning CPU is the impostor. The brackets vs. a path is the giveaway.
Check yourself
In ps auxf output, what does the f flag add that plain ps aux doesn't?
What's listening: ss -tlnp
Connect a suspicious process to the port it opened for inbound access.
A backdoor's whole purpose is to wait for the attacker to connect, which means it holds a listening socket. ss (the modern replacement for netstat) lists sockets, and with -p it names the process behind each one — closing the loop from "weird port" to "which program".
```
$ sudo ss -tlnp
```
```
State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port  Process
LISTEN  0       128      0.0.0.0:22            0.0.0.0:*          users:(("sshd",pid=912,fd=3))
LISTEN  0       128      0.0.0.0:80            0.0.0.0:*          users:(("nginx",pid=1001,fd=6))
LISTEN  0       1        0.0.0.0:31337        0.0.0.0:*          users:(("kworkerd",pid=2207,fd=4))
```
-t
TCP sockets only (where backdoors and most services live).
-l
Listening sockets only — things waiting for an inbound connection.
-n
Numeric: show port 22, not "ssh" — so a service hiding on an odd port can't disguise itself behind a friendly name.
-p
Show the process (name + PID) that owns each socket. Needs root to see other users' processes.
Ports 22 (sshd) and 80 (nginx) are expected. `0.0.0.0:31337` owned by `kworkerd`, pid=2207 is not — and it's the same PID you flagged in ps. You've now tied the high-CPU process to an open inbound door. 0.0.0.0 means it's listening on every interface, i.e. reachable from the network, not just localhost.
No -p, no names
Without sudo, the Process column for sockets you don't own comes back empty — you'll see the port but not who owns it, and may wrongly conclude the box is clean. Always run socket forensics as root.
What's phoning home: lsof -i
Find live outbound connections — the beacon to the attacker's server.
Listening is only half the story. Modern implants also call out to a command-and-control (C2) server — often on 443 to blend in with HTTPS. lsof -i ("list open files: internet") shows every network connection a process holds, including established outbound ones that a listening-only view like ss -l hides.
```
$ sudo lsof -i -P -n
```
```
COMMAND    PID  USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
sshd       912  root    3u  IPv4   14820      0t0  TCP *:22 (LISTEN)
nginx     1001  root    6u  IPv4   15003      0t0  TCP *:80 (LISTEN)
kworkerd  2207  root    4u  IPv4   18402      0t0  TCP *:31337 (LISTEN)
kworkerd  2207  root    5u  IPv4   18500      0t0  TCP 10.0.0.7:54122->198.51.100.55:443 (ESTABLISHED)
```
-i
Show only internet (network) sockets, not regular open files.
-P
Don't resolve port numbers to names — show 443, not "https".
-n
Don't reverse-DNS addresses — show the raw C2 IP, and avoid a slow/blocking lookup.
The last line is the smoking gun: PID 2207 holds an `(ESTABLISHED)` connection from this host (10.0.0.7) out to `198.51.100.55:443`. That's the live C2 channel — the -> points outbound. The same process is both listening on 31337 and beaconing to an external IP. You now have the attacker's infrastructure address for blocking and threat-intel.
ss can do this too
sudo ss -tnp state established shows the same established connections with owning PIDs. ss is faster and always installed; lsof is richer (it also ties in regular open files). Knowing both means you're never stuck when one is missing from a stripped-down box.
Check yourself
In lsof output, an arrow like 10.0.0.7:54122->198.51.100.55:443 (ESTABLISHED) tells you the connection is going in which direction?
Ground truth: /proc/PID
Confirm the suspect's real identity and catch the deleted-binary evasion.
Everything so far trusts the name the process reported. /proc/<PID>/ is the kernel's own view of that process, and it can't be faked from user space. Three files settle the case:
- exe → a symlink to the actual binary on disk
- cmdline → the exact arguments it was launched with
- cwd → its working directory
Example — not runnable on your lab box
```
$ sudo ls -l /proc/2207/exe
```
```
lrwxrwxrwx 1 root root 0 Jun 23 11:50 /proc/2207/exe -> '/tmp/.cache/kworkerd (deleted)'
```
Two findings. The real binary is /tmp/.cache/kworkerd — confirming the ps path over the faked kworker name. And `(deleted)` means the attacker unlinked the file from disk after launching it, so ls /tmp/.cache/ would show nothing. The process keeps running from memory, and a defender grepping the filesystem finds an empty directory.
Why it matters
A (deleted) executable is one of the strongest single indicators of compromise. Legitimate services don't delete their own binary and keep running. Better still, you can recover the malware straight from memory: sudo cp /proc/2207/exe /evidence/kworkerd.bin copies the live binary out for analysis even though it's gone from disk.
```
$ sudo tr '\0' ' ' < /proc/2207/cmdline; echo
```
```
/tmp/.cache/kworkerd --donate-level=0 -o pool.miner.example:443 --tls
```
cmdline stores arguments separated by NUL bytes, so tr '\0' ' ' makes them readable. The flags expose intent without guesswork: this is a cryptominer pointed at a mining pool. Case closed — you have the binary path, the C2/pool address, and the proof of purpose.
Check yourself
ls -l /proc/PID/exe shows the target ends in (deleted). What does that tell you?
Try it in your live terminal
Do the triage yourself on the real sandbox to your right.
You have a real root shell. Run this triage sequence in order — the same workflow you'd use on a real "this box feels slow / looks compromised" call. There may be nothing malicious here; the point is to learn the moves so they're reflex when it counts.
```
$ ps -eo pid,pcpu,user,comm --sort=-pcpu | head
```
Top CPU consumers first. Note the PID of anything surprising — that's your candidate to investigate.
```
$ sudo ss -tlnp
```
Every listening TCP port and the process behind it. Ask of each: do I recognize this service and this port?
```
$ sudo lsof -i -P -n | grep ESTABLISHED
```
Live outbound/inbound connections. An established link to an unfamiliar external IP is worth chasing.
Example — not runnable on your lab box
```
$ sudo ls -l /proc/<PID>/exe
```
Swap <PID> for a candidate from the steps above. You're checking the real binary path — and watching for the (deleted) flag.

Recap

A live threat must run — so it shows up in the process table, the socket table, and /proc. Ask the kernel, not the attacker's tools.
Lineage and location expose planted processes: no legitimate parent, a binary in /tmp, a name mimicking a kernel thread.
ss -tlnp ties a port to a PID (inbound); lsof -i reveals established outbound C2. Listening-only views hide the beacon.
/proc/PID/exe is ground truth — and a (deleted) target both proves evasion and lets you recover the binary from memory.

Tool → what it reveals
Command	Answers	Watch for
`ps auxf`	What's running, and who launched it	High CPU, /tmp paths, no parent, faked names
`ss -tlnp`	What's listening, on which port, owned by which PID	Odd ports (31337), 0.0.0.0 binds
`lsof -i -P -n`	Live connections per process	ESTABLISHED outbound to unknown IPs
`ls -l /proc/PID/exe`	The real binary on disk	`(deleted)` — recover it before the process exits
`tr '\0' ' ' < /proc/PID/cmdline`	Exact launch arguments	Miner pools, reverse-shell targets, flags

Process & Network Forensics

Why hunt processes at all

See everything: ps auxf

What's listening: ss -tlnp

What's phoning home: lsof -i

Ground truth: /proc/PID

Try it in your live terminal