Signal9Signal9 — home
online
all lessons

Linux Memory Forensics: Acquisition & Analysis

Advanced45 min7 steps
What you'll learn
  • Explain why RAM is the most volatile, most revealing evidence on a live host — and why you capture it first
  • Acquire a memory image safely on a modern kernel with LiME or AVML, recording the exact kernel version for analysis
  • Walk a Volatility-style workflow: list processes, catch a hidden one with pslist vs psscan, dump a process, find injected code, and recover network connections
  • Fall back to the /proc surface for a live memory approximation when full RAM capture isn't available
  1. Why memory is the evidence that matters most

    Understand what RAM holds that disk never sees, and why it's the first thing you capture.

    Everything a program actually does happens in memory. The disk only holds what got saved; RAM holds what is happening right now — and that gap is where the best evidence lives.

    Think of it this way: disk is the filing cabinet, memory is the desk mid-work. The cabinet shows you what was filed away — files, logs, installed packages. The desk shows you the documents spread open this second: the half-written letter, the calculator still on, the page someone is reading. A thief can empty the filing cabinet and shred everything in it, but while they're standing there working, they can't hide what's open on the desk in front of them.

    Concretely, memory — and only memory — reliably holds:

    • Decrypted data: passwords, keys, and TLS session material that exist as ciphertext (or not at all) on disk.
    • Injected code: malware that lives only in another process's address space and never touches the filesystem.
    • Unlinked-but-running programs: a binary deleted from disk after launch keeps running from RAM (the (deleted) trick from process forensics).
    • Live network connections: sockets and the peer addresses a process is talking to.
    • The true process list: the kernel's real task list, which a userland rootkit's trojaned ps hides from you.

    Check yourself

    Why is RAM captured before disk in the order of volatility?

  2. Why you can't just copy /dev/mem

    Understand why modern kernels lock down raw memory access and what tools fill the gap.

    Your instinct from file forensics is to dd the evidence to a file. For memory the obvious target would be /dev/mem (raw physical RAM) or /dev/kmem (kernel virtual memory) — and on an old box you could. On a modern kernel you cannot, by design.

    Two kernel hardening features stand in the way. `CONFIG_STRICT_DEVMEM` restricts /dev/mem to a tiny window of legacy device ranges, so reading general RAM through it just fails. `/dev/kmem` is gone entirely on current kernels (CONFIG_DEVKMEM removed). These exist because unrestricted raw-memory access is itself a devastating attack primitive — the same power you want for forensics is the power an attacker wants to read everyone's secrets.

    So memory acquisition needs a tool that runs inside kernel context to read all of physical RAM legitimately. Three approaches matter:

    • LiME (Linux Memory Extractor): a loadable kernel module you insert with insmod. Because it runs in the kernel, it can read all of physical memory and write it out as a clean image. The forensic standard for Linux.
    • AVML (Acquire Volatile Memory for Linux): a portable userspace tool from Microsoft that reads memory via the available source (/proc/kcore, /dev/crash, or /dev/mem) without needing you to compile a module per-kernel. Great when you can't build a module on the target.
    • `/proc/kcore`: the running kernel's memory exposed as a virtual ELF core file. Older and more limited, but it's already there — no module, no install — which makes it a useful fallback and a live-analysis surface.

    Example — not runnable on your lab box

    sudo insmod ./lime.ko "path=/mnt/evidence/mem.lime format=lime"
    [  +0.000000] LiME: module loaded
    [  +0.412233] LiME: dumping memory in lime format
    [  +8.910558] LiME: wrote 0x100000000 bytes to /mnt/evidence/mem.lime
    [  +8.910560] LiME: memory dump complete
    insmod ./lime.ko
    Insert the LiME kernel module so its acquisition code runs in kernel context and can read all of physical RAM.
    path=/mnt/evidence/mem.lime
    Where to write the image. Note it points at MOUNTED EXTERNAL storage, not the suspect's own disk.
    format=lime
    Write the native LiME format — physical memory plus a small header recording each memory range — which Volatility reads directly.
    The module loads, streams all of physical memory to the image, and reports the byte count (here 0x100000000 = 4 GiB). The result is a forensic memory image. The critical detail is the target path: /mnt/evidence/ is external media, never the host's own disk — see the next callout for why.

    Check yourself

    On a hardened modern kernel, why does copying raw RAM out of /dev/mem fail?

  3. Record the kernel version — the #1 gotcha

    Capture the exact kernel build so analysis has a matching symbol table.

    A raw memory image is just a few gigabytes of undifferentiated bytes. To turn it into processes, sockets, and command lines, the analysis tool needs a map of where the kernel keeps its structures in that specific build — a symbol table / profile. Get the map even slightly wrong and the structures don't line up, so every plugin either errors out or prints garbage.

    Analogy: the memory image is a city with no street signs. The symbol table is the map that labels which block is the process list and which is the socket table. A map for the wrong city — a kernel that differs by even a point release — points you at the wrong addresses, and nothing makes sense.

    That map is keyed to the exact kernel version and build. So before (or right after) acquisition, you record it. This is one line, and forgetting it is the number-one reason a memory capture becomes unanalyzable.

    uname -r; cat /etc/os-release | grep PRETTY
    6.8.0-45-generic
    PRETTY_NAME="Ubuntu 24.04.1 LTS"
    uname -r
    Print the running kernel release string — the exact build the memory image came from, and the key to the matching profile/symbols.
    grep PRETTY
    Pull the human-readable distro name/version, which together with the kernel pins down exactly which symbols to build.
    6.8.0-45-generic on Ubuntu 24.04.1 is the fingerprint that tells your analysis station precisely which symbol table to generate (with Volatility 3 you build an ISF/symbol file from this kernel's debug symbols). Write this string into your evidence notes alongside the image hash. Without it, the image is a locked box with no key.

    Check yourself

    Volatility can't parse your memory image and every plugin errors. The most likely root cause is a mismatched what?

  4. Analysis: list processes, then catch the hidden one

    Use pslist vs psscan to expose a process hidden from the kernel's task list.

    Now the payoff. With the image and matching symbols loaded into Volatility (the de-facto memory analysis framework), you run plugins — each reconstructs one kind of evidence from the raw bytes. The first and most important pair is two different ways to enumerate processes, and the gap between them is where hidden malware appears.

    • `pslist` walks the kernel's linked list of active tasks — the same list ps reads. Fast, but anything unlinked from that list (a classic rootkit move: splice your own task out of the doubly-linked list so ps can't see it) is invisible to it.
    • `psscan` ignores the linked list and scans all of physical memory for process structures by their signature — pool tag / type pattern. It finds processes even if they were unlinked, and even some that have already exited but whose structures linger in RAM.

    This is the exact discrepancy-detection technique from rootkit hunting — /proc vs ps — but now applied to a frozen memory image where the attacker can't keep lying to you in real time. A process in `psscan` but missing from `pslist` is a hidden process.

    Example — not runnable on your lab box

    vol -f mem.lime linux.pslist
    PID   PPID  COMM
    1     0     systemd
    912   1     sshd
    1001  1     nginx
    1403  1340  bash
    2207  1     cron
    -f mem.lime
    The captured image file to analyze (Volatility auto-detects the right symbols when they're available).
    linux.pslist
    Walk the kernel's active-task linked list — fast, but blind to any task unlinked from that list.
    Five clean-looking processes, all with sane parents. By itself this looks fine — exactly the false comfort a list-unlinking rootkit relies on. You never trust one enumeration; you run the second view and compare.

    Example — not runnable on your lab box

    vol -f mem.lime linux.psscan
    OFFSET            PID   PPID  COMM
    0xffff9a0c1e3a0000  1     0     systemd
    0xffff9a0c1e3a8000  912   1     sshd
    0xffff9a0c1e3b0000  1001  1     nginx
    0xffff9a0c1e3b8000  1403  1340  bash
    0xffff9a0c1e3c0000  2207  1     cron
    0xffff9a0c1f7e2000  6613  1     [synth-impl]
    linux.psscan
    Scan all of physical memory for process structures by signature, independent of the linked list — so it catches unlinked/hidden tasks.
    psscan finds a sixth process — PID 6613 `[synth-impl]` — that pslist never showed. The structure is sitting in RAM (it's genuinely running) but it was unlinked from the kernel's task list to hide from ps and pslist. The discrepancy is the finding: this is the hidden process, the memory-image analog of the /proc-vs-ps gap. ([synth-impl] is a benign synthetic stand-in for the malicious process you'd see on a real case.)

    Check yourself

    A PID shows up in psscan but is absent from pslist. What does that discrepancy reveal?

  5. Analysis: dump it, find injected code, recover connections

    Pull a suspect process apart in memory — its image, injected regions, and live C2.

    Having found PID 6613, you interrogate it the way /proc let you interrogate a live process — except now from the frozen image, so the suspect can't react.

    • Dump the process (pslist --dump / proc.Maps + extraction) to carve its memory and executable image out for static analysis.
    • Find injected code with a malfind-style scan: look for memory regions that are both writable and executable (RWX) and not backed by any file on disk. Legitimate code is mapped from a file and is read-execute, not write-execute. An anonymous RWX region holding machine code is the signature of process injection — malware running inside a host process with no file to find on disk.
    • Recover network connections with `netscan`, which carves socket structures out of memory to rebuild who was talking to whom — including connections that had already closed but still linger in RAM.

    Example — not runnable on your lab box

    vol -f mem.lime linux.malfind --pid 6613
    PID   Process       Start              Protection       Notes
    6613  [synth-impl]  0x7f3a2c000000     PAGE_EXEC_READ_WRITE   No file mapping; RWX anonymous region
    
    Disassembly (first bytes):
    0x7f3a2c000000  push rbp
    0x7f3a2c000001  mov  rbp, rsp
    0x7f3a2c000004  ...   (shellcode-style prologue)
    linux.malfind
    Hunt for injected/unbacked executable memory — regions that are writable AND executable with no backing file.
    --pid 6613
    Restrict the scan to the suspect process you found in psscan.
    The flagged region is `RWX` (PAGE_EXEC_READ_WRITE), anonymous, no file mapping — three properties normal code never has together. Executable bytes that aren't backed by any on-disk file mean code was injected straight into memory: there is nothing on the filesystem to scan for. This is precisely the evidence that exists only in RAM (the open document on the desk, with no copy in the cabinet).

    Example — not runnable on your lab box

    vol -f mem.lime linux.sockstat
    PID   COMM          PROTO  LOCAL ADDRESS        REMOTE ADDRESS        STATE
    912   sshd          TCP    0.0.0.0:22           0.0.0.0:*             LISTEN
    1001  nginx         TCP    0.0.0.0:80           0.0.0.0:*             LISTEN
    6613  [synth-impl]  TCP    10.0.0.7:49870       203.0.113.66:443      ESTABLISHED
    linux.sockstat
    Carve socket structures from memory to rebuild network connections — the netscan-equivalent, including sockets owned by the hidden PID.
    The last line ties it together: the hidden PID 6613 held an `ESTABLISHED` connection out to `203.0.113.66:443` — a memory-resident command-and-control channel on 443 to blend with HTTPS. Because we carved this from memory, we recovered it even though the process was hidden from ps and the injected code left nothing on disk. (203.0.113.0/24 is a reserved documentation range — a stand-in for the real attacker IP you'd pivot on.)

    Check yourself

    malfind flags a memory region as RWX with no backing file. What activity does that indicate?

  6. When full capture isn't feasible: the /proc surface

    Approximate memory analysis live through /proc when you can't image RAM.

    Sometimes you can't do a full RAM capture — no external media big enough, no ability to build a LiME module, or you just need a fast live look before the formal capture arrives. The kernel already exposes a live window into process memory through `/proc`, and it lets you do a meaningful subset of the analysis above without any tooling.

    • `/proc/<pid>/maps` — the process's memory map: every region, its permissions, and what file (if any) backs it. This is your manual `malfind`: scan for an rwxp region with no pathname.
    • `/proc/<pid>/mem` — the process's actual memory contents, readable (with the right access) at the offsets maps gives you. This is the manual process dump.
    • `/proc/<pid>/exe` — the symlink to the real backing binary (and the (deleted) tell from process forensics).
    • `/proc/kcore` — the running kernel's memory as a virtual ELF core; a live, kernel-wide surface some tools can analyze in place.

    It's an approximation, not a forensic image — it's live, so it changes as you read it, and it's per-process rather than a single consistent snapshot. But it surfaces the same indicators: unbacked executable regions, deleted binaries, the real maps.

    Example — not runnable on your lab box

    sudo grep -E 'rwxp' /proc/6613/maps
    7f3a2c000000-7f3a2c021000 rwxp 00000000 00:00 0
    grep -E 'rwxp'
    Filter the memory map for regions that are simultaneously readable, writable, AND executable — the malfind signature.
    An rwxp region with a zero inode and no pathname at the end of the line means writable+executable memory backed by no file — the same injected-code indicator malfind reported, found here by hand against the live process. The trailing 0 (no path) is the tell: legitimate executable mappings name their file.
  7. Try it in your live terminal

    Run the safe, read-only memory-surface checks yourself on the real sandbox.

    You have a real root shell. Full RAM capture and Volatility aren't in this sandbox — those run on the Phase-2 DFIR lab image, which ships with the memory tooling and a provided memory snapshot to analyze. What you can do here, safely and read-only, is the acquisition pre-flight and the /proc approximation: the exact moves you'd make on a real host before and around a capture. Nothing below changes the system.

    uname -r
    Step one of any capture: record the exact kernel build. On a real acquisition you write this into evidence notes so the analysis station can generate matching symbols. Get in the habit of running it FIRST.
    ls -l /proc/kcore 2>/dev/null
    Note the apparent size — it'll look enormous (the kernel's whole virtual space) yet it's sparse and root-restricted. This is the live kernel-memory surface; you're confirming it exists, not copying it.
    command -v vol volatility avml insmod >/dev/null 2>&1; echo "(memory tooling arrives with the DFIR lab image)"
    A tool-presence check. In this sandbox the forensic memory tools (vol/volatility/avml, and module-loading workflow) generally aren't installed — they come with the Phase-2 DFIR image. The point is to learn the pre-flight, not to fail on a missing binary.
    sudo head -5 /proc/1/maps
    Pick a PID (here PID 1, systemd) and read the first lines of its live memory map: regions, permissions, and backing files. This is the manual malfind surface — on a suspect process you'd scan it for an rwxp region with no pathname.
    sudo ls -l /proc/1/exe
    The symlink to the real backing binary. On a clean process it points at a normal path; on a suspect one you watch for a (deleted) target — a program running from memory with its file unlinked from disk.
Recap
  • Memory is the most volatile and most revealing evidence: it holds decrypted data, injected code, unlinked-but-running programs, live connections, and the true process list — none of which disk reliably keeps. Capture it FIRST, and never reboot before you do.
  • You can't cp /dev/mem on a hardened kernel (CONFIG_STRICT_DEVMEM; /dev/kmem removed). Acquire with LiME (kernel module), AVML (portable userspace), or read /proc/kcore — always to EXTERNAL media, never the suspect's own disk.
  • Record the exact kernel version at capture time. Analysis needs a matching symbol table/profile; without it the image is a locked box with no key — the #1 gotcha.
  • Volatility's pslist vs psscan is discrepancy detection on frozen evidence: a process in psscan but not pslist is hidden (unlinked). malfind catches RWX unbacked memory = injection; netscan/sockstat recovers connections (incl. memory-resident C2); psaux pulls argv/env.
  • When full capture isn't feasible, /proc/<pid>/maps, /mem, /exe and /proc/kcore give a live, read-only approximation — same indicators (unbacked executable regions, deleted binaries), no consistent snapshot.
Memory artifact → why it matters / Volatility-style plugin → what it finds
Artifact in memoryWhy it mattersPlugin / surfaceWhat it finds
Kernel task listThe real running processes a userland rootkit hideslinux.pslistActive tasks via the linked list (fast, but blind to unlinked)
Process structures in RAMUnlinked/hidden processes can't be scrubbed from all of memorylinux.psscanProcesses by signature — pslist∖psscan gap = hidden process
Anonymous RWX regionsInjected code lives only in memory, nothing on disk to scanlinux.malfind / /proc/<pid>/mapsWritable+executable, file-unbacked memory = injection
Socket structuresLive/recent C2 connections never written to disklinux.sockstat / netscanPeer addresses, ports, state — incl. memory-resident C2
Process argv & environmentC2 addresses, configs, passphrases kept in process statelinux.psaux / env recoveryExact launch args and environment, even for hidden PIDs
Backing binary / imageDeleted-but-running programs run from RAMprocess dump / /proc/<pid>/exeThe real executable, recoverable from memory; (deleted) tell
Whole physical RAMThe single source for all of the aboveLiME / AVML / /proc/kcoreA forensic image to analyze offline (keyed to the kernel symbols)