Linux Logging Architecture & journald Investigation

When you walk onto a Linux host you're authorized to review, the first instinct is to cat /var/log/something. That instinct is half-wrong, and on a modern box it can make you miss the evidence entirely. Linux logging is not one file — it's a small pipeline with three distinct sources, and where a message lands depends on who produced it and how the host is configured.

The three sources you must hold in your head:

• The kernel ring buffer — a fixed-size in-memory buffer the kernel writes to directly (disk errors, OOM kills, USB events, network-interface state). You read it with dmesg.
• Classic syslog / rsyslog text logs — plain-text files under /var/log/: /var/log/messages or /var/log/syslog for general system messages, /var/log/secure or /var/log/auth.log for authentication. Written by a daemon, usually rsyslog.
• The systemd journal — a binary, structured, indexed store managed by systemd-journald. Every unit's stdout/stderr, every syslog() call, and the kernel buffer all flow into it. You query it with journalctl.

These coexist. On most modern distros rsyslog reads from the journal and writes the familiar text files for compatibility — so the same event can exist in two places at once.

The analogy that makes this stick: the journal is a structured database — every record carries typed, queryable fields (which unit, which PID, which user, what priority, exact timestamp), so you can ask precise questions and get precise answers. The rsyslog text files are a stack of paper printouts — human-readable, append-only, easy to grep, but flat: to filter them you're reaching for grep, awk, and your eyes, not a query language. A good responder uses both: the database to ask questions, the printouts to cross-check what the database says.

The kernel can't depend on a logging daemon to be alive — it boots before userspace and must record hardware faults, out-of-memory kills, and driver events even if journald never starts. So it writes to its own fixed-size ring buffer in memory. When the buffer fills, the oldest messages are overwritten — which is exactly why you copy what matters out before it scrolls away.

In an investigation the ring buffer answers questions nothing else can: Did the OOM killer reap a process? Was a USB device plugged in? Did a network interface flap or go promiscuous (a sniffer's fingerprint)?

Before journald, all of this lived in flat text under /var/log/, written by a syslog daemon (today almost always rsyslog). These files are still present on most distros, and they're the fastest way to profile a host you've never seen: one ls tells you what services have run, because services create their own log files and directories.

The naming splits by distro family, and knowing which family you're on is itself a finding:

• Debian/Ubuntu: general messages in /var/log/syslog, authentication in /var/log/auth.log.
• RHEL/CentOS/Fedora: general messages in /var/log/messages, authentication in /var/log/secure.

Step 3 noted that auth.log.1 is a rotated older copy. That's only half the picture, and the missing half is where investigations quietly go wrong. Text logs don't grow forever — they rotate, and old generations get compressed. On a real host, logrotate renames the live file to auth.log.1, then gzips the older generations, so /var/log ends up holding a mixed set side by side:

• auth.log — the live file, plain text.
• auth.log.1 — the most recent rotation, usually still plain (logrotate's delaycompress postpones compressing it one cycle).
• auth.log.2.gz, auth.log.3.gz, … — older generations, gzip-compressed (logrotate's compress default extension is .gz).

Here is the trap. Every grep you've seen so far — grep 'Failed password' /var/log/auth.log — reads only the live file. If the break-in you're hunting happened yesterday and has already rotated into auth.log.2.gz, that command returns zero hits, no error, exit 0. A confident, empty, wrong answer. That silent-empty result — not a crash — is the failure mode this whole step exists to prevent.

Now the database. Because the journal stores structured fields on every record, journalctl lets you query rather than grep. The investigative power is in combining filters so you go from millions of lines to the handful that matter. The filters worth memorizing:

• `-b` this boot, `-b -1` the previous boot — scope to the incident's reboot window.
• `-u <unit>` one service's logs (-u sshd).
• `-p err` priority floor (emerg,alert,crit,err,warning,notice,info,debug).
• `--since` / `--until` a wall-clock time window.
• `_PID=`, `_UID=`, `_COMM=` exact structured-field matches — these are the database columns.
• `-o verbose` / `-o json` to dump every field on a record.
• `-k` kernel messages only.

Here is the single fact that decides whether your journal evidence even exists: is the journal volatile or persistent?

• Volatile — stored under `/run/log/journal/`, which is a tmpfs (RAM). It is wiped on every reboot. This is the default when /var/log/journal/ does not exist.
• Persistent — stored under `/var/log/journal/`. Survives reboots. Enabled by creating that directory (or Storage=persistent in journald.conf).

Think back to the analogy: a volatile journal is a database held entirely in RAM with no save-to-disk — pull the power and the whole database is gone. For a responder this is decisive: on a volatile-journal host, a reboot has already destroyed your evidence, and an attacker who knows this only needs the box to restart.

No single source is gospel. The strongest conclusions come from agreement across sources — or from a disagreement that itself reveals tampering. Because rsyslog usually reads from the journal and writes the text files, the same event normally appears in both. When it appears in one but not the other, that delta is a finding.

In this incident the attacker ran sudo systemctl stop rsyslog (seen in step 4). That means after 12:09:40, rsyslog stopped writing the text logs while journald kept recording. The proof is a comparison.

You have a real shell to your right. Every command below is read-only and safe — it inspects logs, it never clears or stops anything. Run them in order: this is the exact opening sweep for a "reconstruct what happened on this host" call. There may be nothing malicious here; the goal is to make the moves reflexive.

One honest note before you start: this lab host logs to flat files in `/var/log`, not to systemd's journal, so the journalctl commands below may come back empty. Run them anyway to make the syntax reflexive — then treat an empty journal as the signal to pivot to /var/log (the last command), which is exactly the fallback a real responder reaches for on a host that doesn't use journald. Watch the persistence check in particular. Here /var/log/journal exists, so it prints persistent, yet journalctl --disk-usage reports 0B and journalctl says No journal files were found. That is step 6's caveat in practice: the directory is only the precondition, and the journal files inside it are the evidence.