Signal9Signal9 — home
online
back to labs

Persistent journald: Stop Losing Logs on Reboot

MediumLFCS Prep15-20 min
The brief

Post-incident review keeps hitting a wall on this host: after every reboot the systemd journal only shows the current boot. On the running fleet host journalctl --list-boots listed exactly one boot, and yesterday's logs — the ones the review actually needs — are gone. The service is logging fine while it runs; the history just does not survive a restart.

That is the signature of a journal kept in volatile storage. systemd-journald can write the journal to a tmpfs under /run (wiped on every boot) or persist it on disk under /var/log/journal (kept across reboots). Which one it does is governed by the Storage= setting in /etc/systemd/journald.conf together with whether /var/log/journal exists. Right now this box is throwing its journal away at every reboot.

Reconfigure journald so the journal is retained across reboots: set the storage policy in /etc/systemd/journald.conf (or in a drop-in under /etc/systemd/journald.conf.d/, which overrides it) to a value that persists, and make sure the on-disk journal directory /var/log/journal exists to hold it. Both the persistent policy and the auto policy (which persists precisely when that directory is present) are acceptable, and so is removing the override altogether: with no Storage= set, journald uses its default, auto. See /root/case/brief.txt. One thing to know before you start: this lab box is a container, so systemd is not PID 1 on it and journald is not running — journalctl answers "No journal files were found." here whatever you ask it, before and after your fix. That is the container, not the fault. You diagnose and repair this one off disk, and grading reads the config (main file plus drop-ins) and the directory — you do not need to restart the daemon.

What you'll do
  • Recognize a single-boot journal history as volatile (in-memory) journal storage
  • Set journald's storage policy so the journal is written to disk
  • Create the on-disk journal directory /var/log/journal that persistence needs
  • Accept either the persistent policy or auto with the journal directory present

Hands-on lab: you work on a real Linux box in your browser, graded on the machine's actual state.