auditd for Responders: Rules, ausearch & aureport

Ordinary logs are written by the programs being logged. sshd decides what to put in the auth log; a shell writes nothing about the commands it runs unless you bolt something on. That is fine for operations but weak for investigation: a program that has been subverted can simply lie, or write nothing at all.

The Linux audit subsystem answers a different question — did this actually happen? — by recording events inside the kernel, at the moment a syscall returns, before any user-space program gets a chance to spin the story. The kernel emits audit events to a dedicated daemon, `auditd`, which writes them to `/var/log/audit/audit.log`. It can capture things no ordinary log sees: every execve (every command run, by whom, with which arguments), every read or write of a watched file, every change to a watched directory, and every login.

Think of auditd as the courtroom stenographer. An ordinary log is a casual diary the suspect keeps themselves — they choose what to write and can tear out a page. The stenographer sits in the corner of the kernel and transcribes every official action verbatim, in order, as it occurs. When you need to prove something in an incident, you reach for the transcript, not the diary.

Before you write rules or trust a search, confirm the subsystem is actually collecting. auditctl -s (status) prints the kernel-side state: whether auditing is enabled, the PID of the auditd process receiving events, how many events were lost (a backlog overflow means you have gaps), and the failure mode.

Audit only records what its rules tell it to. auditctl -l lists the rules currently loaded in the kernel. There are two kinds, and knowing which to reach for is the core skill:

• Watches answer "who touched this path?" — -w <path> -p <perms> -k <key>. Perms are r (read), w (write), x (execute), a (attribute/metadata change). A watch is really shorthand the kernel expands into syscall rules on that path.
• Syscall rules answer "who called this syscall?" — -a always,exit -F arch=b64 -S <syscall> -k <key>. These fire on any invocation of a syscall (optionally filtered with -F on uid, success, paths, etc.), regardless of which file is involved.

The `-k` key is a label you attach to a rule so you can later pull exactly its events with ausearch -k. Choose it carefully — it is how your future self finds these records in a million-line log.

/var/log/audit/audit.log is enormous and the records are dense, so you almost never cat it. `ausearch` is the query tool: it filters the log and, crucially, reassembles the multi-line records that belong to a single event and renders timestamps as human-readable dates.

The filters you will reach for constantly:

• -k <key> — events tagged with a rule key (the payoff for tagging your rules)
• -m <type> — by record type, e.g. USER_LOGIN, EXECVE, SYSCALL, USER_AUTH
• -ui <uid> / -ua <auid> — by user id (or audit login uid, which survives su)
• -ts <when> / -te <when> — time start/end, e.g. -ts today, -ts recent

ausearch answers a specific question; `aureport` gives you the overview — counts and tables across the whole log so you can see where to point ausearch. It is your triage pass: "how many failed logins, by whom?", "what executed today?", "which users were most active?"

An investigator's job often comes down to a yes/no a manager or a court can rely on: did this file get read? did this binary run, and by whom? The audit record is the artifact that settles it — but only if you can read every field. One event is several record lines sharing the same `audit(timestamp:serial)` id; ausearch groups them for you.

You have a real root shell to your right. The audit subsystem often isn't active in a container, so each command below degrades gracefully — it shows you the real output where audit is running and a clear message where it isn't. Run them in order; the goal is to make the inspection workflow reflex, not to depend on a particular box's state.