Package Integrity Verification: Catching Tampered Binaries

Intermediate45 min7 steps

What you'll learn

Explain why package verification catches a trojaned binary: the package database records each file's expected size, mode, owner, and hash
Run rpm -Va / rpm -V and read the SM5DLUGTP flag string to tell a tampered binary (5 or S) from a benign mtime-only change (T)
Run dpkg --verify / debsums on Debian systems and read the ??5?????? output, knowing dpkg's built-in check only covers stored md5sums
Tie a changed file back to the package that owns it with rpm -qf / dpkg -S, and cross-check against an external known-good source when the package DB itself may be tampered
Build and use an AIDE baseline: aide --init, move aide.db.new into place, aide --check and its exit status, aide --update after a verified change, and read the report's Changed entries

Why package verification catches tampering
Frame integrity verification: the package DB is a known-good manifest the attacker has to defeat.
An attacker who gains root often replaces a system binary to hide. A trojaned `ls` omits their files from listings; a trojaned `ps` hides their process; a backdoored `sshd` logs every password or grants a magic login. Editing a config — say slipping a rule into /etc/sudoers or /etc/pam.d/ — is the same move on a text file.
Here is the leverage you have as the responder: when the package manager installed that binary, it recorded exactly what the file should look like — its size, its permissions (mode), its owner and group, and a cryptographic hash of its contents. The attacker's replacement almost never matches all of those at once. The file on disk no longer agrees with the record the package manager kept.
Verification is simply re-checking the file on disk against that stored record, attribute by attribute, and reporting every difference.
Why it matters
Analogy: a tamper-evident seal on a box of medicine. The box states exactly what is inside and is sealed at the factory. You don't need to know what the pills should look like — you just check whether the seal is intact. The package database is that printed manifest plus the seal: it says precisely what every system file should be, so a broken seal (a mismatched hash or mode) tells you someone got there first, even if the swapped binary looks identical to the eye.
Check yourself
Why can package verification detect a trojaned /usr/bin/ls even when the file looks normal in a directory listing?
RPM world: rpm -Va and the flag string
Run a full-system RPM verify and learn to read the SM5DLUGTP output.
On RPM-based systems (RHEL, Fedora, Rocky, AlmaLinux, SUSE) the verifier is built into rpm itself.
- rpm -V <package> verifies the files of one package.
- rpm -Va verifies all installed packages — the whole system in one sweep.
The critical skill is reading the output. Each changed file prints a nine-character flag string followed by an optional attribute marker and the path. A . (dot) in a position means that attribute is unchanged; a letter means that attribute differs from what was installed.
Example — not runnable on your lab box
```
$ sudo rpm -Va
```
```
S.5....T.    /usr/bin/ls
.M.......  g /var/log/journal
S.5....T.  c /etc/ssh/sshd_config
.......T.    /usr/share/man/man1/tar.1.gz
missing      /usr/bin/lsof
```
-V
Verify: compare installed files against the package database record. Long form: --verify.
-a
All packages — verify every installed package on the system.
Read each nine-character string left to right. S.5....T. on `/usr/bin/ls` is the alarm: S = size differs, 5 = MD5/SHA hash digest differs, T = mtime differs. A binary whose size and content hash both changed is a tampered binary — exactly the trojaned-ls signature. The c next to /etc/ssh/sshd_config marks it a config file; the g next to the journal marks a ghost file. The last line, `missing`, means a file the package installed is gone from disk.
Memorise the nine positions in order — `SM5DLUGTP`:
- S — Size differs
- M — Mode differs (permission bits / file type)
- 5 — the MD5/hash digest differs (the file's contents changed)
- D — Device major/minor number differs
- L — symbolic Link target differs
- U — User (owner) differs
- G — Group differs
- T — mTime (last-modified time) differs
- P — caPabilities differ
Plus a trailing marker: `c` = config file, `d` = documentation, `g` = ghost, `l` = license, `r` = readme. A leading missing replaces the whole string when the file is absent.
An mtime-only (T) change is usually benign
Do not treat every line as compromise. A flag string of .......T. — only T set — means just the modification timestamp differs while size, mode, owner, and hash are all intact. That happens routinely from backups, touch, or rebuilds, and on its own is not evidence of tampering. The lines that should make you sit up are the ones with `5` (contents changed) or `S` (size changed), especially on an executable. Triage by what changed, not by how many lines appear.
Check yourself
In an rpm -V flag string, which character position means the file's hash (its contents) no longer matches what was installed?
RPM world: verify one file and find its owner
Narrow from a whole-system sweep to a single suspect file and the package behind it.
A full rpm -Va can be noisy. Two narrower moves let you zero in on a suspect. rpm -Vf <file> verifies just the package that owns one file, and rpm -qf <file> answers which package owns it — so you can name the suspect precisely in your notes and pull a known-good copy from the same package version later.
Example — not runnable on your lab box
```
$ rpm -Vf /usr/bin/ls
```
```
S.5....T.    /usr/bin/ls
```
-V
Verify mode.
-f
...the package that owns this File — locate the owning package and verify it.
Instead of naming a package, you point -Vf at a path and it figures out the owning package and verifies it. The S.5....T. line confirms the same finding in isolation: size and hash changed on /usr/bin/ls. If the binary were untouched, this command would print nothing at all — silence means 'matches the database'.
Example — not runnable on your lab box
```
$ rpm -qf /usr/bin/ps
```
```
procps-ng-4.0.4-2.el9.x86_64
```
-q
Query mode — every rpm read operation begins with -q.
-f
...by File: given a path, name the package that owns it.
rpm -qf returns the full package NVRA (name-version-release-arch). Now you know /usr/bin/ps belongs to `procps-ng-4.0.4-2.el9`. With the exact version in hand you can fetch that same RPM from a trusted mirror and compare its ls/ps against the one on disk — your known-good reference.
Why it matters
Pairing verify with ownership is the whole investigative loop in RPM-land: rpm -Va (or -Vf) tells you that a file changed, and rpm -qf tells you which package and version it should have been. Together they turn a vague 'this box feels off' into 'the ls from coreutils-8.32-35.el9 has a different hash than shipped — here is the package to re-pull and diff.'
Debian world: dpkg --verify and debsums
Run the equivalent checks on a Debian/Ubuntu host and understand dpkg's narrower coverage.
Debian-family systems (Debian, Ubuntu, Mint) verify differently and with an important limitation. dpkg --verify (also written dpkg -V) checks installed files against dpkg's records, using the same nine-position flag layout idea as RPM — but with a catch: by default dpkg only knows the md5sums the maintainer shipped, and only checks the digest (5) for files that have a stored md5sum. Mode/owner verification is far thinner than RPM's, so many genuine changes simply won't surface.
```
$ sudo dpkg --verify
```
```
??5??????   /bin/ls
??5??????   /usr/sbin/sshd
missing     c /etc/cron.d/popularity-contest
```
--verify
Verify installed package files against dpkg's stored checksums. Short form: -V.
dpkg uses the same positional layout, but a ? means 'not checked / no data for this attribute' rather than 'unchanged'. The line `??5?????? /bin/ls` reads: dpkg couldn't check most attributes, but the `5` (digest) position is set — the content hash does not match the shipped md5sum. That is the trojaned-binary tell on Debian. The same 5 on `/usr/sbin/sshd` points at a backdoored SSH daemon. missing again means an installed file is gone.
Because dpkg's built-in check only covers the md5sums it happens to have stored, it can miss files altogether. debsums is the fuller tool: it computes and compares hashes for installed package files and flags each as OK, FAILED, or REPLACED.
Example — not runnable on your lab box
```
$ sudo debsums -c
```
```
/bin/ls
/usr/sbin/sshd
```
-c
Report only files that FAILED their checksum (changed). Without it, debsums prints every file and its OK/FAILED status.
debsums -c is the 'show me only the problems' switch: it lists just the files whose content hash no longer matches. Here it independently confirms /bin/ls and /usr/sbin/sshd were altered. Use dpkg -S /bin/ls to name the owning package (the Debian counterpart to rpm -qf), exactly as you did on RPM.
Don't trust a clean 'dpkg --verify' as proof of integrity
Because dpkg --verify only checks files it has stored md5sums for — and verifies little beyond the digest — a clean run does not mean the system is untouched. Some packages ship no md5sums at all, so their files are never checked. On Debian, reach for `debsums` (and install it first if absent) for fuller coverage, and remember even that only catches files the packaging knows about. Absence of a finding is weaker evidence here than on RPM.
Check yourself
On Debian, dpkg --verify prints '??5??????' for /usr/sbin/sshd. What does the '5' mean, and what do the '?' characters mean?
The responder workflow and the package-DB trap
Sequence verify → identify → confirm, and account for an attacker who tampered the package DB itself.
Put the pieces into a repeatable workflow you can run on any host you're authorized to investigate:
1. Verify the whole system — rpm -Va or dpkg --verify / debsums -c.
2. Triage the output — focus on `5` (contents) and `S` (size) on executables and on changes to security configs (c files like sshd_config, sudoers, pam.d). Set aside lone `T` (mtime-only) noise.
3. Identify the owning package and version — rpm -qf / dpkg -S.
4. Confirm against known-good — re-download that exact package version from a trusted mirror and diff the file, or compare its hash to a vendor-published value.
That last step exists because of one nasty assumption baked into steps 1-3.
The attacker can tamper the package database too
rpm -Va and dpkg --verify compare the file on disk to a record stored on the same compromised host. A capable attacker who swaps /usr/bin/ls can also rewrite the package DB so the stored hash matches their trojan — and now verification reports the file as clean. Trusting rpm -Va alone is the trap. The defense is an external known-good source: re-pull the package from a trusted mirror, compare against a vendor-signed hash, or verify from read-only/offline media or a separate forensic image. Never let the box you're investigating be the sole judge of its own integrity.
Config changes (c) are different from binary changes
A 5/S flag on an executable (/usr/bin/ls, /usr/sbin/sshd) is high-signal — you don't edit shipped binaries in normal operation. A change on a config file (the c marker) is expected: admins are supposed to edit /etc/ssh/sshd_config. Don't dismiss config changes — review what changed — but weigh them differently from an altered binary. Ignoring config edits entirely misses backdoors planted in PAM or sudoers; treating every config edit as compromise floods you with false positives.
Check yourself
Why is 'rpm -Va comes back clean' not sufficient proof that no system binary was tampered with?
AIDE: a baseline you build yourself
Initialise an AIDE database, check against it, read the report, and update it after a legitimate change.
Package verification only knows about files a package shipped, and only the attributes the package manager recorded. Your edited sshd_config, a new file in /etc/cron.d, a script in /usr/local/bin: none of them have a package record to compare against. AIDE (Advanced Intrusion Detection Environment) fills that gap with a baseline you build. It walks the paths its config names, stores each file's hashes, size, owner, mode and timestamps in a database, and later reports every difference: entries added, removed and changed.
The workflow has four moves:
1. `aide --init` builds a new database, aide.db.new, from the current state. Do it on a host you trust, ideally straight after install.
2. Move it into place as aide.db, the file --check reads. Init never overwrites the live baseline by itself.
3. `aide --check` compares the filesystem with aide.db and prints a report. Run it on a schedule (Ubuntu's package installs a daily cron job).
4. `aide --update` after a change you have verified is legitimate: it checks and writes a fresh aide.db.new, which you move into place again.
AIDE isn't installed on the lab box, so the next three blocks show a real Ubuntu 22.04 host (AIDE 0.17.4, apt install aide). On RHEL the package is also aide, and the database is /var/lib/aide/aide.db.gz.
Example — not runnable on your lab box
```
$ aide --config /etc/aide/aide.conf --init | head -n 2
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
```
```
Start timestamp: 2026-10-02 18:48:36 +0000 (AIDE 0.17.4)
AIDE initialized database at /var/lib/aide/aide.db.new
```
--config FILE
Use this configuration (Ubuntu's is /etc/aide/aide.conf; RHEL's is /etc/aide.conf, the default).
--init
Build a new database from the current filesystem, written to database_out (aide.db.new).
Init hashes every path the config selects (about 5,500 entries on a minimal Ubuntu system, done in under a minute) and writes aide.db.new. The mv promotes it to the baseline. Ubuntu's aideinit wrapper does both steps. The full init output ends with the database's own hashes: record them somewhere off the box, because they let you prove later that nobody rebuilt the baseline.
Example — not runnable on your lab box
```
$ echo '203.0.113.7 updates.example.com' >> /etc/hosts
aide --config /etc/aide/aide.conf --check --limit /etc > /root/aide-report.txt; echo "aide exit status: $?"
sed -n '/^Summary:/,/^Detailed/p' /root/aide-report.txt
```
```
aide exit status: 4
Summary:
  Total number of entries:	5563
  Added entries:		0
  Removed entries:		0
  Changed entries:		1

---------------------------------------------------
Changed entries:
---------------------------------------------------

f >b... mc..H.. . : /etc/hosts

---------------------------------------------------
Detailed information about changes:
```
--check
Compare the filesystem with database_in (aide.db) and report differences.
--limit REGEX
Only check paths matching REGEX (here everything under /etc).
A planted hosts entry, caught. The exit status is a bitmask you can alert on: 1 = entries added, 2 = removed, 4 = changed, so 4 means changes only, 5 means added plus changed, and 0 means clean. Each report line starts with a summary string: the first character is the type (f file, d directory), then one position per attribute, a . where nothing changed. Here > the size grew, b the block count changed, m and c the modify and change times moved, and `H` the content hash differs (AIDE before 0.17 printed C for the same thing). An H on a file nobody was supposed to edit is the line to chase. The Detailed information section below the summary gives the old and new value of each attribute.
Example — not runnable on your lab box
```
$ aide --config /etc/aide/aide.conf --update --limit /etc > /dev/null; echo "aide exit status: $?"
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
aide --config /etc/aide/aide.conf --check --limit /etc | sed -n 2p
```
```
aide exit status: 4
AIDE found NO differences between database and filesystem. Looks okay!!
```
--update
Check, then write a new database reflecting the current filesystem to aide.db.new.
Suppose the hosts entry was an approved change. --update runs the same comparison (so it still exits 4) and writes the current state to aide.db.new. Promote it, and the next check is clean. Only update after you have explained every difference: running --update over an unexplained change writes the attacker's edit into your baseline.
A baseline on the same disk can be rebuilt by the attacker
AIDE has the same weakness as the package database: if aide.db lives on the host, an attacker with root can run aide --update, promote the result, and every later check reads clean. Keep a copy of the database (or at least the hashes --init printed) somewhere the host can't write: read-only media, a central server, a ticket. Before trusting a clean check, compare sha256sum /var/lib/aide/aide.db with the copy you kept.
```
$ cat > /root/aide-sample.txt <<'EOF'
AIDE found differences between database and filesystem!!

Summary:
  Total number of entries:	5562
  Added entries:		1
  Removed entries:		0
  Changed entries:		2

---------------------------------------------------
Added entries:
---------------------------------------------------

f+++++++++++++++++: /etc/cron.d/zz-new

---------------------------------------------------
Changed entries:
---------------------------------------------------

d =.... mc.. .. . : /etc/cron.d
f >b... mc..H.. . : /etc/hosts

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------
EOF
sed -n '/^Changed entries:/,/^Detailed/p' /root/aide-sample.txt | grep ' : /'
```
```
d =.... mc.. .. . : /etc/cron.d
f >b... mc..H.. . : /etc/hosts
```
sed -n '/A/,/B/p'
Print only the lines from the first one matching A through the next one matching B.
You will often get the report, not the host: a mailed cron result or a saved file. This block saves a trimmed AIDE 0.17.4 report, then pulls out just the Changed entries section: sed -n '/START/,/END/p' prints from the Changed entries: header to the Detailed header, and grep ' : /' keeps the entry lines. Read them as a pair. The directory /etc/cron.d changed only its times (mc), which is what adding a file to it does, and the Added section shows that file. /etc/hosts changed content (H). Two leads: a new cron job and an edited hosts file.
Check yourself
aide --check exits with status 2. What kind of difference did it find: added, removed or changed entries?
Try it in your live terminal
Run the verification workflow yourself on the real sandbox, distro-agnostic and read-only.
You have a real root shell. These commands only read state — they change nothing — and they work whether the sandbox is RPM-based or Debian-based. There may be nothing tampered here; the goal is to make the moves reflex. Remember: on RPM, no output from a verify means the file matches the database.
```
$ command -v rpm >/dev/null && echo RPM || (command -v dpkg >/dev/null && echo DPKG)
```
Detect which package world you're in. Prints RPM on RHEL/Fedora/SUSE-family hosts, or DPKG on Debian/Ubuntu — so you know which verify command applies before you run anything.
```
$ rpm -Vf $(command -v ls) 2>/dev/null || dpkg -V coreutils 2>/dev/null | head || echo "no pkg verify"
```
Verify the package that owns your real ls. On RPM it runs rpm -Vf against the resolved path; on Debian it verifies coreutils (which ships ls) and shows the first lines; if neither tool exists it prints a fallback note. Clean output (or no output on RPM) means ls matches what was installed.
```
$ rpm -qf $(command -v ps) 2>/dev/null || dpkg -S $(command -v ps) 2>/dev/null
```
Identify which package owns the ps binary — the ownership half of the loop. RPM answers with the package NVRA; Debian answers with package: /path. This is the version you'd re-pull from a trusted mirror to confirm a suspected tamper against known-good.
```
$ rpm -Va 2>/dev/null | grep -E '^..5|^S' | head || sudo debsums -c 2>/dev/null | head || echo "install debsums for full Debian hash checks"
```
The high-signal sweep: on RPM, filter a full verify down to lines where the hash (5) or size (S) changed — the tampered-binary candidates — hiding the mtime-only noise. On Debian it falls back to debsums -c (changed files only), or prompts you to install debsums if it's missing.

Recap

The package database is a known-good manifest: it records each file's expected size, mode, owner, group, and content hash, so a tampered binary stops matching its own record.
On RPM, read the SM5DLUGTP flag string — 5 (hash) and S (size) on an executable mean tampering; a lone T (mtime) is usually benign; c marks a config file you'd expect admins to edit.
On Debian, dpkg --verify only checks stored md5sums (mostly the 5 position, ? = not checked), so use debsums for fuller coverage and never read a clean run as proof of integrity.
Run the loop verify -> triage -> identify owner (rpm -qf / dpkg -S) -> confirm against an EXTERNAL known-good source, because an attacker with root can also rewrite the local package DB to hide the swap.
AIDE covers what packages don't (configs, unpackaged files): aide --init, move aide.db.new to aide.db, aide --check (exit bits 1 added, 2 removed, 4 changed; H = content hash changed), and aide --update only after every difference is explained. Keep a copy of the baseline off the host.

rpm -V flag → meaning
Flag	Position / marker	Means	Weight as an IOC
`S`	1 — Size	File size differs from installed	High on a binary
`M`	2 — Mode	Permission bits or file type differ	Medium (e.g. world-writable)
`5`	3 — Digest	Content hash differs — the bytes changed	Highest on a binary
`D`	4 — Device	Device major/minor differs	Low / rare
`L`	5 — Link	Symlink target differs	Medium
`U`	6 — User	Owner differs	Medium
`G`	7 — Group	Group differs	Medium
`T`	8 — mTime	Modified-time differs only	Low — often benign noise
`P`	9 — caPs	File capabilities differ	Medium (privilege escalation)
`c`	marker	File is a config file	Expected to change — review, don't panic
`missing`	whole string	Installed file is absent from disk	High — investigate

Package Integrity Verification: Catching Tampered Binaries

Why package verification catches tampering

RPM world: rpm -Va and the flag string

RPM world: verify one file and find its owner

Debian world: dpkg --verify and debsums

The responder workflow and the package-DB trap

AIDE: a baseline you build yourself

Try it in your live terminal