Signal9Signal9 — home
online
all lessons

Authentication Log Analysis: Reconstructing Logins

Intermediate35 min6 steps
What you'll learn
  • Locate auth events on any host — auth.log, secure, or the journal — and know which one your distro actually writes to
  • Read sshd log lines fluently: tell a failed password, an invalid user, and a successful Accepted login apart at a glance
  • Use last, lastb, and lastlog to read the binary login records (who got in, who failed, when each user last connected)
  • Run the failures-per-source-IP workflow that exposes a brute force, then find the first Accepted from that IP — the moment of compromise
  1. Why reconstruct logins at all

    Frame auth-log analysis as answering who got in, when, and from where — on a host you are authorized to investigate.

    You are handed a host and one question: did someone log in who shouldn't have? Before an attacker can run a process, open a port, or plant a backdoor, they almost always have to authenticate first — and authentication is one of the most reliably logged events on a Linux system. Every SSH attempt, every sudo, every console login leaves a line behind.

    This lesson follows one incident from start to finish: an SSH brute force from 203.0.113.66 that hammered the host with failed passwords and then, eventually, succeeded. Your job is to reconstruct that story purely from the auth records and pin down the exact moment of compromise.

    Think of it like a building's badge-entry log. A door reader records every swipe — accepted or denied. A legitimate employee shows up as a single clean accepted swipe each morning. But a thief trying stolen badges leaves a very different trace: a long run of denied swipes at one door, late at night, then suddenly one accepted swipe. That one accepted swipe — right after the wall of denials — is the break-in moment. Auth logs are that badge reader, and 203.0.113.66 is the person at the door.

  2. Where auth events live

    Find the auth log on any distro — auth.log, secure, or the journal — instead of guessing.

    The single most common way an auth investigation stalls is looking in the wrong file, finding nothing, and wrongly concluding the box is clean. Where sshd, sudo, and PAM write their authentication events depends on the distribution:

    • Debian / Ubuntu → /var/log/auth.log
    • RHEL / CentOS / Fedora / Rocky → /var/log/secure
    • systemd hosts → the same events also live in the journal, queryable by unit (ssh or sshd) or by command name (_COMM=sshd)

    On most modern systems both exist: rsyslog writes the flat file and journald keeps a binary copy. Knowing all three means you are never stuck on an unfamiliar box.

    ls -l /var/log/auth.log /var/log/secure 2>/dev/null
    -rw-r----- 1 root adm 184204 Jun 23 02:13 /var/log/auth.log
    Only auth.log exists here, so this is a Debian/Ubuntu host — the auth events are in /var/log/auth.log. On a RHEL box you would see /var/log/secure instead and auth.log would be missing. Running this first, before anything else, tells you which file to grep. The 2>/dev/null hides the 'No such file' error for whichever one isn't present.
    sudo journalctl -u ssh -u sshd --no-pager | tail -n 4
    Jun 23 02:09:51 web01 sshd[4418]: Failed password for root from 203.0.113.66 port 51110 ssh2
    Jun 23 02:11:03 web01 sshd[4502]: Accepted password for webadmin from 203.0.113.66 port 51990 ssh2
    Jun 23 02:11:03 web01 sshd[4502]: pam_unix(sshd:session): session opened for user webadmin by (uid=0)
    Jun 23 02:14:22 web01 sudo[4530]:  webadmin : TTY=pts/0 ; PWD=/home/webadmin ; USER=root ; COMMAND=/usr/bin/apt install curl
    -u ssh -u sshd
    Filter to the SSH service unit. Distros name it differently — ssh on Debian/Ubuntu, sshd on RHEL — so asking for both covers either without you having to know which.
    --no-pager
    Dump straight to stdout so you can pipe to tail/grep instead of landing inside less.
    The journal holds the same auth events as the flat file, structured and timestamped. Note you can query the same data by command name with journalctl _COMM=sshd when you don't know the unit name. This is your fallback when the flat file has been rotated away — or tampered with (next steps).

    Check yourself

    You're triaging a RHEL host and cat /var/log/auth.log says 'No such file or directory'. Which file holds the sshd and sudo auth events on this distro?

  3. Reading sshd log lines

    Tell failed, invalid-user, and Accepted lines apart, and read the connection-closed and PAM session lines around them.

    sshd narrates every connection in a small, predictable vocabulary. Learn the handful of phrases and you can read an attack like a sentence. Here are the lines that matter, from a single sequence in our incident:

    sudo grep sshd /var/log/auth.log | grep '203.0.113.66' | head
    Jun 23 02:08:31 web01 sshd[4390]: Failed password for invalid user admin from 203.0.113.66 port 50902 ssh2
    Jun 23 02:08:31 web01 sshd[4390]: Connection closed by invalid user admin 203.0.113.66 port 50902 [preauth]
    Jun 23 02:08:55 web01 sshd[4402]: Invalid user oracle from 203.0.113.66 port 50988
    Jun 23 02:09:51 web01 sshd[4418]: Failed password for root from 203.0.113.66 port 51110 ssh2
    Jun 23 02:11:03 web01 sshd[4502]: Accepted password for webadmin from 203.0.113.66 port 51990 ssh2
    Five distinct phrases, each meaning something specific: - `Failed password for invalid user admin` — wrong password AND admin is not even a real account on this host. The invalid user tag means the username doesn't exist in /etc/passwd — a dead giveaway of guessing. - `Connection closed ... [preauth]` — the client gave up (or was dropped) before authenticating. [preauth] means it happened before login completed; a flood of these is automated guessing, not a human. - `Invalid user oracle` — sshd announcing the target account doesn't exist, separate from the password check. - `Failed password for root` — wrong password for a real account (root exists). No invalid user tag here, because root is real — but direct root SSH should usually be disabled anyway. - `Accepted password for webadmin` — the password worked. This is the one that matters. webadmin is a real account and the credential succeeded. From 203.0.113.66, right after the failures, at 02:11 — this is the compromise.

    Two more line types complete the picture — what happens after a successful login. PAM (the Pluggable Authentication Modules layer) logs the session lifecycle, and any privilege escalation shows up as sudo or su lines:

    sudo grep -E 'session opened|session closed|sudo|su\[' /var/log/auth.log | head
    Jun 23 02:11:03 web01 sshd[4502]: pam_unix(sshd:session): session opened for user webadmin by (uid=0)
    Jun 23 02:14:22 web01 sudo[4530]:  webadmin : TTY=pts/0 ; PWD=/home/webadmin ; USER=root ; COMMAND=/usr/bin/apt install curl
    Jun 23 02:14:22 web01 sudo[4530]: pam_unix(sudo:session): session opened for user root by webadmin(uid=1000)
    Jun 23 02:18:40 web01 su[4602]: Successful su for root by webadmin
    Jun 23 02:19:55 web01 sshd[4502]: pam_unix(sshd:session): session closed for user webadmin
    -E
    Extended regex so the | alternation works without backslashes — match any of the four session/escalation patterns in one pass.
    This is what the attacker did once in: - `session opened for user webadmin by (uid=0)` — PAM confirms the interactive session started. The matching `session closed` at 02:19 brackets the session: webadmin was on the box from 02:11 to 02:19. - `sudo ... USER=root ; COMMAND=/usr/bin/apt install curl` — webadmin escalated to root and ran a command. sudo logs the exact command, so you can see they pulled in curl (often a precursor to downloading a second-stage payload). - `Successful su for root by webadmin` — a second escalation, this time via su. Both sudo and su lines are where a foothold turns into root. The session-open/close pair plus these escalations are the bridge from who logged in to what they did — the pivot you'll follow next.

    Check yourself

    Which sshd phrase marks the actual successful login — the moment a credential worked — as opposed to a failed or rejected attempt?

  4. The binary login records: last, lastb, lastlog

    Read the wtmp/btmp/lastlog binary databases to see successful logins, failed logins, and per-user last login.

    Alongside the text logs, the kernel and login programs keep three binary databases you read with dedicated tools — not with cat or grep, because they're not text files. Each answers a different question:

    • `last` reads `/var/log/wtmp` → a history of successful logins, logouts, and reboots
    • `lastb` reads `/var/log/btmp` → a history of failed login attempts (root-only)
    • `lastlog` reads `/var/log/lastlog` → the single most recent login time for every account

    The trap to internalize now: last and lastb read different files. Look in the wrong one and you'll see the wrong outcome.

    last -a | head
    webadmin pts/0        Tue Jun 23 02:11   still logged in    203.0.113.66
    webadmin pts/0        Mon Jun 22 09:02 - 17:40  (08:38)   198.51.100.42
    reboot   system boot  Mon Jun 22 08:55   still running      6.1.0-21-amd64
    webadmin pts/0        Sun Jun 21 09:14 - 18:01  (08:47)   198.51.100.42
    -a
    Put the remote hostname/IP in the LAST column so long addresses aren't truncated — critical when the source IP is the thing you care about.
    last reads wtmp, the successful-login record. Read it for what breaks the pattern. webadmin normally logs in around 09:00 from the internal 198.51.100.42. The top line is a login at 02:11 from the external 203.0.113.66 — odd hour, unfamiliar source, still connected. That external login at 02:11 is exactly the Accepted password line from the previous step. The reboot line shows the box last booted Monday 08:55.
    sudo lastb 2>/dev/null | head || echo "no btmp"
    oracle   ssh:notty    203.0.113.66     Tue Jun 23 02:09 - 02:09  (00:00)
    root     ssh:notty    203.0.113.66     Tue Jun 23 02:09 - 02:09  (00:00)
    admin    ssh:notty    203.0.113.66     Tue Jun 23 02:08 - 02:08  (00:00)
    admin    ssh:notty    203.0.113.66     Tue Jun 23 02:08 - 02:08  (00:00)
    2>/dev/null
    btmp may not exist on a freshly built host; suppress the error so the || echo fallback can report that cleanly instead of a confusing stderr message.
    lastb reads btmp, the failed-login record — the mirror image of last, and root-only. A burst of failures across many usernames (oracle, root, admin) from the same 203.0.113.66 in the same minute is the unmistakable shape of a brute force or password spray. It's the same IP that succeeded as webadmin at 02:11 — so the binary records alone already tell you: spray at 02:08–02:09, success at 02:11. An empty lastb on a clean box is itself a useful (negative) result.
    lastlog | grep -vi 'never logged in'
    Username         Port     From             Latest
    root             pts/0    198.51.100.42    Mon Jun 22 21:30:14 +0000 2026
    webadmin         pts/0    203.0.113.66     Tue Jun 23 02:11:02 +0000 2026
    lastlog reads /var/log/lastlog and prints the most recent login for every account in one table. Filtering out the 'Never logged in' system accounts leaves the accounts that actually log in. webadmin's latest login is from 203.0.113.66 at 02:11 — confirming the compromised account at a glance, and flagging that its newest login came from the attacker IP rather than the usual internal address.

    Check yourself

    Which command shows you FAILED login attempts — the signature of a brute force — by reading /var/log/btmp?

  5. The workflow: count failures, find the breach

    Aggregate failures by source IP to expose the brute force, then find the first Accepted from that IP — the compromise moment.

    Now you put it together into the workflow that turns thousands of log lines into one conclusion. A real attack buries its one successful login under a mountain of failures — you can't read them line by line. So you aggregate: count failed passwords per source IP. The IP with hundreds of failures is the brute force; everything else is noise.

    This is the badge-log method from step 1, made concrete: tally the denied swipes by who was swiping, find the one address responsible for the wall of denials, then look for that same address's first accepted swipe.

    sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
        412 203.0.113.66
          6 198.51.100.42
          2 198.51.100.51
    grep "Failed password"
    Keep only the failed-authentication lines — the brute-force fuel.
    awk '{print $(NF-3)}'
    Print the 4th-from-last field. In '...from <IP> port <N> ssh2' the source IP sits 3 fields before the end, so NF-3 extracts the IP regardless of how long the username is.
    sort | uniq -c
    uniq -c needs its input sorted; together they collapse identical IPs into one line prefixed with a count.
    sort -rn
    Sort that count numerically (-n), highest first (-r), so the worst offender floats to the top.
    The verdict in three lines. 203.0.113.66 is responsible for 412 failed passwords — orders of magnitude more than the handful of fat-fingered failures from the legitimate internal hosts (198.51.100.x, 2–6 each). That lopsided count is the brute force. The two internal IPs with a few failures each are normal human typos, not an attack. You've found the attacker's address without reading a single line individually.

    You have the attacker IP. Now answer the question that actually matters: did they ever get in, and when? Search that exact IP for the first Accepted line. A brute force with zero Accepted lines failed; one with an Accepted line succeeded, and that line's timestamp is the moment of compromise.

    sudo grep "Accepted" /var/log/auth.log | grep "203.0.113.66" | head -n 1
    Jun 23 02:11:03 web01 sshd[4502]: Accepted password for webadmin from 203.0.113.66 port 51990 ssh2
    This single line is the breach. The brute-force IP 203.0.113.66 finally guessed webadmin's password at 02:11:03. Everything matters here: it's password auth (not a key, so a guessable credential), the account is the real webadmin, the IP is the one with 412 failures, and the time sits immediately after the 02:08–02:09 spray. head -n 1 gives you the first success — the entry point. From here you pivot: grep the same sshd[4502] PID and the webadmin session to follow exactly what they did after getting in (the sudo/su lines from step 3).

    Check yourself

    After your aggregation pipeline names 203.0.113.66 as the brute-force source, what single phrase do you grep for (filtered to that IP) to find out whether — and exactly when — they got in?

  6. Try it in your live terminal

    Run the auth-log reconstruction yourself on the real sandbox to your right.

    You have a real root shell. Run these read-only reads in order — the same sequence you'd use on a 'did someone log in?' call. There may be nothing malicious on this box; the point is to make the moves reflexive. Each command is safe and read-only: you are reading existing records, not changing anything. Note anything that breaks the pattern — odd hours, external IPs, a lopsided failure count — as you go.

    last -a | head
    Successful logins from wtmp, source IP in the last column. Look for unfamiliar source addresses or logins at odd hours. This is your who-got-in baseline.
    sudo lastb 2>/dev/null | head || echo "no btmp"
    Failed logins from btmp. A burst from one IP across many usernames is a brute force. An empty result (or 'no btmp') is a clean, reassuring sign — not an error.
    sudo journalctl -q -u ssh -u sshd --no-pager 2>/dev/null | tail | grep . || sudo grep -h sshd /var/log/auth.log /var/log/secure 2>/dev/null | tail
    The detailed sshd lines, journal-first with a flat-file fallback. Grepping both auth.log and secure means it works on Debian and RHEL alike. Read the tail for recent Accepted/Failed/Invalid user lines. Two details make the fallback actually fire on a host with no journal: -q suppresses journalctl's -- No entries -- banner, and the trailing | grep . turns 'produced no output' into a non-zero exit the || can see. Without them the pipeline ends in tail, which succeeds on empty input, so || would never run and you'd stare at a blank result believing the box was clean.
    sudo grep -h "Failed password" /var/log/auth.log /var/log/secure 2>/dev/null | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
    The failures-per-IP aggregation against whichever auth file exists. The top line by count is your brute-force candidate. Then grep Accepted for that IP to find the first successful login — the compromise moment.
Recap
  • Auth events live in /var/log/auth.log (Debian) or /var/log/secure (RHEL), and in the journal (journalctl -u ssh -u sshd / _COMM=sshd) — confirm which sink the host uses before trusting an empty grep.
  • sshd speaks a small vocabulary: Failed password, Invalid user, Connection closed [preauth] are misses; Accepted password/publickey is the hit. The presence of 'invalid user' tells you whether the target account even exists.
  • last (wtmp) shows successful logins, lastb (btmp) shows failed ones, lastlog shows each account's most-recent login — read the right database for the outcome you want, and cross-check the journal against tampered flat files.
  • The workflow names the attack and times the breach: count Failed password by source IP to expose the brute force, then find the first Accepted from that IP — that timestamp is the moment of compromise and the anchor for everything after.
  • Judge the pattern, not the line: hundreds of failures across many usernames from one IP is an attack; a single failed password is a typo.
Artifact → source → what it proves
Artifact / commandSource fileWhat it proves
grep sshd auth.log / journalctl _COMM=sshd/var/log/auth.log, /var/log/secure, journalPer-attempt sshd narrative: failed, invalid-user, and Accepted logins
last -a/var/log/wtmpSuccessful logins, logouts, and reboots — who got in, from where, when
lastb/var/log/btmpFailed login attempts — the brute-force / password-spray fingerprint (root-only)
lastlog/var/log/lastlogThe single most-recent login per account — spot a compromised account fast
sudo / su lines/var/log/auth.log / securePrivilege escalation after login — the exact command run as root
Failures-by-IP pipeline → first Acceptedauth.log / secure / journalNames the brute-force source IP and timestamps the moment of compromise