Endpoint Telemetry & Live Response at Scale

In the process-forensics lesson you triaged one suspicious host with ps, ss, and lsof. That works beautifully — on one box, while the threat is still running. Now scale the problem up. You are responsible for four thousand Linux hosts and an alert fires at 3 a.m. naming a host you have never heard of. By the time you SSH in, the malicious process has done its work and exited. ps shows you nothing: you are photographing an empty room.

This is the gap endpoint telemetry fills. Instead of asking a host "what is running right now?", telemetry continuously records the events themselves as they happen — every process that executes, every file written, every network connection opened, and the parent-child lineage tying them together — and ships them somewhere central. When the alert fires you don't interrogate the host; you query the recording. And because the recording is centralized, the same query runs across all four thousand hosts at once: that is hunting at scale.

Think of point-in-time triage as a single photograph and telemetry as the continuous CCTV recording. A photo of an empty corridor tells you nothing. The recording catches the intruder who walked through twenty minutes ago, shows which door they came from, and — because every camera in the building feeds one control room — lets you ask "where else in the building did this person appear?"

There is no single "Linux EDR" the way there is one Windows kernel API. Instead, a Linux endpoint agent stitches together several collection sources, each with a different vantage point. Knowing which source feeds your data store tells you what blind spots to expect.

• The audit subsystem (`auditd`) — the kernel's tamper-evident record of syscalls, watched-file access, and logins. It is rule-driven and authoritative for who did what, and it is covered in depth in the auditd lesson — this lesson treats it as one source among several, not the whole story.
• eBPF-based tooling — programs loaded into the kernel that attach to tracepoints, kprobes, and the scheduler to emit events with very low overhead. This is the modern firehose: bpftrace/bcc for ad-hoc tracing, and agents like Falco and Cilium Tetragon for continuous policy-driven detection.
• Sysmon for Linux (`sysmonforlinux`) — Microsoft's port of Sysmon, built on eBPF, that emits structured ProcessCreate, NetworkConnect, FileCreate and similar events into the systemd journal, configured by an XML ruleset. Familiar to Windows responders, schema-stable.
• Process accounting (`acct`/`psacct`) — the old-guard kernel feature that writes one fixed record per process as it exits to /var/log/account/pacct, read with lastcomm/sa. Lightweight and ubiquitous, but coarse: command name and resource use, no full argv, no network, no rich lineage.

The distinction that matters most: polling versus event-driven. ps polls — it reads /proc at the instant you run it, so anything that lived and died between two polls is invisible. eBPF, audit, and Sysmon are event-driven: the kernel pushes an event the moment a process is created, so nothing falls through the cracks between samples.

eBPF lets you run a small, verified program inside the kernel that fires on an event and hands a record back to user space — no polling, no kernel module to compile, minimal overhead. The classic worked example is `execsnoop`: it attaches to process execution and prints one line per execve across the whole system, in real time. It is the simplest possible demonstration of the firehose, and exactly the signal an EDR's process-create collector is built on.

Run execsnoop-bpfcc (the bcc tool) and in another terminal someone runs a command — even one that finishes instantly — and you see it appear.

execsnoop is great for an ad-hoc look, but a real pipeline needs structured, schema-stable events that a SIEM can parse the same way every time. Sysmon for Linux provides exactly that. Built on eBPF, driven by an XML config, it emits typed events — ProcessCreate (Event ID 1), NetworkConnect (Event ID 3), FileCreate (Event ID 11) and more — into the systemd journal under the sysmon identifier. Where ps gives you a process name, a ProcessCreate record gives you the full command line, the resolved image path, the parent image and command line, the user, and a hash — a complete, query-shaped story of one execution.

Step back from the tools and look at what kind of evidence telemetry recovers that point-in-time triage structurally cannot. There are three, and an attacker's playbook targets all of them.

• The process that already exited. A dropper runs, fetches a payload, and exits in seconds. ps never sees it; the exec event is recorded forever.
• The parent that spawned the shell and died. Lineage is the strongest single signal in intrusion detection — nginx spawning bash spawning sh -c "...|sh" is the shape of a webshell. But the parent often exits, and once it does, a live ps tree shows the child re-parented to PID 1 (init/systemd), erasing the link. Telemetry recorded the parent at exec time, so the lineage survives the parent's death.
• The short-lived beacon. A C2 implant may connect once, send a heartbeat, and close — a connection that exists for milliseconds and will never appear in ss or lsof. A NetworkConnect event captures it.

The analogy holds all the way down: the CCTV recording shows you the person who walked through and left, the door they came in by (even after that door closed), and the brief handoff in the corner that no still photo would ever freeze.

Now put it together as a workflow, because at fleet scale you cannot hand-triage every host — you triage the fleet. Two ideas anchor it.

An EDR detection is a starting line, not a finish line. EDRs ship detections: pre-built rules that fire an alert ("reverse shell indicator on web07"). That alert is a tiny, opinionated slice of what happened — one rule's view. Underneath it sits the raw telemetry: every process, file, and network event around that moment. The skill is to treat the alert as a pointer into the recording and pivot to the raw events to learn the full story the rule didn't encode.

A consistent triage set, collected the same way on every host. Rather than improvising on each box, scale response collects a standard artifact set — recent process events, network connections, persistence locations, logged-in users — uniformly across many hosts, so findings are comparable and a hunt query means the same thing everywhere.

The loop in practice:

1. Alert fires on web07 (Sysmon flagged the /dev/tcp reverse shell).

2. Pivot to raw telemetry on web07: pull every event around 03:11 and reconstruct the chain — webshell → bash → curl payload → reverse shell → rm cleanup.

3. Extract pivots: the C2 IP 198.51.100.55, the payload URL, the www-data parent, the command-line pattern.

4. Hunt the fleet: run those pivots as one query across all hosts' telemetry — which other hosts connected to that IP, or ran a `/dev/tcp` shell, or have that webshell process tree? That is how one alert becomes the scope of the whole incident.

You have a real root shell to your right. Fleet-scale telemetry tooling is mostly not installed in this sandbox — that is expected; it arrives with the DFIR lab image. So each command below is safe and inspection-only, and degrades gracefully: it shows real output where a source exists and a clear message where it doesn't. The goal is to make the "what telemetry surface does this host even have?" reflex automatic, since that inventory is the first thing you establish on any unfamiliar box.