LPIC-2 Advanced Network Troubleshooting: ss, tcpdump, nc, and Route Diagnosis

Advanced30 min7 steps

What you'll learn

Work a connectivity failure bottom-up — link, address, route, DNS, service — instead of guessing
Confirm a usable default route with ip route get before you ever suspect DNS
Map each diagnostic question to the right tool: ss, nc, tcpdump, dig/getent
Read nc's two failure modes correctly: CONNECTION REFUSED (service down) versus TIMEOUT (firewall drop)
Use -n everywhere so reverse-DNS never adds latency or hangs your diagnosis

Why a method beats a guess
Adopt the bottom-up layered method so you test causes in dependency order, not at random.
"The site is down" is not a diagnosis — it's a symptom with a dozen possible causes stacked on top of each other. The trap is to grab whatever tool you remember first (usually ping or dig) and start poking. If you guess wrong about which layer failed, every test after that is noise.
The fix is a method. Networking is a stack, and each layer depends on the one beneath it: a name can't resolve over a connection that has no route; a route is useless if the address layer never came up; the address layer needs a working link. So you test in that order, bottom-up:
- Link — is the interface up and carrying signal?
- Address — does the interface have an IP?
- Route — is there a path to the destination, including a default route?
- DNS — does the name resolve to an address?
- Service — is the thing on the other end actually listening?
Think of it like an electrician tracing a dead outlet: you don't start by replacing the lamp. You check the breaker, then the wiring, then the socket, then the bulb — upstream before downstream. Each confirmed-good layer eliminates a whole class of causes and tells you the fault is higher up.
Why it matters
The bottom-up order is not arbitrary — it follows the dependency chain. The moment you confirm a layer is healthy, everything below it is exonerated, and you've shrunk the search space. Guessing skips that elimination, which is why two engineers can stare at the same outage for an hour: they keep re-testing layers that were never the problem.
Note
There is a separate Network Troubleshooting guide covering the general first-pass workflow. This LPIC-2 lesson goes deeper on the part that separates juniors from seniors: reading nc's refused-vs-timeout signal and confirming it on the wire with tcpdump.
Confirm the route before you blame DNS
Prove a default route exists and is chosen for your destination — the layer everyone skips.
Here's the most common misdiagnosis in the field: a connection fails, the engineer runs dig, sees it hang, and declares "DNS is broken." But dig has to reach a resolver over the network — and if there's no route off the box, the DNS query never leaves either. The real fault is one layer down. Always confirm the route before you suspect DNS.
Don't just look at the routing table and eyeball it — ask the kernel what it would actually do for your specific destination. ip route get runs the real route-selection logic and tells you the exact path a packet would take.
```
$ ip route get 192.0.2.1
```
```
192.0.2.1 via 10.0.2.2 dev eth0 src 10.0.2.15 uid 0
    cache
```
route get <dest>
Run the kernel's real route lookup for one destination — not a table dump, the actual decision.
This is the healthy answer. The kernel says: to reach 192.0.2.1, send via 10.0.2.2 (the next-hop gateway) out dev eth0, using source address 10.0.2.15. A next-hop gateway plus an egress interface means the route layer is good — the fault is higher up, so now DNS or the service is a fair suspect.
```
$ ip route get 192.0.2.1
```
```
RTNETLINK answers: Network is unreachable
```
This is the broken answer, and it's decisive. Network is unreachable means there is no route to that destination — almost always a missing default route. Running dig now would also hang, and you'd wrongly blame DNS. Confirm with ip route (below): if there's no line starting default, that's your root cause.
```
$ ip route
```
```
default via 10.0.2.2 dev eth0 proto dhcp metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
```
The full table. The line that matters is `default via 10.0.2.2` — the catch-all route for any destination not on a directly-connected subnet (here, the 10.0.2.0/24 line). No `default` line = no internet, no off-subnet DNS, nothing. That single missing line explains an enormous share of "DNS is down" reports.
Blaming DNS when there's no default route
The classic false alarm: dig hangs, so DNS gets blamed. But dig needs to reach a resolver, and with no default route the query can't leave the box. The symptom looks identical to a DNS outage. Rule: a hung `dig` is your cue to run `ip route get`, not to restart `named`. Fix the route and the "DNS problem" vanishes.
Check yourself
A connection fails and dig example.com hangs with no answer. Before touching DNS, which command tells you whether a packet can even leave the box toward the destination?
Who's listening? ss -tlnp
Confirm the service is bound and listening on the expected port and address.
Once you know packets can reach the host, the next question is about the far end: is there actually a service listening on the port you're trying to hit? Run this on the server itself. ss (the modern replacement for netstat) lists sockets, and the right flags turn it into a precise "who is listening, where, and as what process" report.
```
$ sudo ss -tlnp
```
```
State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port  Process
LISTEN  0       128      127.0.0.1:5432        0.0.0.0:*          users:(("postgres",pid=842,fd=5))
LISTEN  0       511      0.0.0.0:80            0.0.0.0:*          users:(("nginx",pid=1001,fd=6))
LISTEN  0       128      0.0.0.0:22            0.0.0.0:*          users:(("sshd",pid=712,fd=3))
```
-t
TCP sockets only — where almost all services you'll diagnose live.
-l
Listening sockets only: things waiting for an inbound connection.
-n
Numeric ports and addresses — show :80, not "http". Skips reverse-DNS so the command never hangs on a name lookup.
-p
Show the owning process (name + PID). Needs root to see other users' sockets.
Read the Local Address:Port column closely. nginx and sshd listen on 0.0.0.0 — every interface, so they're reachable from the network. But postgres listens on `127.0.0.1:5432` — loopback only. A remote client will never connect to it no matter how perfect the route is, because the service isn't bound to a reachable address. That bind-address detail is the kind of fault ss exposes in one line.
127.0.0.1 vs 0.0.0.0 is a real bug, not a typo
A service bound to 127.0.0.1 answers only connections that originate on the same host. Remote clients get a connection refused — and admins burn hours on firewalls when the real fix is changing the listen address to 0.0.0.0 (or a specific interface IP). When ss shows loopback for something that's supposed to be remote-facing, you've found it.
Check yourself
ss -tlnp shows your database LISTENing on 127.0.0.1:5432, yet remote clients can't connect. Is the listening process down?
Is the port open? nc — and the RST-vs-timeout signal
Probe the port from the client side and decode nc's two failure modes correctly.
Now test from the client's perspective. nc (netcat) makes a single TCP connection attempt and reports what happened. This is the highest-value diagnostic in the whole lesson, because how it fails tells you exactly which layer to fix. First, success:
```
$ nc -vz 192.0.2.10 80
```
```
Connection to 192.0.2.10 80 port [tcp/*] succeeded!
```
-v
Verbose — print whether the connection succeeded or how it failed (you need this to read the result).
-z
Zero-I/O "scan" mode: just test if the port accepts a connection, then close it — send no data.
succeeded! means the full TCP handshake completed — route, firewall, and a listening service are all good. If this passes but the app still misbehaves, your problem is at the application layer, not the network.
The two failures look superficially similar ("it didn't connect") but have opposite causes. This distinction is the heart of advanced network troubleshooting.
```
$ nc -vz 192.0.2.10 80
```
```
nc: connect to 192.0.2.10 port 80 (tcp) failed: Connection refused
```
Connection refused comes back fast. It means the host replied with a TCP RST (reset): "I received your packet, but nothing is listening on that port." The packet reached the host — so the route and firewall are fine. The fault is the service: it's stopped, crashed, or (as the previous step showed) bound to the wrong address. Go fix the daemon, not the firewall.
```
$ nc -vz 192.0.2.10 80
```
```
nc: connect to 192.0.2.10 port 80 (tcp) failed: Connection timed out
```
Connection timed out hangs for seconds and then gives up. It means no reply came back at all — not even a polite refusal. Something is silently dropping your packet, classically a firewall with a DROP rule (a REJECT rule would have sent the RST you saw above). The route may be fine and the service may be running; a packet filter in the middle is eating the SYN. Go look at firewall rules — iptables/nftables on the host or a network ACL in the path.
The one distinction that defines this lesson: refused ≠ timeout
Connection refused = a TCP RST came back = the host is reachable but the service is down. Fix the service.
Connection timed out = nothing came back = a firewall is silently DROPPING the packet. Fix the firewall.
Misreading a timeout as "the service is down" sends you restarting a daemon that was never the problem, while a DROP rule keeps eating every packet. The tell is time: refused is instant (the host answered); timeout makes you wait (the host, or a box in front of it, never answered).
Check yourself
nc -vz to a port returns "Connection refused" almost instantly. Is a firewall silently dropping your packets, or is the service down?
Is the packet even arriving? tcpdump -ni
Confirm on the wire whether the SYN reaches the host and whether anything replies.
When nc times out, tcpdump settles the argument by showing the actual packets on the interface. It's the ground truth: a CCTV camera on the network door. Run it on the server and watch what arrives while you probe from the client.
The critical habit: always pass `-n`. Without it, tcpdump tries to reverse-DNS every address it sees — which adds latency and, when DNS itself is the problem you're debugging, can make tcpdump hang before it prints a single packet. -n turns that off and shows raw IPs immediately.
```
$ sudo tcpdump -ni eth0 'tcp port 80'
```
```
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:42:07.013 IP 192.0.2.50.51884 > 192.0.2.10.80: Flags [S], seq 0, win 64240, length 0
11:42:08.015 IP 192.0.2.50.51884 > 192.0.2.10.80: Flags [S], seq 0, win 64240, length 0
11:42:10.019 IP 192.0.2.50.51884 > 192.0.2.10.80: Flags [S], seq 0, win 64240, length 0
```
-n
Don't resolve addresses or ports to names — show raw IPs. Avoids latency and the hang that reverse-DNS causes when DNS is the very thing that's broken.
-i eth0
Capture on this specific interface — match it to the egress device ip route get reported.
'tcp port 80'
A BPF filter: capture only TCP traffic on port 80 so the output isn't drowned in noise.
This is the firewall-drop signature. The client at 192.0.2.50 sends a SYN (Flags [S]) — and you see it arrive at the host. But there's no `Flags [S.]` reply (SYN-ACK), and the client just retransmits the same SYN at 1s, 2s, 4s intervals. The packet got in but the host never answered: confirmation that something is dropping it (a DROP firewall rule), which is exactly the nc timeout case proven on the wire.
Why it matters
tcpdump collapses a guessing game into a fact. SYN arrives but no reply = packet reaches the host, firewall/service drops it. No SYN arrives at all = the packet never made it here, so the problem is upstream (routing, or a firewall on a box between you and the host). SYN arrives and a RST comes back = the host actively refused — the service is down. You've matched each nc result to a packet-level cause.
Omitting -n makes tcpdump hang on reverse-DNS
Run tcpdump without `-n` while DNS is broken and it tries to reverse-resolve every IP it captures. Each lookup blocks waiting for a resolver that isn't answering, so tcpdump can sit there printing nothing — and you conclude "no packets are arriving" when they're arriving fine; the tool is just stuck on name resolution. `-n` is mandatory during a DNS-suspect outage. The same reasoning is why you pass -n to ss and traceroute too.
Check yourself
Why must you pass -n to tcpdump (and ss, and traceroute) when DNS might be the problem you're debugging?
Does the name resolve? dig and getent
Separate a true name-resolution failure from the route/service faults already ruled out.
Only now — route confirmed, service and firewall understood — is DNS a fair suspect. Two tools answer two different questions, and conflating them hides bugs:
- dig asks a DNS server directly, bypassing the local config.
- getent hosts resolves the way your applications actually do — through /etc/nsswitch.conf, which means it honors /etc/hosts and the configured resolver order.
```
$ dig +short example.com
```
```
192.0.2.10
```
+short
Print only the answer (the IP), suppressing the verbose query/header sections.
An IP comes back, so DNS resolution works at the server level. An empty result instead would mean the name didn't resolve — but read that carefully against the route step: if ip route get already showed Network is unreachable, an empty dig is a symptom of the routing fault, not a DNS fault. Layer order keeps you honest.
```
$ getent hosts example.com
```
```
192.0.2.10  example.com
```
getent resolves through the name service switch — the exact path your apps use. The lesson is when dig and getent disagree: if dig returns the public IP but getent returns a different address (or nothing), the difference lives in `/etc/hosts` or `/etc/nsswitch.conf`, not in DNS. dig tests the server; getent tests what your software will actually get.
When dig works but the app doesn't, trust getent
dig talks straight to a nameserver, so it can succeed while your application fails — because the app goes through nsswitch, where a stale /etc/hosts line or a misordered hosts: directive overrides DNS. getent hosts reproduces the app's real answer. Comparing the two pinpoints local config bugs that dig alone will never reveal.
Try it in your live terminal
Walk the bottom-up method yourself with safe, read-only commands on the real sandbox.
You have a real root shell to your right. Every command below is read-only — it inspects, changes nothing. Run them in bottom-up order and narrate the answer to yourself at each layer: route → listening → name resolution. That sequence is the muscle memory this lesson is building.
```
$ ip route
```
Layer: route. Look for a line starting default via …. If it's missing, stop — nothing off-subnet will work, and any "DNS" symptom is really this.
```
$ ip route get 192.0.2.1
```
Layer: route (decisive). Ask the kernel the real path to a destination. A via … dev … answer means routing is healthy; Network is unreachable means it isn't — that's your root cause, not DNS.
```
$ ss -tlnp
```
Layer: service. List every listening TCP socket and its owning process. Check each bind address — 0.0.0.0 is remote-reachable, 127.0.0.1 is local-only. Note the -n so it never stalls on a name lookup.
```
$ getent hosts localhost; echo "--- example.com ---"; getent hosts example.com; echo "getent exit code: $?"
```
Layer: DNS, the app's-eye view. getent resolves through nsswitch exactly as your software would. localhost answers from /etc/hosts. example.com prints nothing and exits 2 (not found), because lab boxes have no internet access and its lookup has to go to DNS. Now compare dig example.com +time=2 +tries=1, which asks the nameserver directly and shows the failure in its status: line. When getent and dig disagree, look at /etc/hosts and /etc/nsswitch.conf, not the nameserver.

Recap

Troubleshoot bottom-up — link, address, route, DNS, service — so each confirmed layer eliminates a whole class of causes.
Confirm the route first. A hung dig is a cue to run ip route get <dest>, not to restart DNS — a missing default route mimics a DNS outage perfectly.
`nc` refused (RST) = service down (host is reachable); `nc` timeout (no reply) = firewall silently dropping the packet. The two failures have opposite fixes.
`tcpdump -ni` is ground truth: SYN-arrives-no-reply confirms a drop; a returned RST confirms refusal; no SYN at all means the problem is upstream.
Pass `-n` everywhere (tcpdump, ss, traceroute). Reverse-DNS adds latency and will hang your tools when DNS is the very thing that's broken.

Symptom → tool → meaning
Symptom you see	Tool / command	What it actually means
`dig` hangs, connection fails	`ip route get <dest>`	Likely no default route — fix routing, not DNS
No `default via` line	`ip route`	No path off the subnet; off-subnet DNS and internet are dead
Service unreachable remotely	`ss -tlnp`	Check bind: `0.0.0.0` = remote-OK, `127.0.0.1` = local-only bug
`nc` says Connection refused (fast)	`nc -vz host port`	TCP RST — host reachable, service is down
`nc` says Connection timed out (hangs)	`nc -vz host port`	No reply — firewall is dropping the packet
SYN arrives, no SYN-ACK back	`tcpdump -ni <if>`	Packet drop confirmed on the wire (DROP rule)
`dig` works but the app can't resolve	`getent hosts <name>`	Difference is in `/etc/hosts` or `nsswitch.conf`, not DNS

LPIC-2 Advanced Network Troubleshooting: ss, tcpdump, nc, and Route Diagnosis

Why a method beats a guess

Confirm the route before you blame DNS

Who's listening? ss -tlnp

Is the port open? nc — and the RST-vs-timeout signal

Is the packet even arriving? tcpdump -ni

Does the name resolve? dig and getent

Try it in your live terminal