Network Troubleshooting
- Diagnose "the network is broken" with a repeatable, layer-by-layer method instead of guessing
- Check each layer in order — link, address/route, reachability, name resolution, service — and stop at the one that fails
- Tell a connectivity problem (ping/traceroute) apart from a DNS problem (dig/resolv.conf)
- Know when to stop reasoning and watch the actual packets with tcpdump
Why you work UP the stack
Frame troubleshooting as isolating the failing layer, not guessing at fixes.
"The network is broken" is not a diagnosis — it's a symptom that could come from any of five different layers. The slow way to fix it is to guess: restart the interface, flush DNS, reboot, reach for
tcpdump, and hope one of them lands. The fast way is to test each layer in order and stop at the first one that fails. Everything above a broken layer is broken too, so once you find the bottom failure you've found the cause.Think of your connection as a water supply running from the city main to your kitchen tap. If no water comes out, you don't tear open the faucet first. You walk the path backwards: is the main on? Is the valve into the house open? Is the pipe to the kitchen clear? You test each junction in order until you hit the dry one. Networking is the same — you just walk it from the bottom up.
The five junctions, bottom to top:
- LINK — is the network cable/interface physically up? (
ip link) - IP / ROUTE — do I have an address and a way out? (
ip addr,ip route) - REACHABILITY — can packets actually get there and back? (
ping) - NAME RESOLUTION — can I turn names into addresses? (
dig) - SERVICE / APP — is the thing I want actually listening? (
ss,curl)
- LINK — is the network cable/interface physically up? (
Layer 1 — is the link up?
Confirm the interface itself is present and physically up before anything else.
Start at the bottom: the interface — the kernel's handle on your network card. If it's
DOWN, no address, route, or DNS query can possibly work, so there's no point looking higher.ip linkshows every interface and its state.ip link show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT link/ether 52:54:00:a1:b2:c3 brd ff:ff:ff:ff:ff:ff- link
- The link-layer object: interfaces and their up/down state, MAC address, MTU.
- show
- List rather than change. The default verb, so
ip linkalone works too.
Two things to read. `state UP` on eth0means the kernel has the interface administratively up, and `LOWER_UP` (inside the angle brackets) means the physical carrier is present — a cable is plugged in or the virtual link is live. If you sawstate DOWNor theLOWER_UPflag were missing, you'd stop right here: nothing above this layer can work until the link is up.lois the loopback and is always present.Check yourself
In
ip linkoutput, which field tells you the interface is administratively enabled and ready to carry traffic?Layer 2 — address and default route
Confirm you have an IP and a gateway to send off-subnet traffic to.
An up interface with no IP address can't send anything. And even with an address, you can only reach machines on your own subnet unless you have a default route — the gateway that off-subnet traffic (i.e. the whole rest of the internet) gets handed to. These are two separate checks. In the water analogy: an address is having a pipe connected; the default route is the valve to the street main.
ip addr show eth02: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP link/ether 52:54:00:a1:b2:c3 brd ff:ff:ff:ff:ff:ff inet 192.0.2.50/24 brd 192.0.2.255 scope global eth0- addr
- The address-layer object: IP addresses bound to each interface.
- show eth0
- Limit output to one interface instead of listing all of them.
The `inet 192.0.2.50/24` line is what you're looking for: this host has the IPv4 address 192.0.2.50with a /24 netmask. If this line were missing, the interface is up but has no address — you'd be chasing a DHCP or static-config problem, not anything higher up. The/24tells you the local subnet is192.0.2.0/24.ip route showdefault via 192.0.2.1 dev eth0 proto dhcp metric 100 192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.50
- route
- The routing table: which next-hop the kernel uses for a given destination.
- show
- List the routes (default verb).
The `default via 192.0.2.1` line is the gateway: anything not on the local 192.0.2.0/24subnet gets sent to192.0.2.1to be forwarded onward. No `default` line = no way off your own subnet — local hosts would work but the internet would be unreachable, a very common and very specific failure. Note the gateway address (192.0.2.1); it's your next test target.Check yourself
Your host has an IP and can reach other machines on its LAN, but nothing on the internet. Which line in
ip routeoutput is most likely missing?Layer 3 — reachability with ping
Probe outward in steps — gateway first, then a known external IP — to localize where packets die.
Now test whether packets actually make a round trip. Ping outward in stages, nearest-first, so a failure tells you where the path breaks:
1. Ping your gateway (
192.0.2.1) — proves the link to your first hop works.2. Ping a known external IP (here
192.0.2.1stands in for any reliable off-net address) — proves routing past the gateway works.Using a raw IP address here, never a name, is deliberate: it keeps DNS out of the picture so you're testing pure connectivity. (We test names separately in the next step — that separation is the heart of this lesson.)
ping -c1 192.0.2.1PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data. 64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=0.43 ms --- 192.0.2.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms
- -c1
- Send exactly 1 packet and exit, instead of pinging forever — scriptable and quick.
`1 received, 0% packet loss` means a packet went to the gateway and came back: layer-2/3 connectivity to your first hop is healthy. If this returned 100% packet lossorDestination Host Unreachable, the problem is between you and the gateway — back down to link/route, not up to DNS. The lowtime=0.43 msconfirms it's a local hop.Example — not runnable on your lab box
traceroute 192.0.2.1traceroute to 192.0.2.1 (192.0.2.1), 30 hops max, 60 byte packets 1 192.0.2.1 (192.0.2.1) 0.402 ms 0.388 ms 0.351 ms
- (target)
- The destination to trace toward; each router hop along the way prints one line.
Where pinganswers can I reach it?,tracerouteanswers *where does the path die? Each line is one router hop. If reachability fails partway to an external host, `traceroute` shows the last hop that answered before a run of `timeouts — that's the point the path breaks.tracepathis a near-identical, unprivileged alternative iftraceroute` isn't installed.Check yourself
Why ping a raw IP like 192.0.2.1 rather than a hostname when testing connectivity?
Layer 4 — name resolution with dig
Prove the signature DNS fault: pings by IP succeed, but names fail.
Here's the most common real-world "the internet is down" call that isn't a connectivity problem at all. The tell:
ping -c1 192.0.2.1(by IP) → worksping -c1 example.com(by name) → fails with "Name or service not known"
If packets reach an IP fine but anything by name fails, connectivity is healthy — what's broken is the translation from names to addresses, i.e. DNS. Connectivity and naming are different layers, and this exact split is how you tell them apart in one move.
dig +short example.com;; communications error to 192.0.2.53#53: timed out ;; no servers could be reached
- +short
- Print just the answer (or the error), not dig's full multi-section report — ideal for a quick check.
A healthy lookup prints an IP address. Here, dig can't even reach its configured resolver ( 192.0.2.53) — the lookup times out. That confirms the diagnosis: the network carries packets fine (your gateway ping worked) but name resolution is failing. The next question is why the resolver is unreachable — which sends you to the resolver config.dig example.com; <<>> DiG 9.18.24 <<>> example.com ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41822 ;; QUESTION SECTION: ;example.com. IN A ;; ANSWER SECTION: example.com. 300 IN A 192.0.2.80 ;; Query time: 24 msec ;; SERVER: 192.0.2.53#53(192.0.2.53)
When DNS is working, read two things: the `status:` in the header and the ANSWER SECTION. status: NOERRORplus anArecord in the ANSWER section (example.com → 192.0.2.80) means a clean resolution. Two failure statuses matter: `NXDOMAIN` = the name genuinely doesn't exist (a typo, or a dead record); `SERVFAIL` = the resolver tried but couldn't complete (broken upstream or DNSSEC). An empty ANSWER section withNOERRORmeans the name exists but has no record of that type.cat /etc/resolv.conf# Generated by NetworkManager nameserver 192.0.2.53 search corp.example.com
- (file)
- /etc/resolv.conf lists the DNS servers (nameserver lines) the system queries, in order.
This is where DNS gets its servers. The `nameserver 192.0.2.53` line is the resolver dig was trying (and failing) to reach. A wrong, stale, or unreachable nameserverentry here is a textbook cause of "ping by IP works, by name fails". Fixing DNS often means correcting this one line (or the config that generates it).Check yourself
ping -c1 192.0.2.1succeeds, butping example.comfails with "Name or service not known". Which layer is broken?Layer 5 — service, and when to watch packets
Confirm the target app is actually listening, and know when tcpdump earns its place.
You've reached the top. The network delivers packets and names resolve, but "I can't connect to the web app" can still mean the service itself isn't there. Two checks finish the climb: is something listening on the expected port locally (
ss), and does an actual request get a response (curl)?ss -tlnpState Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=701,fd=3)) LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=812,fd=6))- -t
- TCP sockets only.
- -l
- Listening sockets only — services waiting for a connection.
- -n
- Numeric ports (80, not "http") so nothing is disguised behind a name.
- -p
- Show the owning process/PID for each socket (needs root to see others').
Each LISTENline is a service accepting connections. Herenginxis listening on `0.0.0.0:80` — present and reachable on every interface. If the port you expected (say 80) is absent from this list, the service is down or never started — the problem is the application, not the network, and you'd go restart the service.0.0.0.0means all interfaces;127.0.0.1would mean localhost-only, a common "works locally, refused remotely" cause.curl -sS -o /dev/null -w '%{http_code}\n' http://192.0.2.80/200
- -sS
- Silent, but still show errors — no progress bar, but failures aren't swallowed.
- -o /dev/null
- Discard the response body; we only care whether it answered.
- -w '%{http_code}\n'
- Print just the HTTP status code the server returned.
`200` means the service answered successfully end-to-end — the full stack works. A Connection refusedwould mean nothing is listening on that port (back toss); a hang/timeout points at a firewall silently dropping packets.curltests the application layer the way a real client would.Only now — when the lower layers check out but something still doesn't add up (a silent drop, a one-way connection, a handshake that never completes) — do you reach for `tcpdump` and watch the actual packets on the wire. It's the most powerful tool here, which is exactly why it's last: it shows you everything, so you want to use it once you've narrowed where to look, not as a shot in the dark.
sudo tcpdump -ni eth0 host 192.0.2.80 and port 80 -c5tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:48:02.114 IP 192.0.2.50.51992 > 192.0.2.80.80: Flags [S], seq 1, win 64240 11:48:02.115 IP 192.0.2.80.80 > 192.0.2.50.51992: Flags [S.], seq 1, ack 2, win 65160 11:48:02.115 IP 192.0.2.50.51992 > 192.0.2.80.80: Flags [.], ack 1, win 502
- -n
- Don't resolve addresses/ports to names — show raw IPs and avoid a blocking lookup.
- -i eth0
- Capture on this specific interface.
- host ... and port ...
- A capture filter so you see only the traffic you care about, not everything.
- -c5
- Stop after 5 packets so the capture ends on its own.
These three lines are a healthy TCP handshake: `[S]` (SYN out), `[S.]` (SYN-ACK back), `[.]` (ACK out) — connection established. If you only ever saw the outbound [S]with no reply, the server isn't answering (firewall drop or service down); seeing[R](reset) means the port is actively refusing. tcpdump is where you confirm what the higher-layer tools only implied.Try it in your live terminal
Walk the whole stack yourself, bottom to top, on the real sandbox to your right.
You have a real shell. Run these in order — this is the exact bottom-up sweep you'd do on a live "the network's broken" call. Each command checks one layer; the first one that fails is your culprit. All of these are read-only and safe to run.
ip -br addrLINK + ADDRESS at a glance: -br(brief) shows every interface, its UP/DOWN state, and its IP on one line each. Confirm your interface is UP and has an address before going further.ip routeROUTE: confirm there's a default via ...line. No default route = no path off your subnet.ping -c1 192.0.2.1REACHABILITY by IP: proves connectivity without involving DNS. Substitute your real gateway/target. Loss here = a connectivity (not naming) problem. grep nameserver /etc/resolv.conf; dig example.com +time=2 +tries=1; echo "dig exit code: $?"NAME RESOLUTION: which server you ask, then a lookup. On a connected host, status: NOERRORwith an A record in the ANSWER SECTION means DNS works, and a failure here while ping-by-IP succeeded is the classic DNS-only fault. This sandbox has no internet access, so here the lookup fails (status: SERVFAIL/REFUSED, orno servers could be reachedwith exit code 9). The ping above failed too, so the sweep's verdict is reachability, not DNS: the lowest failing layer is the culprit.ss -tlnpSERVICE: every listening port and the process behind it. Confirm the service you expect is actually present and bound where you think.
- "The network is broken" is a symptom, not a diagnosis — work UP the stack and the first layer that fails is your root cause.
- Test connectivity with raw IPs (
ping/traceroute) and names separately (dig); keeping them apart is what isolates DNS. - Ping by IP works but by name fails = DNS. It's the most over-diagnosed outage there is; run the IP-vs-name test before you escalate.
tcpdumpis the most powerful tool and therefore the LAST one — use it to confirm where you've already narrowed the fault, not to start the hunt.
| Layer | Symptom when it fails | Command to check |
|---|---|---|
| Link | Interface DOWN — nothing works at all | ip link show |
| IP / Route | No address, or LAN works but internet doesn't (no default route) | ip addr / ip route |
| Reachability | Packets to an IP time out or are unreachable | ping -c1 <ip> / traceroute |
| Name resolution | Ping by IP works, by name fails (NXDOMAIN/SERVFAIL/timeout) | dig <name> / cat /etc/resolv.conf |
| Service / App | Connectivity + DNS fine, but the app won't connect | ss -tlnp / curl |
| Ground truth | Lower layers look fine but something still drops silently | tcpdump -ni <if> host <ip> |
