Signal9Signal9 — home
online
all lessons

Network Troubleshooting

Intermediate40 min7 steps
What you'll learn
  • Diagnose "the network is broken" with a repeatable, layer-by-layer method instead of guessing
  • Check each layer in order — link, address/route, reachability, name resolution, service — and stop at the one that fails
  • Tell a connectivity problem (ping/traceroute) apart from a DNS problem (dig/resolv.conf)
  • Know when to stop reasoning and watch the actual packets with tcpdump
  1. Why you work UP the stack

    Frame troubleshooting as isolating the failing layer, not guessing at fixes.

    "The network is broken" is not a diagnosis — it's a symptom that could come from any of five different layers. The slow way to fix it is to guess: restart the interface, flush DNS, reboot, reach for tcpdump, and hope one of them lands. The fast way is to test each layer in order and stop at the first one that fails. Everything above a broken layer is broken too, so once you find the bottom failure you've found the cause.

    Think of your connection as a water supply running from the city main to your kitchen tap. If no water comes out, you don't tear open the faucet first. You walk the path backwards: is the main on? Is the valve into the house open? Is the pipe to the kitchen clear? You test each junction in order until you hit the dry one. Networking is the same — you just walk it from the bottom up.

    The five junctions, bottom to top:

    • LINK — is the network cable/interface physically up? (ip link)
    • IP / ROUTE — do I have an address and a way out? (ip addr, ip route)
    • REACHABILITY — can packets actually get there and back? (ping)
    • NAME RESOLUTION — can I turn names into addresses? (dig)
    • SERVICE / APP — is the thing I want actually listening? (ss, curl)
  2. Layer 1 — is the link up?

    Confirm the interface itself is present and physically up before anything else.

    Start at the bottom: the interface — the kernel's handle on your network card. If it's DOWN, no address, route, or DNS query can possibly work, so there's no point looking higher. ip link shows every interface and its state.

    ip link show
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT
        link/ether 52:54:00:a1:b2:c3 brd ff:ff:ff:ff:ff:ff
    link
    The link-layer object: interfaces and their up/down state, MAC address, MTU.
    show
    List rather than change. The default verb, so ip link alone works too.
    Two things to read. `state UP` on eth0 means the kernel has the interface administratively up, and `LOWER_UP` (inside the angle brackets) means the physical carrier is present — a cable is plugged in or the virtual link is live. If you saw state DOWN or the LOWER_UP flag were missing, you'd stop right here: nothing above this layer can work until the link is up. lo is the loopback and is always present.

    Check yourself

    In ip link output, which field tells you the interface is administratively enabled and ready to carry traffic?

  3. Layer 2 — address and default route

    Confirm you have an IP and a gateway to send off-subnet traffic to.

    An up interface with no IP address can't send anything. And even with an address, you can only reach machines on your own subnet unless you have a default route — the gateway that off-subnet traffic (i.e. the whole rest of the internet) gets handed to. These are two separate checks. In the water analogy: an address is having a pipe connected; the default route is the valve to the street main.

    ip addr show eth0
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
        link/ether 52:54:00:a1:b2:c3 brd ff:ff:ff:ff:ff:ff
        inet 192.0.2.50/24 brd 192.0.2.255 scope global eth0
    addr
    The address-layer object: IP addresses bound to each interface.
    show eth0
    Limit output to one interface instead of listing all of them.
    The `inet 192.0.2.50/24` line is what you're looking for: this host has the IPv4 address 192.0.2.50 with a /24 netmask. If this line were missing, the interface is up but has no address — you'd be chasing a DHCP or static-config problem, not anything higher up. The /24 tells you the local subnet is 192.0.2.0/24.
    ip route show
    default via 192.0.2.1 dev eth0 proto dhcp metric 100
    192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.50
    route
    The routing table: which next-hop the kernel uses for a given destination.
    show
    List the routes (default verb).
    The `default via 192.0.2.1` line is the gateway: anything not on the local 192.0.2.0/24 subnet gets sent to 192.0.2.1 to be forwarded onward. No `default` line = no way off your own subnet — local hosts would work but the internet would be unreachable, a very common and very specific failure. Note the gateway address (192.0.2.1); it's your next test target.

    Check yourself

    Your host has an IP and can reach other machines on its LAN, but nothing on the internet. Which line in ip route output is most likely missing?

  4. Layer 3 — reachability with ping

    Probe outward in steps — gateway first, then a known external IP — to localize where packets die.

    Now test whether packets actually make a round trip. Ping outward in stages, nearest-first, so a failure tells you where the path breaks:

    1. Ping your gateway (192.0.2.1) — proves the link to your first hop works.

    2. Ping a known external IP (here 192.0.2.1 stands in for any reliable off-net address) — proves routing past the gateway works.

    Using a raw IP address here, never a name, is deliberate: it keeps DNS out of the picture so you're testing pure connectivity. (We test names separately in the next step — that separation is the heart of this lesson.)

    ping -c1 192.0.2.1
    PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data.
    64 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=0.43 ms
    
    --- 192.0.2.1 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    -c1
    Send exactly 1 packet and exit, instead of pinging forever — scriptable and quick.
    `1 received, 0% packet loss` means a packet went to the gateway and came back: layer-2/3 connectivity to your first hop is healthy. If this returned 100% packet loss or Destination Host Unreachable, the problem is between you and the gateway — back down to link/route, not up to DNS. The low time=0.43 ms confirms it's a local hop.

    Example — not runnable on your lab box

    traceroute 192.0.2.1
    traceroute to 192.0.2.1 (192.0.2.1), 30 hops max, 60 byte packets
     1  192.0.2.1 (192.0.2.1)  0.402 ms  0.388 ms  0.351 ms
    (target)
    The destination to trace toward; each router hop along the way prints one line.
    Where ping answers can I reach it?, traceroute answers *where does the path die? Each line is one router hop. If reachability fails partway to an external host, `traceroute` shows the last hop that answered before a run of ` timeouts — that's the point the path breaks. tracepath is a near-identical, unprivileged alternative if traceroute` isn't installed.

    Check yourself

    Why ping a raw IP like 192.0.2.1 rather than a hostname when testing connectivity?

  5. Layer 4 — name resolution with dig

    Prove the signature DNS fault: pings by IP succeed, but names fail.

    Here's the most common real-world "the internet is down" call that isn't a connectivity problem at all. The tell:

    • ping -c1 192.0.2.1 (by IP) → works
    • ping -c1 example.com (by name) → fails with "Name or service not known"

    If packets reach an IP fine but anything by name fails, connectivity is healthy — what's broken is the translation from names to addresses, i.e. DNS. Connectivity and naming are different layers, and this exact split is how you tell them apart in one move.

    dig +short example.com
    ;; communications error to 192.0.2.53#53: timed out
    
    ;; no servers could be reached
    +short
    Print just the answer (or the error), not dig's full multi-section report — ideal for a quick check.
    A healthy lookup prints an IP address. Here, dig can't even reach its configured resolver (192.0.2.53) — the lookup times out. That confirms the diagnosis: the network carries packets fine (your gateway ping worked) but name resolution is failing. The next question is why the resolver is unreachable — which sends you to the resolver config.
    dig example.com
    ; <<>> DiG 9.18.24 <<>> example.com
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41822
    
    ;; QUESTION SECTION:
    ;example.com.			IN	A
    
    ;; ANSWER SECTION:
    example.com.		300	IN	A	192.0.2.80
    
    ;; Query time: 24 msec
    ;; SERVER: 192.0.2.53#53(192.0.2.53)
    When DNS is working, read two things: the `status:` in the header and the ANSWER SECTION. status: NOERROR plus an A record in the ANSWER section (example.com → 192.0.2.80) means a clean resolution. Two failure statuses matter: `NXDOMAIN` = the name genuinely doesn't exist (a typo, or a dead record); `SERVFAIL` = the resolver tried but couldn't complete (broken upstream or DNSSEC). An empty ANSWER section with NOERROR means the name exists but has no record of that type.
    cat /etc/resolv.conf
    # Generated by NetworkManager
    nameserver 192.0.2.53
    search corp.example.com
    (file)
    /etc/resolv.conf lists the DNS servers (nameserver lines) the system queries, in order.
    This is where DNS gets its servers. The `nameserver 192.0.2.53` line is the resolver dig was trying (and failing) to reach. A wrong, stale, or unreachable nameserver entry here is a textbook cause of "ping by IP works, by name fails". Fixing DNS often means correcting this one line (or the config that generates it).

    Check yourself

    ping -c1 192.0.2.1 succeeds, but ping example.com fails with "Name or service not known". Which layer is broken?

  6. Layer 5 — service, and when to watch packets

    Confirm the target app is actually listening, and know when tcpdump earns its place.

    You've reached the top. The network delivers packets and names resolve, but "I can't connect to the web app" can still mean the service itself isn't there. Two checks finish the climb: is something listening on the expected port locally (ss), and does an actual request get a response (curl)?

    ss -tlnp
    State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
    LISTEN  0       128      0.0.0.0:22           0.0.0.0:*         users:(("sshd",pid=701,fd=3))
    LISTEN  0       511      0.0.0.0:80           0.0.0.0:*         users:(("nginx",pid=812,fd=6))
    -t
    TCP sockets only.
    -l
    Listening sockets only — services waiting for a connection.
    -n
    Numeric ports (80, not "http") so nothing is disguised behind a name.
    -p
    Show the owning process/PID for each socket (needs root to see others').
    Each LISTEN line is a service accepting connections. Here nginx is listening on `0.0.0.0:80` — present and reachable on every interface. If the port you expected (say 80) is absent from this list, the service is down or never started — the problem is the application, not the network, and you'd go restart the service. 0.0.0.0 means all interfaces; 127.0.0.1 would mean localhost-only, a common "works locally, refused remotely" cause.
    curl -sS -o /dev/null -w '%{http_code}\n' http://192.0.2.80/
    200
    -sS
    Silent, but still show errors — no progress bar, but failures aren't swallowed.
    -o /dev/null
    Discard the response body; we only care whether it answered.
    -w '%{http_code}\n'
    Print just the HTTP status code the server returned.
    `200` means the service answered successfully end-to-end — the full stack works. A Connection refused would mean nothing is listening on that port (back to ss); a hang/timeout points at a firewall silently dropping packets. curl tests the application layer the way a real client would.

    Only now — when the lower layers check out but something still doesn't add up (a silent drop, a one-way connection, a handshake that never completes) — do you reach for `tcpdump` and watch the actual packets on the wire. It's the most powerful tool here, which is exactly why it's last: it shows you everything, so you want to use it once you've narrowed where to look, not as a shot in the dark.

    sudo tcpdump -ni eth0 host 192.0.2.80 and port 80 -c5
    tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
    11:48:02.114 IP 192.0.2.50.51992 > 192.0.2.80.80: Flags [S], seq 1, win 64240
    11:48:02.115 IP 192.0.2.80.80 > 192.0.2.50.51992: Flags [S.], seq 1, ack 2, win 65160
    11:48:02.115 IP 192.0.2.50.51992 > 192.0.2.80.80: Flags [.], ack 1, win 502
    -n
    Don't resolve addresses/ports to names — show raw IPs and avoid a blocking lookup.
    -i eth0
    Capture on this specific interface.
    host ... and port ...
    A capture filter so you see only the traffic you care about, not everything.
    -c5
    Stop after 5 packets so the capture ends on its own.
    These three lines are a healthy TCP handshake: `[S]` (SYN out), `[S.]` (SYN-ACK back), `[.]` (ACK out) — connection established. If you only ever saw the outbound [S] with no reply, the server isn't answering (firewall drop or service down); seeing [R] (reset) means the port is actively refusing. tcpdump is where you confirm what the higher-layer tools only implied.
  7. Try it in your live terminal

    Walk the whole stack yourself, bottom to top, on the real sandbox to your right.

    You have a real shell. Run these in order — this is the exact bottom-up sweep you'd do on a live "the network's broken" call. Each command checks one layer; the first one that fails is your culprit. All of these are read-only and safe to run.

    ip -br addr
    LINK + ADDRESS at a glance: -br (brief) shows every interface, its UP/DOWN state, and its IP on one line each. Confirm your interface is UP and has an address before going further.
    ip route
    ROUTE: confirm there's a default via ... line. No default route = no path off your subnet.
    ping -c1 192.0.2.1
    REACHABILITY by IP: proves connectivity without involving DNS. Substitute your real gateway/target. Loss here = a connectivity (not naming) problem.
    grep nameserver /etc/resolv.conf; dig example.com +time=2 +tries=1; echo "dig exit code: $?"
    NAME RESOLUTION: which server you ask, then a lookup. On a connected host, status: NOERROR with an A record in the ANSWER SECTION means DNS works, and a failure here while ping-by-IP succeeded is the classic DNS-only fault. This sandbox has no internet access, so here the lookup fails (status: SERVFAIL/REFUSED, or no servers could be reached with exit code 9). The ping above failed too, so the sweep's verdict is reachability, not DNS: the lowest failing layer is the culprit.
    ss -tlnp
    SERVICE: every listening port and the process behind it. Confirm the service you expect is actually present and bound where you think.
Recap
  • "The network is broken" is a symptom, not a diagnosis — work UP the stack and the first layer that fails is your root cause.
  • Test connectivity with raw IPs (ping/traceroute) and names separately (dig); keeping them apart is what isolates DNS.
  • Ping by IP works but by name fails = DNS. It's the most over-diagnosed outage there is; run the IP-vs-name test before you escalate.
  • tcpdump is the most powerful tool and therefore the LAST one — use it to confirm where you've already narrowed the fault, not to start the hunt.
Layer → symptom → command
LayerSymptom when it failsCommand to check
LinkInterface DOWN — nothing works at allip link show
IP / RouteNo address, or LAN works but internet doesn't (no default route)ip addr / ip route
ReachabilityPackets to an IP time out or are unreachableping -c1 <ip> / traceroute
Name resolutionPing by IP works, by name fails (NXDOMAIN/SERVFAIL/timeout)dig <name> / cat /etc/resolv.conf
Service / AppConnectivity + DNS fine, but the app won't connectss -tlnp / curl
Ground truthLower layers look fine but something still drops silentlytcpdump -ni <if> host <ip>