Signal9Signal9 — home
online
all lessons

Network Namespaces: veth Pairs, ip netns, and What Each Namespace Owns

Intermediate35 min7 steps
What you'll learn
  • Explain what a network namespace isolates (interfaces, addresses, routes, firewall, port space) and what it shares
  • Create named namespaces with ip netns add and explain why /run/netns/NAME is the name
  • Join two namespaces with a veth pair and read the NAME@PEER and @ifN notation
  • Run commands inside a namespace with ip -n, ip netns exec, nsenter and ss -N, and prove a service is contained
  • Give a namespace its own resolv.conf with /etc/netns/NAME/ and know which tools honour it
  • Tear namespaces down without leaving an invisible one behind: ip netns pids before del, lsns -t net after
  1. What a network namespace isolates

    Name what a network namespace gets its own copy of, and what it still shares with the host.

    A network namespace is a separate copy of the kernel's networking stack. A process inside one sees its own world:

    • its own interfaces. A brand-new namespace has exactly one, a loopback lo, and even that starts DOWN;
    • its own addresses and routing tables;
    • its own firewall: nftables and iptables rulesets are per namespace;
    • its own `/proc/net` and, through ip netns exec, its own `/sys/class/net`;
    • its own port space. Two processes in two namespaces can both listen on TCP 8080 without either getting Address already in use.

    Everything else is shared. It is the same kernel, the same filesystem, the same users and, unless other namespace types are also in play, the same process list. That split is the whole trick behind containers. Docker, Podman and every Kubernetes pod put their processes in a fresh network namespace and wire it to the host with the same virtual cable you will build in this lesson.

    Every process belongs to exactly one network namespace, and a child inherits its parent's. That one rule explains most namespace surprises: a service is in whatever namespace the shell that started it was in. You can see a process's namespace as a handle under /proc/PID/ns/.

    readlink /proc/$$/ns/net
    net:[4026531840]
    $$ is your shell's PID, and /proc/PID/ns/net is a handle on the network namespace that process is in. The number in brackets is the namespace's inode number, its identity. Two processes that print the same number share one network stack; a different number means a different world. Run the same command on PID 1 (readlink /proc/1/ns/net) and you'll usually see the same value, because your shell lives in the host's initial namespace.

    Check yourself

    Two processes in different network namespaces both bind TCP port 8080. Does the second one fail with 'Address already in use'?

  2. A named namespace is just a file

    Create named namespaces with ip netns add and explain what the name actually is.

    A namespace lives for as long as something refers to it: a process inside it, an open file descriptor on it, or a bind mount of its handle onto a file. ip netns add NAME creates a namespace and bind-mounts its handle onto /run/netns/NAME. That file is the name. ip netns list simply lists that directory, and every other ip netns subcommand finds the namespace through it.

    This matters because the other way to make a namespace, unshare --net, creates an anonymous one. It exists only while the process inside it runs, and ip netns list never shows it. You can name one after the fact by bind-mounting its handle yourself (touch /run/netns/x && unshare --net=/run/netns/x true), which is exactly what ip netns add does for you.

    ip netns add demo-a && ip netns add demo-b
    Two new, empty namespaces, each pinned at /run/netns/<name>. Nothing prints on success, which is normal for ip write commands.
    ip netns list
    demo-b
    demo-a
    netns list
    show named network namespaces (the entries under /run/netns)
    Lists the files in /run/netns, newest first. Once a namespace is linked to another one (you'll do that next), the kernel assigns it a numeric id and the list shows it as demo-a (id: 0).
    findmnt -t nsfs
    TARGET            SOURCE                 FSTYPE OPTIONS
    /run/netns/demo-a nsfs[net:[4026532301]] nsfs   rw
    /run/netns/demo-b nsfs[net:[4026532362]] nsfs   rw
    -t nsfs
    only show mounts of the namespace filesystem (namespace handles)
    Here are the names for what they really are: mounts. Each /run/netns/<name> has a namespace handle (nsfs, with the same net:[inode] identity readlink showed you) bind-mounted onto it, and that mount is what keeps the namespace alive. You never create or delete these by hand; ip netns add and ip netns del mount and unmount them.
    ip netns exec demo-a ip -br link
    lo               DOWN           00:00:00:00:00:00 <LOOPBACK> 
    netns exec demo-a
    run the following command inside the named namespace
    -br
    brief, one line per interface
    Runs ip -br link inside demo-a. The only interface is lo, and it is DOWN. There is no eth0, because the host's NIC belongs to the host's namespace. A new namespace is an island until you give it a link.

    Check yourself

    What file does ip netns add blue create, and what keeps the namespace alive when no process is running inside it?

  3. veth: a virtual cable with two ends

    Wire two namespaces together with a veth pair and read the NAME@PEER notation.

    To connect islands you need a cable. A veth (virtual Ethernet) device always comes as a pair: a frame sent into one end comes out of the other, like the two ends of a patch cable. Create the pair, move one end into each namespace, then give each end an address and bring it up. Every step is an ip subcommand.

    The pair is created in whatever namespace you're in (the host), and ip link set END netns NS moves an end across. An interface can only be in one namespace at a time, so once moved it disappears from the host's ip link.

    ip link add va type veth peer name vb
    type veth
    a virtual Ethernet pair
    peer name vb
    the name of the other end
    Creates both ends at once, va and vb, in the host namespace. They start DOWN and without addresses.
    ip -br link show type veth
    vb@va            DOWN           86:1e:0c:4a:7f:21 <BROADCAST,MULTICAST,M-DOWN> 
    va@vb            DOWN           1a:9b:22:c4:05:e0 <BROADCAST,MULTICAST,M-DOWN> 
    show type veth
    only list veth devices
    Read the first column as NAME@PEER: va@vb means "this is va, and its other end is vb". That's the quickest way to see which veth is plugged into which. You don't need ethtool -S for it, and ethtool isn't on the lab image anyway. M-DOWN means the peer is down.
    ip link set va netns demo-a && ip link set vb netns demo-b
    netns demo-a
    move the device into that namespace
    Moves one end into each namespace. Both vanish from the host's ip link: they're now inside demo-a and demo-b.
    ip -n demo-a addr add 10.200.0.1/24 dev va && ip -n demo-a link set va up && ip -n demo-a link set lo up
    -n demo-a
    operate inside the demo-a namespace
    ip -n NS runs the ip command inside NS, a shorthand for ip netns exec NS ip. The /24 matters: it installs the connected route to 10.200.0.0/24, so demo-a knows the whole subnet is out of va. Bringing lo up is good hygiene, because a lot of software expects 127.0.0.1 to work.
    ip -n demo-b addr add 10.200.0.2/24 dev vb && ip -n demo-b link set vb up && ip -n demo-b link set lo up
    Configures the other end the same way, with a different host address in the same /24. Both ends have to be up before the link carries anything.
    ip -n demo-b -br addr
    lo               UNKNOWN        127.0.0.1/8 ::1/128 
    vb@if5           UP             10.200.0.2/24 fe80::841e:cff:fe4a:7f21/64 
    Now the peer shows as @ifN instead of a name. The peer lives in another namespace, so this one can only report its interface index over there. That number is vb's peer va's ifindex inside demo-a. UP plus an address in the right /24 is what a working end looks like.

    Check yourself

    Inside demo-b you see vb@if5. What does the if5 part tell you?

  4. Crossing the boundary: ip netns exec, nsenter, ss -N

    Run commands inside a namespace and show a service is visible only in its own namespace.

    There are four ways to act inside a namespace, and they suit different jobs:

    • ip -n NS … runs ip itself inside NS. It's quick for links, addresses and routes.
    • ip netns exec NS CMD runs any command inside NS. It also gives the command a matching /sys and applies /etc/netns/NS/ (next step).
    • nsenter --net=/run/netns/NS CMD is the util-linux equivalent. It also works with an anonymous namespace via a PID: nsenter --net --target PID.
    • ss -N NS … lists sockets inside NS without switching into it.

    A process started with ip netns exec stays in that namespace, and so do all its children. That's how you run a service "inside" one.

    ip netns exec demo-a ping -c1 -W2 10.200.0.2
    PING 10.200.0.2 (10.200.0.2) 56(84) bytes of data.
    64 bytes from 10.200.0.2: icmp_seq=1 ttl=64 time=0.051 ms
    
    --- 10.200.0.2 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 0.051/0.051/0.051/0.000 ms
    -c1
    send one echo request
    -W2
    wait at most 2 seconds for the reply
    demo-a reaches demo-b across the veth. There's no router and no host involvement: the frame goes in va and out vb.
    ip netns exec demo-b python3 -m http.server 8080 >/dev/null 2>&1 &
    Starts a small web server in demo-b and backgrounds it. Its socket belongs to demo-b's port space. python3 -m http.server never reads the terminal, so it backgrounds cleanly. (nc -lk 8080 & would stop on a background read unless you give it </dev/null.)
    ss -ltn 'sport = :8080'
    State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
    -l
    listening sockets
    -t
    TCP
    -n
    numeric ports
    'sport = :8080'
    filter on source (local) port 8080
    From the host: nothing, just the header. The host's port space has no listener on 8080, because the socket is in demo-b.
    ss -N demo-b -ltn 'sport = :8080'
    State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
    LISTEN 0      5            0.0.0.0:8080      0.0.0.0:*
    -N demo-b
    switch to that network namespace before listing
    The same query inside demo-b shows the listener. That pair of results, absent on the host and present in the namespace, is how you prove a service is contained.
    nsenter --net=/run/netns/demo-a curl -s -o /dev/null -w '%{http_code}\n' http://10.200.0.2:8080/
    200
    --net=/run/netns/demo-a
    enter the network namespace pinned at that file
    The client side, through nsenter this time: from demo-a, the service in demo-b answers 200. The host couldn't make this request, because it has no route to 10.200.0.0/24. Both ends of the only cable onto that subnet are inside namespaces.
    ip netns exec demo-a nft list tables
    Prints nothing. The lab box's host has its own firewall tables (nft list tables there shows them), but a namespace starts with an empty ruleset. Firewalls are per namespace, like everything else on the network side.

    Check yourself

    A service was started from a normal root shell on port 9000. Will ss -N demo-b -ltn show it?

  5. Per-namespace DNS: /etc/netns/NAME/

    Give one namespace its own resolv.conf with /etc/netns and know which entry points honour it.

    A namespace has its own network but shares the filesystem, so a command inside it reads the host's /etc/resolv.conf. That's often wrong. The host may point at 127.0.0.53 (systemd-resolved's stub), and inside a namespace 127.0.0.53 is the namespace's own loopback, where nothing is listening. DNS fails even though routing is fine.

    ip netns exec has a built-in fix. For every file in /etc/netns/NAME/, it bind-mounts that file over its /etc counterpart, visible only to the command it runs. Drop a resolv.conf (or a hosts file) there and that namespace gets its own.

    mkdir -p /etc/netns/demo-a && echo 'nameserver 10.200.0.2' > /etc/netns/demo-a/resolv.conf
    A resolv.conf that exists only for demo-a. It points at an address demo-a can actually reach.
    ip netns exec demo-a cat /etc/resolv.conf
    nameserver 10.200.0.2
    Through ip netns exec, /etc/resolv.conf is the per-namespace file. The host's own /etc/resolv.conf is untouched and every other process still sees it.
    nsenter --net=/run/netns/demo-a cat /etc/resolv.conf
    nameserver 127.0.0.53
    The same namespace entered with nsenter shows the host's resolv.conf. The /etc/netns overlay is an ip netns exec feature, not a kernel one, so nsenter and ip -n don't apply it. What prints is whatever this box's host file says.

    Check yourself

    You set up /etc/netns/red/resolv.conf, but nsenter --net=/run/netns/red dig example.com still uses the host's resolver. Why?

  6. The cleanup trap: ip netns pids before del

    Tear namespaces down cleanly without leaving an invisible namespace behind.

    ip netns del NAME does less than it sounds like. It removes the name: it unmounts /run/netns/NAME and deletes the file. The namespace itself is destroyed only when its last reference goes. If a process is still running inside, the namespace lives on with its interfaces, addresses and listening sockets, but no longer appears in ip netns list. It's a network stack you can't see, holding ports you can't find from the host.

    So check before you delete: ip netns pids NAME lists the processes inside. Stop them, then delete. When a namespace really is destroyed, the veth ends inside it go too, and deleting either end of a veth pair deletes both.

    ip netns pids demo-b
    4211
    One PID: the web server from step 4. As long as it runs, demo-b can't really be destroyed.
    ps -o pid,cmd -p "$(ip netns pids demo-b)"
        PID CMD
       4211 python3 -m http.server 8080
    -o pid,cmd
    show only the PID and full command
    -p
    select by PID
    Confirms what that PID is before you kill it. Always look before you stop something you didn't start.
    ip netns pids demo-b | xargs -r kill
    -r
    don't run the command when there is no input
    Stops every process in demo-b. -r makes xargs do nothing when the list is empty, instead of running a bare kill.
    ip netns del demo-a && ip netns del demo-b && rm -rf /etc/netns/demo-a && ip netns list
    Now the deletes really destroy both namespaces, and with them va and vb. The final ip netns list prints nothing.
    lsns -t net
            NS TYPE NPROCS PID USER NETNSID NSFS COMMAND
    4026531840 net      12   1 root unassigned      /sbin/init
    -t net
    only network namespaces
    The safety net. It lists every network namespace that still has a process in it, named or not, with the process holding it. After a clean teardown, only the host's initial namespace is left. An unexpected extra row is exactly the invisible namespace this step warns about.

    Check yourself

    After ip netns del blue, a service that was running inside blue is still reachable from red. How is that possible?

  7. Try it in your live terminal

    Build a two-namespace link end to end, verify it, and tear it down cleanly.

    Now build one from memory in your lab terminal, using different names and a different subnet so nothing collides with the demo. Use the one-command veth form this time. After each command, predict what ip -n NS -br addr will show, then check. When you're done, the lab net-namespace-veth gives you your own names, subnet and service port, and grades the live result.

    ip netns add lab-a && ip netns add lab-b
    Two named namespaces. Check them with ip netns list.
    ip link add la netns lab-a type veth peer name lb netns lab-b
    Each end is created directly in its namespace. The host never sees them.
    ip -n lab-a addr add 192.168.77.1/24 dev la && ip -n lab-a link set la up
    Address and up for the lab-a end.
    ip -n lab-b addr add 192.168.77.2/24 dev lb && ip -n lab-b link set lb up
    Address and up for the lab-b end.
    ip netns exec lab-a ping -c2 192.168.77.2
    Two replies means the cable works. Network is unreachable means a missing /24 on la. Silence means an end is still DOWN.
    lsns -t net
    Your two namespaces won't appear here, because no process is running in them. They're held only by their /run/netns mounts, which ip netns list shows.
    ip netns del lab-a && ip netns del lab-b && ip netns list
    Nothing was running inside, so the deletes destroy both namespaces and the veth pair with them. The list comes back empty.
Recap
  • A network namespace is its own network stack: interfaces, addresses, routes, firewall and port space. It still shares the kernel and the filesystem.
  • ip netns add NAME pins a namespace at /run/netns/NAME. That file is the name. A bare unshare --net namespace is anonymous and short-lived.
  • A veth pair is one cable: create it, put one end in each namespace, then address and bring up both ends. NAME@PEER means the peer is local, @ifN means it's in another namespace.
  • A process lives in the namespace of whatever started it. Start services with ip netns exec NS …, and prove containment with ss -ltn versus ss -N NS -ltn.
  • /etc/netns/NAME/resolv.conf gives one namespace its own DNS, but only through ip netns exec.
  • ip netns del removes only the name. Check ip netns pids first, and use lsns -t net to find a namespace a process is still holding.