Octet

Linux networking

0 of 36 nailed
Practice this trackRun a scenario

Core14

  1. Why did the ip command replace ifconfig and route, and what are the ip equivalents of the old commands?

    Say this first

    ifconfig, route, arp and netstat come from net-tools, which talks to the kernel through old ioctl calls and /proc files and has barely changed in years. iproute2 (ip, ss, bridge, tc) uses netlink, the kernel's native networking interface, so it sees everything the kernel can do: several addresses per interface, multiple routing tables and policy rules, namespaces, VRFs and tunnels. Many minimal distros no longer install net-tools at all.

    Go deeper

    The mapping I use

    Old (net-tools)New (iproute2)
    ifconfigip addr and ip link
    ifconfig eth0 upip link set eth0 up
    route -nip route
    arp -nip neigh
    netstat -tulpnss -tulpn
    netstat -iip -s link
    brctl showbridge link or ip link show master br0
    vconfig add eth0 10ip link add link eth0 name eth0.10 type vlan id 10

    What ifconfig gets wrong

    • It shows only the primary IPv4 address of an interface. A second address added with ip addr add is invisible to ifconfig unless it was given a label like eth0:1. People have chased "missing" addresses for hours because of this.
    • route -n shows only the main table. If policy routing is in use, it hides the tables that matter.
    • It knows nothing about namespaces, VRFs or most modern tunnel types.

    Habits worth showing

    Shell
    ip -br addr        # one line per interface: name, state, addresses
ip -br link        # one line per interface: name, state, MAC, flags
ip -c route        # colour output
ip -s link show ens192   # packet, error and drop counters

    Traps

    • Saying ifconfig shows all addresses. It hides secondary addresses that have no label.
    • Forgetting that ip changes are runtime only and vanish on reboot or on a network service restart.
    How did you do?Own page
  2. Walk me through the output of ip addr show. What do UP, LOWER_UP and state mean?

    Say this first

    The flags in angle brackets are what the interface is configured and able to do: UP means administratively up, LOWER_UP means the driver sees carrier (layer 1 is up). state is the operational state the kernel computed: UP, DOWN, or UNKNOWN for things like loopback that do not report carrier. Then come the MTU, the queueing discipline, the MAC, and each address with its prefix, scope and lifetime.

    Go deeper

    A real example

    Output
    $ ip addr show dev ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a1:2b:3c brd ff:ff:ff:ff:ff:ff
    inet 10.0.10.21/24 brd 10.0.10.255 scope global dynamic ens192
       valid_lft 85912sec preferred_lft 85912sec
    inet6 fe80::250:56ff:fea1:2b3c/64 scope link
       valid_lft forever preferred_lft forever
    • 2: is the interface index. Other commands refer to it (for example veth0@if7).
    • UP admin up, LOWER_UP carrier present. A cable pull shows NO-CARRIER and state DOWN while UP stays, because the admin state did not change.
    • mtu 1500, qdisc mq (multiqueue NIC), qlen 1000 is the transmit queue length.
    • inet 10.0.10.21/24 brd 10.0.10.255 scope global dynamic: address with prefix, broadcast, scope global (routable), and dynamic means it came from DHCP; valid_lft counts down the lease.
    • inet6 fe80::.../64 scope link is the IPv6 link-local address, built here from the MAC with EUI-64.

    Scopes

    global is usable anywhere, link only on this link (fe80::, 169.254/16), host only inside this machine (127.0.0.1 on lo).

    The short form

    Output
    $ ip -br addr
lo               UNKNOWN        127.0.0.1/8 ::1/128
ens192           UP             10.0.10.21/24 fe80::250:56ff:fea1:2b3c/64

    Traps

    • Reading LOWER_UP as "the interface is enabled". It only means carrier is present.
    • Ignoring dynamic and valid_lft. A DHCP address with a short lifetime left can explain intermittent loss.
    How did you do?Own page
  3. How do you read the Linux routing table, and how does Linux choose between routes?

    Say this first

    ip route shows the main table. Linux picks the longest prefix match first. Between routes with the same prefix it picks the lowest metric. There is no administrative distance as on Cisco: whoever installs the route sets the metric. Each line shows the prefix, the next hop (via), the interface (dev), who installed it (proto), the scope and the preferred source address (src).

    Go deeper

    A real table

    Output
    $ ip route
default via 10.0.10.1 dev ens192 proto static
10.0.10.0/24 dev ens192 proto kernel scope link src 10.0.10.21
172.16.40.0/22 via 10.0.10.254 dev ens192 proto static
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
    • default via 10.0.10.1: the default route, written as default instead of 0.0.0.0/0.
    • proto kernel scope link: the connected route the kernel added when the address was configured. scope link means the destination is directly on the wire, no gateway.
    • proto static: installed by the network manager from config. proto dhcp comes from a lease. Routes added by hand with ip route add get proto boot, which ip route does not print.
    • src 10.0.10.21: the source address the kernel prefers for traffic using this route.
    • linkdown: the interface has no carrier, here libvirt's bridge with no VMs running.

    Changing it

    Shell
    ip route add 172.16.40.0/22 via 10.0.10.254 dev ens192
ip route replace 172.16.40.0/22 via 10.0.10.253   # add or overwrite in one step
ip route del 172.16.40.0/22
ip route add default via 10.0.10.1 metric 100

    Metrics in practice

    NetworkManager gives Ethernet default routes metric 100 and Wi-Fi 600, which is why a laptop prefers the cable when both are connected. The lower metric wins.

    Other tables

    ip route only shows table main. The local table holds the host's own addresses and broadcast addresses. ip route show table all shows everything.

    Traps

    • Talking about administrative distance in the Linux kernel. Longest match, then metric. Routing daemons such as FRR apply their own distances before they install a route.
    • Reading only ip route when policy routing rules send traffic to another table.
    How did you do?Book 3.3 Internet (IP)Own page
  4. How do you see which ports are listening on a Linux server and which process owns them?

    Say this first

    ss -tulpn: TCP and UDP, listening sockets, with the process, numeric (no name lookups). Run it as root, or -p cannot show processes owned by other users. ss replaced netstat -tulpn because it reads sockets through netlink instead of parsing /proc/net/tcp, so it is much faster on busy hosts. The local address column tells you whether the service listens on all addresses or only on loopback.

    Go deeper

    A real example

    Output
    $ sudo ss -tlnp
State   Recv-Q  Send-Q    Local Address:Port    Peer Address:Port  Process
LISTEN  0       4096      127.0.0.53%lo:53           0.0.0.0:*      users:(("systemd-resolve",pid=612,fd=14))
LISTEN  0       128             0.0.0.0:22           0.0.0.0:*      users:(("sshd",pid=901,fd=3))
LISTEN  0       4096          127.0.0.1:8080         0.0.0.0:*      users:(("java",pid=1532,fd=41))
LISTEN  0       511                   *:443                *:*      users:(("apache2",pid=1204,fd=6))
LISTEN  0       128                [::]:22              [::]:*      users:(("sshd",pid=901,fd=4))
    • 127.0.0.53%lo:53 is the systemd-resolved stub, bound to loopback only.
    • 0.0.0.0:22 accepts on every IPv4 address. [::]:22 is the IPv6 socket.
    • 127.0.0.1:8080 is reachable only from the host itself. If clients cannot connect, this is why.
    • *:443 is one socket that accepts IPv4 and IPv6.
    • For a LISTEN socket, Recv-Q is the number of connections waiting for the application to accept them and Send-Q is the backlog size.

    The letters

    -t TCP, -u UDP, -l listening only, -a all states, -p process, -n numeric, -4 or -6 to pick a family, -x for Unix sockets, -s for a summary.

    Traps

    • Running ss -p without root and concluding no process owns the port.
    • Treating 0.0.0.0 as a destination. In a listening socket it means "every local IPv4 address".
    How did you do?Book 1.4 SoftwareOwn page
  5. Walk me through dig output. Where do you look to know whether a lookup really worked?

    Say this first

    I read the header first: status (NOERROR, NXDOMAIN, SERVFAIL or REFUSED), the flags, and the answer count. NOERROR with ANSWER 0 means the name exists but has no record of that type. Then the answer section for the records and their TTLs, and the footer for which server answered and how long it took. aa in the flags means an authoritative server answered; without it the answer came from a resolver's cache.

    Go deeper

    Example through the systemd-resolved stub

    Output
    $ dig www.example.com A

; <<>> DiG 9.18.28 <<>> www.example.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41872
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.example.com.               IN      A

;; ANSWER SECTION:
www.example.com.        300     IN      CNAME   web.example.net.
web.example.net.        60      IN      A       203.0.113.80

;; Query time: 12 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Nov 10 09:14:02 UTC 2026
;; MSG SIZE  rcvd: 89
    • status: NOERROR and ANSWER: 2: success. The name is a CNAME to web.example.net, which has the A record.
    • Flags: qr this is a response, rd we asked for recursion, ra the server offers it. No aa, so cached.
    • TTL 300 and 60 seconds: how long a resolver may cache each record. Repeat the query and they count down.
    • SERVER: 127.0.0.53: the local systemd-resolved stub answered, not the real DNS server.

    Status codes

    StatusMeaning
    NOERROR, ANSWER > 0Records found
    NOERROR, ANSWER 0Name exists, no record of that type (NODATA)
    NXDOMAINThe name does not exist
    SERVFAILThe resolver failed, often DNSSEC validation or unreachable authoritative servers
    REFUSEDThe server will not answer you, typically recursion not allowed for your source

    Options I use

    dig @10.0.20.53 name (ask a specific server), +short, +trace (walk from the root), -x 203.0.113.80 (PTR), dig example.com MX, +norecurse (see what a server has itself).

    Traps

    • Treating NOERROR as success without checking the answer count.
    • Forgetting that dig talks to the server in resolv.conf, which on many hosts is the local stub.
    How did you do?Book 9.3 Infrastructure ApplicationsOwn page
  6. How do you use curl to debug a web or API problem from a Linux box?

    Say this first

    curl -v shows each stage: DNS result, TCP connect, TLS handshake, the request headers sent and the response headers received, so I can see where it breaks. -w with timing variables splits the delay into DNS, connect, TLS and server time. --resolve sends the request to a specific backend IP while keeping the real hostname for SNI and the Host header. The exit code names the failing stage: 6 DNS, 7 connect, 28 timeout, 35 TLS handshake, 60 certificate.

    Go deeper

    Verbose run (trimmed)

    Output
    $ curl -sv -o /dev/null https://app.example.com/health
*   Trying 203.0.113.80:443...
* Connected to app.example.com (203.0.113.80) port 443
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=app.example.com
> GET /health HTTP/2
> Host: app.example.com
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 200
< content-type: application/json

    Lines with * are curl's own notes, > is what it sent, < is what came back.

    Where the time goes

    Output
    $ curl -s -o /dev/null -w 'dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total}\n' https://app.example.com/health
dns 0.004213 connect 0.031870 tls 0.089402 ttfb 0.912337 total 0.915006

    The values are cumulative from the start. Here DNS, TCP and TLS are fast and the server takes about 0.8 s to produce the first byte: an application problem, not a network one.

    Other options I reach for

    • --resolve app.example.com:443:10.0.20.15: test one backend behind a load balancer with correct SNI.
    • -I headers only, -L follow redirects, -k skip certificate checks (testing only).
    • --connect-timeout 3 and -m 10 so a script does not hang.
    • -4 or -6 to force a family, -x http://proxy:3128 to test through a proxy.
    • curl -v telnet://10.0.20.15:5432 as a quick TCP port test when telnet and nc are not installed.

    Traps

    • Leaving -k in production scripts and hiding real certificate problems.
    • Reading time_total alone. The breakdown is what tells you who to call.
    How did you do?Book 9.1 Traditional ApplicationsOwn page
  7. How do you turn a Linux box into a router, and why is enabling ip_forward often not enough?

    Say this first

    By default Linux behaves as a host and drops packets that are not addressed to it. net.ipv4.ip_forward=1 lets it route IPv4 between interfaces; IPv6 has its own switch, net.ipv6.conf.all.forwarding=1. sysctl -w changes the running value; a file in /etc/sysctl.d/ makes it survive a reboot. After that the packets still have to pass the firewall FORWARD chain and rp_filter, and the far side needs a route back.

    Go deeper

    Check and set

    Output
    $ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
$ sudo sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
    Shell
    echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/90-router.conf
sudo sysctl --system      # reload every sysctl file
cat /proc/sys/net/ipv4/ip_forward

    The checklist after forwarding is on

    1. Firewall FORWARD chain. Docker sets the filter FORWARD policy to DROP when it starts; hardened images often do the same. iptables -S FORWARD or nft list ruleset.
    2. rp_filter. With strict mode, packets that arrive on an interface the router would not use to reach their source are dropped silently.
    3. Return route. The next router or the server must know the way back to the source subnet, or you need NAT.
    4. NAT if the inside uses private addresses and the outside does not route them.

    Who turns it on for you

    Docker, libvirt and Kubernetes components set ip_forward=1 when they start, which is why a host can suddenly start routing after someone installs a container runtime.

    Traps

    • Using sysctl -w and forgetting to persist it. The router stops routing after the next reboot.
    • Assuming forwarding is the whole job. The FORWARD chain policy is the next most common reason.
    How did you do?Book 3.3 Internet (IP)Own page
  8. What are the iptables tables and chains, and what is each one for?

    Say this first

    A table groups rules by job, a chain is the point in the packet's path where they run. filter (the default) accepts or drops in INPUT, FORWARD and OUTPUT. nat rewrites addresses in PREROUTING (DNAT), POSTROUTING (SNAT, MASQUERADE), OUTPUT and INPUT, and only sees the first packet of each connection. mangle changes packets or sets marks in all five chains. raw runs before connection tracking, mostly to exempt traffic from it. security is for SELinux labels.

    Go deeper

    The map

    TableJobBuilt-in chains
    filterAllow or blockINPUT, FORWARD, OUTPUT
    natAddress and port translationPREROUTING, INPUT, OUTPUT, POSTROUTING
    mangleMarks, TTL, DSCP, TCPMSSPREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING
    rawBefore conntrack, CT --notrackPREROUTING, OUTPUT
    securitySELinux security marksINPUT, FORWARD, OUTPUT

    The five hooks in one line each

    • PREROUTING: every packet arriving, before the routing decision.
    • INPUT: packets for this host.
    • FORWARD: packets routed through this host.
    • OUTPUT: packets created by this host.
    • POSTROUTING: every packet leaving, after routing.

    Working with it

    Shell
    iptables -S                  # filter table as commands
iptables -t nat -S           # nat table
iptables -N SSH-IN           # user-defined chain
iptables -A INPUT -p tcp --dport 22 -j SSH-IN

    Without -t, iptables works on filter. Terminating targets are ACCEPT, DROP and REJECT; LOG and MARK continue to the next rule; RETURN goes back to the calling chain.

    Traps

    • Saying "the iptables INPUT chain" as if there were only one. Several tables have an INPUT chain.
    • Forgetting that iptables with no -t is the filter table, so iptables -L hides all NAT rules.
    How did you do?Book 8.5 Example SystemsOwn page
  9. How does iptables evaluate rules, and how do you add, remove and save them safely?

    Say this first

    Rules in a chain are checked top to bottom and the first rule with a terminating target (ACCEPT, DROP, REJECT) wins. If nothing matches, the chain's policy applies; only built-in chains have a policy, ACCEPT or DROP. -A appends to the bottom, -I inserts at the top or at a position, -D deletes by spec or line number. Rules live in kernel memory, so they need iptables-save and a restore at boot to persist.

    Go deeper

    Reading a chain

    Output
    $ sudo iptables -S INPUT
-P INPUT DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

    iptables -L INPUT -n -v --line-numbers shows the same rules with packet counters and positions.

    Changing it

    Shell
    iptables -I INPUT 3 -p tcp -s 10.0.50.0/24 --dport 9100 -j ACCEPT   # insert as rule 3
iptables -D INPUT 3                                                 # delete rule 3
iptables -Z INPUT                                                   # zero the counters

    Counters are the best debugging tool: zero them, reproduce, and see which rule's packet count moved.

    Persisting

    • Debian and Ubuntu: iptables-persistent restores /etc/iptables/rules.v4 at boot; netfilter-persistent save writes it.
    • RHEL family with iptables-services: /etc/sysconfig/iptables.
    • By hand: iptables-save > file and iptables-restore < file. Restore replaces every table in the file.

    Traps

    • Setting the INPUT policy to DROP before adding the SSH allow rule over SSH.
    • Appending an allow rule with -A below an existing catch-all drop. It never matches.
    How did you do?Own page
  10. What is a Linux bridge, and how do you build one?

    Say this first

    A Linux bridge is a layer 2 switch inside the kernel. It learns source MACs per port, forwards known destinations, and floods broadcasts and unknown unicast. Its ports can be physical NICs, VLAN interfaces, veth ends or VM tap devices. The bridge device itself (br0) is also the host's own port on that switch, so the host's IP address goes on br0, never on the enslaved NIC. STP is off by default. KVM, Docker and LXC all build on bridges.

    Go deeper

    Build one

    Shell
    ip link add br0 type bridge
ip link set ens224 master br0
ip link set ens224 up
ip addr add 10.30.0.5/24 dev br0
ip link set br0 up

    Look at it

    Output
    $ ip -br link show master br0
ens224           UP             00:50:56:a1:6c:10 <BROADCAST,MULTICAST,UP,LOWER_UP>
vnet0            UNKNOWN        fe:54:00:6b:1e:01 <BROADCAST,MULTICAST,UP,LOWER_UP>
$ bridge fdb show br br0 | grep -v permanent
52:54:00:6b:1e:01 dev vnet0 master br0
00:1c:7f:3a:11:02 dev ens224 master br0

    The fdb is the MAC address table: the VM's MAC learned on its tap port, the upstream router learned on the NIC.

    Defaults worth knowing

    • MAC ageing 300 seconds.
    • STP disabled (stp_state 0), so ports forward immediately and a loop is a storm.
    • bridge-utils and brctl are the old tools; ip link and bridge replace them.

    Traps

    • Leaving the IP on the physical NIC after enslaving it.
    • Bridging two uplinks to the same switch with STP off and creating a loop.
    How did you do?Book 3.2 Switched EthernetBook 3.5 ImplementationOwn page
  11. How do you put a Linux server on a VLAN trunk?

    Say this first

    Create a VLAN subinterface on top of the physical NIC, one per VLAN: ip link add link ens192 name ens192.10 type vlan id 10, then put the address on ens192.10. The 8021q module tags outgoing frames and sorts incoming tagged frames to the right subinterface. The switch port must be a trunk carrying those VLANs. Untagged frames still arrive on the parent ens192.

    Go deeper

    Runtime

    Shell
    ip link add link ens192 name ens192.10 type vlan id 10
ip addr add 10.10.0.21/24 dev ens192.10
ip link set ens192 up
ip link set ens192.10 up
    Output
    $ ip -d link show ens192.10
5: ens192.10@ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:a1:2b:3c brd ff:ff:ff:ff:ff:ff promiscuity 0 ...
    vlan protocol 802.1Q id 10 <REORDER_HDR> ...

    @ens192 names the parent. The VLAN interface uses the parent's MAC.

    Persistent, netplan

    YAML
    network:
  version: 2
  ethernets:
    ens192: {}
  vlans:
    ens192.10:
      id: 10
      link: ens192
      addresses: [10.10.0.21/24]

    Details

    • The VLAN interface MTU defaults to the parent's and cannot exceed it. The NIC carries the 4-byte tag.
    • QinQ: ip link add link ens192 name ens192.100 type vlan proto 802.1ad id 100, then a second 802.1Q VLAN on top of it.
    • The interface name is free text; ens192.10 is a convention.

    Traps

    • Configuring VLAN interfaces against an access port. The switch drops or untags the frames.
    How did you do?Book 3.2 Switched EthernetOwn page
  12. What is a veth pair, and how do you find which container a host-side veth belongs to?

    Say this first

    A veth pair is two virtual Ethernet interfaces joined like a patch cable: a frame sent into one comes out of the other. You create both ends together, put one in a namespace and leave the other in the root namespace, often attached to a bridge. That is how a container's eth0 connects to docker0 or a CNI bridge. To match ends, use the peer interface index shown after @if in ip link, or ethtool -S.

    Go deeper

    Create and place

    Shell
    ip netns add red
ip link add veth-red type veth peer name veth-red-br
ip link set veth-red netns red
ip link set veth-red-br master br0
ip link set veth-red-br up
ip -n red link set veth-red up

    Matching the ends

    Output
    $ ip -br link show type veth
veth-red-br@if5  UP             d2:5f:0a:91:3c:11 <BROADCAST,MULTICAST,UP,LOWER_UP>
$ ethtool -S veth-red-br
NIC statistics:
     peer_ifindex: 5

    Index 5 is the peer's interface index inside the other namespace. For a container, compare it with nsenter -t <pid> -n ip -br link or cat /sys/class/net/eth0/iflink inside the container.

    Rules

    • Interface names are limited to 15 characters.
    • Both ends must be up, and if one end is deleted the other disappears with it.
    • veth reports a nominal 10 Gbps but real throughput depends on CPU.

    Traps

    • Bringing up the host end and forgetting the namespace end (or the reverse).
    • Using a name longer than 15 characters; the command fails.
    How did you do?Own page
  13. How do you write tcpdump filters, and how do you read a tcpdump line?

    Say this first

    tcpdump takes a BPF filter in pcap-filter syntax: primitives like host, net, port, portrange, with src or dst qualifiers and protocol names (tcp, udp, icmp, arp), joined with and, or, not and parentheses. I always use -nn so it does not resolve names or ports, -i for the interface, -c to stop after a count, and -w to save a pcap for Wireshark. Quote the filter.

    Go deeper

    Filters I actually use

    Shell
    tcpdump -i ens192 -nn host 10.0.10.30
tcpdump -i ens192 -nn 'src net 10.20.0.0/24 and dst port 443'
tcpdump -i any -nn 'port 53'
tcpdump -i ens192 -nn -e arp
tcpdump -i ens192 -nn 'icmp or tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
tcpdump -i ens192 -nn 'tcp port 22 and not host 10.0.50.10'
tcpdump -i ens192 -nn -c 2000 -w /tmp/web.pcap 'host 198.51.100.80 and port 443'
tcpdump -nn -r /tmp/web.pcap 'tcp[tcpflags] & tcp-rst != 0'

    Reading a handshake

    Output
    09:21:14.502113 IP 10.0.10.21.51544 > 198.51.100.80.443: Flags [S], seq 3125110123, win 64240, options [mss 1460,sackOK,TS val 1840233 ecr 0,nop,wscale 7], length 0
09:21:14.505391 IP 198.51.100.80.443 > 10.0.10.21.51544: Flags [S.], seq 2287712361, ack 3125110124, win 65160, options [mss 1460,sackOK,TS val 99120 ecr 1840233,nop,wscale 7], length 0
09:21:14.505430 IP 10.0.10.21.51544 > 198.51.100.80.443: Flags [.], ack 1, win 502, length 0
    • Address and port are joined with a dot: 10.0.10.21.51544.
    • Flags: S SYN, S. SYN-ACK, . ACK, P. data with push, F. FIN, R reset.
    • After the handshake tcpdump prints relative sequence numbers (ack 1); -S shows absolute ones.
    • win 502 is scaled: with wscale 7 that is 502 x 128 bytes.
    • The 3.3 ms between SYN and SYN-ACK is the network round trip to the server.

    Options

    -e link-layer header (MACs, VLAN tag), -v more IP detail, -A payload as ASCII, -X hex, -Q in or -Q out for direction, -s snap length (the modern default captures whole packets).

    Traps

    • Forgetting quotes, so the shell eats the parentheses or &.
    • Writing port 53 and thinking it only matches UDP. It matches TCP too; add udp if you mean it.
    How did you do?Book 5.2 Reliable Byte Stream (TCP)Own page
  14. What are netplan, systemd-networkd and NetworkManager, and how do you tell which one manages a host?

    Say this first

    systemd-networkd and NetworkManager are daemons that configure interfaces, addresses, routes and DNS. Netplan, on Ubuntu, is only a YAML front end: it renders its files into config for networkd (the default on Ubuntu Server) or NetworkManager (Ubuntu Desktop) and does nothing at runtime. RHEL and Fedora use NetworkManager with nmcli. Anything done with ip is temporary; the daemon's config is what survives a reboot. networkctl list and nmcli device status show who manages each interface.

    Go deeper

    Netplan (Ubuntu)

    YAML
    # /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    ens192:
      addresses: [10.0.10.21/24]
      routes:
        - to: default
          via: 10.0.10.1
      nameservers:
        addresses: [10.0.20.53, 10.0.20.54]
        search: [corp.example.com]

    netplan try applies it and rolls back after 120 seconds unless you confirm, which is the safe way over SSH. netplan apply applies without a safety net. gateway4 is deprecated; use a route to: default.

    systemd-networkd

    Output
    $ networkctl list
IDX LINK   TYPE     OPERATIONAL SETUP
  1 lo     loopback carrier     unmanaged
  2 ens192 ether    routable    configured

2 links listed.

    Native files are /etc/systemd/network/*.network with [Match], [Network] and [Link] sections; networkctl status ens192 shows addresses, gateway, DNS and recent log lines.

    NetworkManager

    Shell
    nmcli device status
nmcli con mod ens192 ipv4.method manual ipv4.addresses 10.0.10.21/24 ipv4.gateway 10.0.10.1 ipv4.dns "10.0.20.53 10.0.20.54"
nmcli con up ens192

    Connections are stored as keyfiles in /etc/NetworkManager/system-connections/. RHEL 9 deprecated the old ifcfg files. The connection name and the device name can differ: nmcli con show lists both.

    Traps

    • Editing the files netplan generates under /run. They are rewritten on the next netplan apply or boot.
    • Running netplan apply on a remote box with a typo instead of netplan try.
    How did you do?Own page

Working16

  1. How do you find out exactly which route, interface and source address Linux will use for a destination?

    Say this first

    ip route get <destination> asks the kernel to do the real lookup, through the policy rules and every table, and prints the chosen next hop, output interface and source address. I add from, iif, oif or mark to simulate other traffic, for example a packet arriving on a given interface. It beats reading the table by eye whenever policy routing, ECMP or a cached path MTU is involved.

    Go deeper

    Examples

    Output
    $ ip route get 172.16.41.9
172.16.41.9 via 10.0.10.254 dev ens192 src 10.0.10.21 uid 1000
    cache

    The kernel matched 172.16.40.0/22 via 10.0.10.254 and will source from 10.0.10.21.

    Shell
    ip route get 198.51.100.80 from 203.0.113.10    # what does traffic sourced from this IP do
ip route get 198.51.100.80 mark 0x2              # what does marked traffic do
ip route get 10.0.99.21 from 10.0.50.10 iif ens224   # simulate a packet arriving on ens224

    Source address selection

    If the route has a src hint, Linux uses it. Otherwise it picks the primary address of the output interface whose scope fits the destination. Applications can override it by binding to an address.

    Two things it reveals that ip route does not

    • Reverse path filter drops. Simulating an inbound packet with iif returns RTNETLINK answers: Invalid cross-device link when rp_filter would drop it.
    • Path MTU. If the kernel learned a smaller MTU from an ICMP "fragmentation needed", the output shows cache expires 412sec mtu 1400.

    Traps

    • Trusting a hand reading of ip route on a host with ip rule entries. Let the kernel do the lookup.
    How did you do?Own page
  2. How do you filter ss output, and how do you check the health of a single TCP connection?

    Say this first

    ss has its own filter language: states (state established, state time-wait), addresses (dst, src) and ports (dport = :443), combined with and, or and parentheses. For one connection, ss -tni prints the kernel's TCP internals: RTT, RTO, congestion window, MSS, retransmissions and the congestion algorithm, which tells you whether a slow transfer is loss, latency or the application.

    Go deeper

    Filters

    Shell
    ss -tn state established '( dport = :443 or sport = :443 )'
ss -tn dst 10.0.20.0/24
ss -tan state time-wait | wc -l
ss -to state established dport = :5432    # -o adds timers (keepalive, retransmit)
ss -s                                     # totals per state and protocol

    Quote the expression so the shell does not eat the parentheses. Ports are written with a leading colon, :443, as in the man page examples.

    TCP internals for one flow (trimmed)

    Output
    $ ss -tni dst 198.51.100.80
State  Recv-Q  Send-Q   Local Address:Port    Peer Address:Port
ESTAB  0       0          10.0.10.21:51544   198.51.100.80:443
     cubic wscale:7,7 rto:204 rtt:3.1/1.2 mss:1448 pmtu:1500 cwnd:10 retrans:0/2 send 37.4Mbps
    • rtt:3.1/1.2: smoothed RTT and its variation in ms.
    • rto:204: retransmission timeout. Linux keeps a 200 ms minimum, so short RTT paths sit just above it.
    • cwnd:10 segments, mss:1448 (1500 minus IP, TCP and the 12-byte timestamp option).
    • retrans:0/2: retransmissions outstanding now / in total. A growing total means loss on the path.
    • send 37.4Mbps is cwnd times MSS divided by RTT, the rate the window currently allows.

    Traps

    • Forgetting the quotes around a filter with parentheses, so the shell eats them and ss reports a syntax error.
    • Blaming the network for CLOSE-WAIT build-up.
    How did you do?Book 5.2 Reliable Byte Stream (TCP)Own page
  3. What do the states in ip neigh mean, such as REACHABLE, STALE, DELAY and FAILED?

    Say this first

    ip neigh is the ARP table for IPv4 and the NDP cache for IPv6, with a state machine. REACHABLE: the mapping was confirmed recently. STALE: the confirmation is old, but the entry is still used. DELAY: the entry was used while stale and the kernel waits a few seconds for an upper layer, such as a TCP ACK, to confirm it. PROBE: it is sending unicast ARP requests. FAILED: nobody answered. INCOMPLETE: first resolution in progress. PERMANENT is a static entry.

    Go deeper

    Example

    Output
    $ ip neigh show dev ens192
10.0.10.1 lladdr 00:1c:7f:3a:11:02 REACHABLE
10.0.10.30 lladdr 00:50:56:a1:9c:44 STALE
10.0.10.31 lladdr 00:50:56:a1:47:e0 DELAY
10.0.10.44 FAILED
10.0.10.5 lladdr 00:50:56:a1:00:05 PERMANENT

    The lifecycle and default timers

    1. No entry: the kernel broadcasts a request, entry is INCOMPLETE (3 tries, mcast_solicit, 1 s apart).
    2. Reply received: REACHABLE for a random time between 15 and 45 seconds (base_reachable_time_ms 30000, randomised by half either way).
    3. Timer expires: STALE. Still used to send traffic, nothing is sent on the wire yet.
    4. A packet is sent using a stale entry: DELAY for 5 seconds (delay_first_probe_time). If TCP sees an ACK from that neighbour in that window, it confirms the entry and it returns to REACHABLE.
    5. No confirmation: PROBE, 3 unicast requests (ucast_solicit). No answer: FAILED.

    So STALE is normal and healthy. FAILED and INCOMPLETE are the states that mean trouble.

    Fixing entries

    Shell
    ip neigh flush dev ens192                    # clear dynamic entries on one interface
ip neigh del 10.0.10.30 dev ens192
ip neigh replace 10.0.10.5 lladdr 00:50:56:a1:00:05 dev ens192 nud permanent
ip -s neigh show 10.0.10.30                  # adds used/confirmed/updated timers

    Traps

    • Calling STALE an error. It is the normal resting state of an entry nobody has confirmed lately.
    • Using arp -n and missing IPv6 entries. ip neigh shows both families.
    How did you do?Book 3.3 Internet (IP)Own page
  4. What is policy routing on Linux, and how do ip rule and multiple routing tables work together?

    Say this first

    Linux can hold up to 2^32 routing tables, and ip rule decides which table a packet is looked up in. Rules are checked in priority order and match on source, destination, incoming or outgoing interface, firewall mark, user ID or TOS. The default rules are priority 0 lookup local, 32766 lookup main and 32767 lookup default. If the chosen table has no matching route, the kernel moves on to the next rule.

    Go deeper

    Default state

    Output
    $ ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

    Table IDs: local 255, main 254, default 253. Names live in /etc/iproute2/rt_tables (or a file in rt_tables.d/), but you can always use the number.

    Adding a table and a rule

    Shell
    ip route add default via 203.0.113.9 dev ens224 table 200
ip rule add from 203.0.113.10 lookup 200 priority 1000
ip rule add fwmark 0x2 lookup 200 priority 1100
ip rule add iif ens256 lookup 200 priority 1200
    Output
    $ ip rule
0:      from all lookup local
1000:   from 203.0.113.10 lookup 200
1100:   from all fwmark 0x2 lookup 200
1200:   from all iif ens256 lookup 200
32766:  from all lookup main
32767:  from all lookup default

    Selectors and actions

    • Selectors: from, to, iif, oif, fwmark, tos, uidrange, and not to invert.
    • Actions: lookup <table>, or blackhole, unreachable, prohibit to drop.
    • suppress_prefixlength 0: use the table but ignore a default route found in it. wg-quick uses this with not fwmark 51820 lookup 51820 so WireGuard can take the default route without looping its own encrypted packets into the tunnel.

    Where it is used

    Two uplinks with symmetric return traffic, sending one subnet or one application out a different ISP, VPN split tunnelling, and Kubernetes and cloud CNIs that give each secondary interface its own table.

    Traps

    • Thinking ip route shows everything. It only shows table main.
    • Putting a rule at a priority above 32766 and wondering why main always wins.
    How did you do?Book 3.3 Internet (IP)Own page
  5. How does name resolution work on a modern Linux host, and what is 127.0.0.53?

    Say this first

    An application calls getaddrinfo in glibc, which follows the hosts: line in /etc/nsswitch.conf: usually files (/etc/hosts) first, then DNS. DNS goes to the servers in /etc/resolv.conf. On hosts with systemd-resolved, resolv.conf points at 127.0.0.53, a local caching stub, and the real upstream servers are kept per interface; resolvectl status shows them. dig and nslookup read resolv.conf directly and skip /etc/hosts; getent hosts follows the same path as applications.

    Go deeper

    The pieces

    Output
    $ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Mar  3 10:12 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
$ grep -v '^#' /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search corp.example.com
    Output
    $ resolvectl status ens192
Link 2 (ens192)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.0.20.53
       DNS Servers: 10.0.20.53 10.0.20.54
        DNS Domain: corp.example.com

    The upstream servers come from netplan, NetworkManager or DHCP. /run/systemd/resolve/resolv.conf lists them in classic format for tools that need it.

    resolv.conf rules worth knowing (glibc)

    • At most 3 nameserver lines are used, tried in order.
    • options timeout:5 attempts:2 are the defaults: a dead first server costs 5 seconds per try.
    • search domains are appended to names with fewer dots than ndots (default 1).

    Commands

    Shell
    getent hosts app.corp.example.com     # same path as applications, includes /etc/hosts
resolvectl query app.corp.example.com
resolvectl flush-caches
resolvectl statistics                 # cache hits and misses

    Traps

    • Editing /etc/resolv.conf by hand on a host where systemd-resolved or NetworkManager manages it. It gets overwritten or bypasses the stub.
    • Calling nameserver 127.0.0.53 a misconfiguration. It is the stub, and the real servers are in resolvectl.
    How did you do?Book 9.3 Infrastructure ApplicationsOwn page
  6. What is rp_filter, what do the values 0, 1 and 2 mean, and when does it break things?

    Say this first

    rp_filter is reverse path filtering, the anti-spoofing check from RFC 3704. For each incoming packet the kernel looks up the route back to the source. 1 strict: the best route back must point out the same interface the packet arrived on, otherwise drop. 2 loose: the source only needs to be reachable through some interface. 0 off. The effective value per interface is the higher of conf.all and conf.<iface>. Strict mode silently drops asymmetric traffic on multihomed hosts.

    Go deeper

    Check it

    Output
    $ sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.ens224.rp_filter
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.ens224.rp_filter = 1

    The kernel default is 0, but distributions ship their own value (1 or 2 are both common). Check, do not assume. conf.default only applies to interfaces created after it is set.

    Seeing the drops

    Shell
    sysctl -w net.ipv4.conf.all.log_martians=1
dmesg | grep martian
nstat -az TcpExtIPReversePathFilter
ip route get 10.0.99.21 from 10.0.50.10 iif ens224
    Output
    IPv4: martian source 10.0.99.21 from 10.0.50.10, on dev ens224
RTNETLINK answers: Invalid cross-device link

    In the martian log the first address is the destination and the second the source. "Invalid cross-device link" from ip route get is the same check failing.

    Typical breakage

    • A server with a management NIC: monitoring traffic arrives on the management interface, but the route back to the monitoring subnet points out the data interface.
    • Two uplinks with traffic coming in on the one that is not the default.
    • Routers in an ECMP or asymmetric design.

    Fixes, best first

    1. Make the routing symmetric (policy routing by source address).
    2. Loose mode (2) on that interface, which still drops sources with no route at all.
    3. 0 only when you understand why, for example inside some overlay or VPN designs.

    Traps

    • Setting the interface value to 0 while all is 1 and believing it is off.
    • Disabling rp_filter everywhere instead of fixing the asymmetric routing.
    How did you do?Own page
  7. Trace a packet through netfilter for traffic to the host, through the host and from the host.

    Say this first

    Every packet that arrives hits PREROUTING (raw, then conntrack, mangle, nat DNAT), then the routing decision. If it is for this host it goes to INPUT; if it is to be routed it goes to FORWARD and then POSTROUTING (where SNAT happens). Locally created packets are routed first, then go through OUTPUT and POSTROUTING. DNAT happens before routing so the new destination decides the path; SNAT happens last so filter rules still see the original source.

    Go deeper

    The three paths

    Output
    to the host:      in -> PREROUTING -> routing -> INPUT -> local process
through the host: in -> PREROUTING -> routing -> FORWARD -> POSTROUTING -> out
from the host:    local process -> routing -> OUTPUT -> POSTROUTING -> out

    Table order inside each hook

    HookOrder
    PREROUTINGraw, conntrack, mangle, nat (DNAT)
    INPUTmangle, filter, security, nat (SNAT for local delivery, rare)
    FORWARDmangle, filter, security
    OUTPUTraw, conntrack, mangle, nat (DNAT), filter, security
    POSTROUTINGmangle, nat (SNAT, MASQUERADE)

    These come from the hook priorities: raw -300, conntrack -200, mangle -150, DNAT -100, filter 0, security 50, SNAT 100. nftables uses the same numbers, which is why its chains declare priority filter or priority srcnat.

    Consequences interviewers ask about

    • A port-forward needs a FORWARD rule that matches the inside address and port, because DNAT already happened in PREROUTING.
    • A MASQUERADE rule does not change what FORWARD sees: the source is still the private address there.
    • Traffic to the router's own address hits INPUT, never FORWARD, even if it arrived on the "outside".
    • The nat table is consulted only for the first packet; later packets reuse the conntrack entry.
    • After OUTPUT, if mangle changed the mark or nat changed the destination, the kernel re-routes the packet.

    Traps

    • Writing the FORWARD rule with the public IP for a port-forward. After DNAT, FORWARD sees the private one.
    • Expecting INPUT rules to protect containers or VMs behind the host. Their traffic goes through FORWARD.
    How did you do?Book 8.5 Example SystemsOwn page
  8. What is the difference between MASQUERADE and SNAT, and when do you use each?

    Say this first

    Both rewrite the source address in nat POSTROUTING. SNAT uses the fixed address you give it with --to-source. MASQUERADE looks up the outgoing interface's address for each new connection, so it suits DHCP or PPPoE links whose address changes, and it drops the interface's connections when the interface goes down. With a static public address, SNAT is the cleaner choice; it does the same work without the lookup.

    Go deeper

    The two rules

    Shell
    iptables -t nat -A POSTROUTING -s 10.20.0.0/24 -o ens192 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.20.0.0/24 -o ens192 -j SNAT --to-source 198.51.100.10

    The nftables equivalents in an ip family nat table:

    Output
    oifname "ens192" ip saddr 10.20.0.0/24 masquerade
oifname "ens192" ip saddr 10.20.0.0/24 snat to 198.51.100.10

    How the translation works

    • Only the first packet of a connection goes through the nat table. The mapping is stored in conntrack and applied to every later packet in both directions.
    • Linux keeps the original source port if it is free on the public address, and picks another if not. --random-fully randomises port choice, which helps when many clients behind one IP hit the same server.
    • -s 10.20.0.0/24 -o ens192 keeps the rule from translating traffic that should stay internal, like traffic between two inside interfaces or to a VPN.

    Seeing it

    Output
    $ sudo conntrack -L -p tcp --dport 443
tcp      6 431998 ESTABLISHED src=10.20.0.15 dst=203.0.113.80 sport=40112 dport=443 src=203.0.113.80 dst=198.51.100.10 sport=443 dport=40112 [ASSURED] mark=0 use=1

    The reply tuple is addressed to 198.51.100.10: the NAT is visible there.

    Traps

    • Writing MASQUERADE without -o or -s and translating traffic that should stay internal.
    • Thinking the nat table is checked for every packet. Only the first one; conntrack does the rest.
    How did you do?Own page
  9. How do you forward a port on a Linux router to an internal server, and what usually goes wrong?

    Say this first

    Three pieces: a DNAT rule in nat PREROUTING that rewrites the destination, a FORWARD rule that allows the new inside address and port, and a server whose return path goes back through the router. What goes wrong: the FORWARD rule matching the public address, the server's default gateway pointing elsewhere, and inside users who try the public address and hit the hairpin problem.

    Go deeper

    Setup

    R1: outside ens192 198.51.100.10, inside ens224 10.20.0.1/24. Forward TCP 8443 to 10.20.0.50:443.

    Shell
    iptables -t nat -A PREROUTING -i ens192 -p tcp --dport 8443 -j DNAT --to-destination 10.20.0.50:443
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i ens192 -o ens224 -p tcp -d 10.20.0.50 --dport 443 -m conntrack --ctstate NEW -j ACCEPT

    nftables:

    Output
    table ip nat {
    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        iifname "ens192" tcp dport 8443 dnat to 10.20.0.50:443
    }
}

    Hairpin NAT

    An inside client 10.20.0.15 opens 198.51.100.10:8443. The rule above has -i ens192, so nothing happens. Even with the DNAT applied on the inside interface, the server replies straight to 10.20.0.15 on the same subnet, from 10.20.0.50, and the client drops it because it expected 198.51.100.10. Fixes:

    • Match -d 198.51.100.10 instead of -i ens192, and add iptables -t nat -A POSTROUTING -s 10.20.0.0/24 -d 10.20.0.50 -p tcp --dport 443 -j MASQUERADE so the server replies to the router.
    • Or better, split DNS: inside clients resolve the name to 10.20.0.50 directly.

    Traffic from the router itself

    Locally generated packets skip PREROUTING. To redirect them, add the DNAT in nat OUTPUT as well.

    Traps

    • Allowing -d 198.51.100.10 --dport 8443 in FORWARD. After DNAT, FORWARD sees 10.20.0.50 and port 443.
    • Forgetting that the server must route its replies through the box doing the DNAT.
    How did you do?Own page
  10. What is nftables, and how is it different from iptables?

    Say this first

    nftables is the replacement for iptables, ip6tables, arptables and ebtables, in the kernel since 3.13 and the default on current Debian, Ubuntu and RHEL. It has one tool, nft, one ruleset for all families, and no built-in tables or chains: you create them and attach chains to hooks with a priority. It adds sets and maps, so one rule can match hundreds of addresses, and it loads a whole ruleset atomically from a file. Today's iptables command is usually iptables-nft, which writes nftables rules under the hood.

    Go deeper

    A small ruleset

    Output
    table inet filter {
    set admin_nets {
        type ipv4_addr
        flags interval
        elements = { 10.0.50.0/24, 10.0.51.0/24 }
    }
    chain input {
        type filter hook input priority filter; policy drop;
        ct state established,related accept
        iif "lo" accept
        ip saddr @admin_nets tcp dport 22 accept
        tcp dport { 80, 443 } accept
    }
}

    Differences that matter

    iptablesnftables
    Toolsiptables, ip6tables, ebtables, arptablesnft
    Tables and chainsPredefinedYou define them, any name
    IPv4 and IPv6Separate rulesetsOne inet table can do both
    Many addressesOne rule each, or ipsetNative sets and maps
    CountersAlways onOnly where you write counter
    UpdatesRule by ruleAtomic, whole file with nft -f

    Which backend am I on

    Output
    $ iptables -V
iptables v1.8.10 (nf_tables)

    (nf_tables) means iptables-nft: rules you add with iptables appear in nft list ruleset. (legacy) means the old x_tables backend. iptables-translate converts a rule to nft syntax.

    Traps

    • Running nft flush ruleset on a box where Docker or firewalld also own rules. It wipes theirs too.
    • Forgetting counter and then reading empty counters as "no traffic".
    How did you do?Book 8.5 Example SystemsOwn page
  11. Write a minimal stateful host firewall that allows SSH from a management subnet and blocks everything else inbound.

    Say this first

    Default drop on input, then in order: allow loopback, allow established and related traffic, drop invalid, allow ICMP echo, allow TCP 22 only from the management subnet. Outbound stays open. The established rule near the top lets replies to the host's own connections back in and makes the rule set cheap, because most packets match it first. ICMP errors for existing flows arrive as RELATED, so PMTUD keeps working.

    Go deeper

    iptables

    Shell
    iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p tcp -s 10.0.50.0/24 --dport 22 -j ACCEPT
iptables -P INPUT DROP

    The policy is set last so an SSH session building the rules is never cut off halfway.

    nftables, same logic, IPv4 and IPv6 together

    Output
    #!/usr/sbin/nft -f
flush ruleset

table inet host {
    chain input {
        type filter hook input priority filter; policy drop;
        iif "lo" accept
        ct state established,related accept
        ct state invalid drop
        meta l4proto { icmp, ipv6-icmp } accept
        ip saddr 10.0.50.0/24 tcp dport 22 accept
    }
}

    Saved as /etc/nftables.conf (Debian, Ubuntu) or /etc/sysconfig/nftables.conf (RHEL), loaded at boot by nftables.service.

    Why ICMPv6 must be open

    IPv6 neighbour discovery and router advertisements are ICMPv6. Drop them and IPv6 stops working entirely, not just ping.

    Traps

    • Dropping all ICMP. IPv4 PMTUD and the whole of IPv6 depend on it.
    • Forgetting loopback. Local services talking to each other on 127.0.0.1 break in odd ways.
    How did you do?Book 8.5 Example SystemsOwn page
  12. What is conntrack, and what do the states NEW, ESTABLISHED, RELATED and INVALID mean?

    Say this first

    conntrack is the kernel's connection tracking table. It records every flow as a pair of tuples, original and reply, and it is what makes stateful filtering and NAT possible. NEW is the first packet of a flow. ESTABLISHED means traffic has been seen in both directions. RELATED is a new flow tied to an existing one, such as an ICMP error or FTP data with a helper. INVALID is a packet that fits no known flow, like a TCP segment far outside the window. UNTRACKED is traffic exempted in the raw table.

    Go deeper

    Reading the table

    Output
    $ sudo conntrack -L -p tcp --dport 443
tcp      6 431997 ESTABLISHED src=10.20.0.15 dst=198.51.100.80 sport=40112 dport=443 src=198.51.100.80 dst=203.0.113.10 sport=443 dport=40112 [ASSURED] mark=0 use=1
tcp      6 117 SYN_SENT src=10.20.0.16 dst=198.51.100.81 sport=52210 dport=443 [UNREPLIED] src=198.51.100.81 dst=203.0.113.10 sport=443 dport=52210 mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.
    • 6 is the protocol number, then the seconds left before the entry expires.
    • ESTABLISHED, SYN_SENT are conntrack's internal TCP states, more detailed than the firewall view.
    • The first tuple is the original direction, the second the expected reply. A different reply destination (203.0.113.10) means the flow is source NATed.
    • [UNREPLIED]: nothing came back yet. [ASSURED]: seen both ways, will not be evicted early.

    Default timeouts worth knowing

    TCP established 432000 seconds (5 days), TCP SYN_SENT 120 seconds, a UDP flow that has seen one packet 30 seconds; UDP flows with traffic both ways get a longer "stream" timeout. They are sysctls named net.netfilter.nf_conntrack_<protocol>_timeout....

    Firewall view

    Shell
    iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
nft add rule inet filter forward ct state established,related accept

    UDP and ICMP have no connections, but conntrack still gives them states: a DNS reply is ESTABLISHED because a matching query went out.

    Traps

    • Confusing conntrack's TCP state (SYN_SENT, TIME_WAIT) with the firewall state (NEW, ESTABLISHED).
    • Forgetting that conntrack keeps old NAT mappings after rule changes.
    How did you do?Own page
  13. How do you make a Linux bridge VLAN aware, with trunk and access ports?

    Say this first

    Create the bridge with vlan_filtering 1, then set VLAN membership per port with bridge vlan. A port with several tagged VLANs is a trunk. A port with one VLAN marked pvid untagged is an access port: untagged frames coming in join that VLAN, and frames going out lose their tag. Every port starts in VLAN 1 as PVID untagged, so remove it where you do not want it. One VLAN-aware bridge replaces the old pattern of one bridge per VLAN.

    Go deeper

    Config

    Shell
    ip link add br0 type bridge vlan_filtering 1
ip link set ens224 master br0          # trunk to the switch
ip link set vnet0 master br0           # VM in VLAN 10
ip link set vnet1 master br0           # VM in VLAN 20
bridge vlan add dev ens224 vid 10
bridge vlan add dev ens224 vid 20
bridge vlan del dev vnet0 vid 1
bridge vlan add dev vnet0 vid 10 pvid untagged
bridge vlan del dev vnet1 vid 1
bridge vlan add dev vnet1 vid 20 pvid untagged

    Result

    Output
    $ bridge vlan show
port              vlan-id
ens224            1 PVID Egress Untagged
                  10
                  20
br0               1 PVID Egress Untagged
vnet0             10 PVID Egress Untagged
vnet1             20 PVID Egress Untagged
    • ens224 is a trunk: VLAN 1 untagged (the native VLAN, which must match the switch) plus 10 and 20 tagged.
    • vnet0 and vnet1 are access ports in 10 and 20.
    • br0 is the host's own port. To give the host an address in VLAN 10: bridge vlan add dev br0 vid 10 self and ip link add link br0 name br0.10 type vlan id 10.

    The Cisco mapping

    PVID is the access VLAN or native VLAN. "Egress Untagged" is the VLAN that leaves without a tag. A VLAN listed without flags is tagged, as on an 802.1Q trunk.

    Traps

    • Forgetting VLAN 1 stays on every port as PVID untagged unless you remove it.
    • Adding VLANs to the ports but not to br0 self, then wondering why the host cannot reach a VLAN.
    How did you do?Book 3.2 Switched EthernetOwn page
  14. What are the Linux bonding modes, and which ones need configuration on the switch?

    Say this first

    Seven modes, 0 to 6. Mode 1, active-backup, needs nothing on the switch and works across two switches. Mode 4, 802.3ad, is LACP and needs an LACP port-channel on the switch (or MLAG across two). Modes 0 (balance-rr), 2 (balance-xor) and 3 (broadcast) need a static EtherChannel. Modes 5 (balance-tlb) and 6 (balance-alb) balance without switch support by playing with MACs. In a data centre I use mode 4 with miimon 100, and mode 1 when the switches cannot bundle.

    Go deeper

    The table

    ModeNameSwitch sideHow it sends
    0balance-rrStatic channelPacket by packet round robin, reorders TCP
    1active-backupNothingOne active link, others standby
    2balance-xorStatic channelHash per flow (xmit_hash_policy)
    3broadcastStatic channelEvery packet on every link
    4802.3adLACPHash per flow, LACP negotiates members
    5balance-tlbNothingOutbound spread by load, inbound on one link
    6balance-albNothingtlb plus inbound IPv4 spread by answering ARP with different slave MACs

    Parameters that matter

    • miimon 100: check link state every 100 ms. Without a monitor the bond never notices a failure.
    • lacp_rate fast: ask the partner to send LACPDUs every second instead of every 30.
    • xmit_hash_policy: layer2 (default), layer2+3, layer3+4.
    • primary ens1f0 in active-backup to prefer one link.

    Checking it

    cat /proc/net/bonding/bond0 shows the mode, the active slave or the LACP aggregator, and each slave's link state and failure count.

    Traps

    • Using mode 0 or 2 against switch ports with no channel configured. The switch sees the MAC flap between ports.
    • Leaving miimon unset and discovering the bond never failed over.
    How did you do?Own page
  15. What is a network namespace, and what does it isolate?

    Say this first

    A network namespace is a separate copy of the kernel network stack: its own interfaces, addresses, routing tables and rules, neighbour table, netfilter rules, conntrack table, sockets and net.* sysctls. A new namespace contains only a loopback interface, and it is down. Every container runs in its own network namespace; ip netns creates and enters named ones for labs and tooling.

    Go deeper

    Working with them

    Shell
    ip netns add red
ip netns list
ip netns exec red ip -br addr     # run any command inside
ip -n red link set lo up          # short form of ip netns exec red ip ...
ip link set ens256 netns red      # move a physical NIC in
ip netns exec red bash            # a shell inside
    Output
    $ ip netns exec red ip -br addr
lo               DOWN

    Bring lo up or anything that talks to 127.0.0.1 inside the namespace fails.

    Getting traffic in and out

    • A veth pair, one end inside and one in the root namespace or on a bridge.
    • Moving a physical NIC or SR-IOV virtual function inside.
    • macvlan or ipvlan devices on top of a host NIC.

    Finding namespaces you did not create

    ip netns list only shows names bind-mounted under /run/netns. Docker and Kubernetes namespaces belong to processes: lsns -t net lists them, and nsenter -t <pid> -n ip addr runs a command inside one.

    Traps

    • Forgetting ip link set lo up in a new namespace.
    • Expecting ip netns list to show Docker containers.
    How did you do?Own page
  16. How do you check, test and change MTU on Linux, and how do you find a path MTU problem?

    Say this first

    ip link show shows each interface's MTU; ip link set dev ens192 mtu 9000 changes it until reboot, and the network manager's config makes it permanent. To test a path, ping with the DF bit set and a payload of MTU minus 28: ping -M do -s 1472. Linux caches a smaller path MTU learned from ICMP "fragmentation needed" per destination, and ip route get shows it. The classic symptom of a broken path MTU is small requests working while large transfers or TLS handshakes hang.

    Go deeper

    Test

    Output
    $ ping -M do -s 1472 -c 1 10.0.10.30
PING 10.0.10.30 (10.0.10.30) 1472(1500) bytes of data.
1480 bytes from 10.0.10.30: icmp_seq=1 ttl=64 time=0.412 ms

$ ping -M do -s 8972 -c 1 10.0.10.30
PING 10.0.10.30 (10.0.10.30) 8972(9000) bytes of data.
ping: local error: message too long, mtu=1500

$ ping -M do -s 1472 -c 1 198.51.100.80
PING 198.51.100.80 (198.51.100.80) 1472(1500) bytes of data.
From 10.0.10.1 icmp_seq=1 Frag needed and DF set (mtu = 1400)
    • 1472 + 8 (ICMP) + 20 (IP) = 1500. For jumbo, 8972 + 28 = 9000.
    • "local error" means our own interface MTU is smaller than the packet.
    • "Frag needed" comes from a router on the path, here telling us the next link is 1400.

    The cache

    Output
    $ ip route get 198.51.100.80
198.51.100.80 via 10.0.10.1 dev ens192 src 10.0.10.21 uid 1000
    cache expires 586sec mtu 1400

    Learned values expire after 10 minutes (net.ipv4.route.mtu_expires 600). tracepath walks the path and reports where the MTU drops.

    Common overheads

    VXLAN adds 50 bytes (1450 inside a 1500 underlay), GRE 24 (1476), and WireGuard interfaces default to 1420.

    On a Linux router

    Clamp TCP MSS for tunnels and PPPoE: iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu, or in nftables tcp flags syn tcp option maxseg size set rt mtu.

    Traps

    • Testing with ping -s 1500. The payload size excludes the 28 bytes of headers.
    • Raising the server MTU without the switch ports, which then drop the jumbo frames as giants.
    How did you do?Book 3.3 Internet (IP)Own page

Expert6

  2. A Linux firewall starts dropping new connections and dmesg says the conntrack table is full. How do you handle it?

    Say this first

    When nf_conntrack_count reaches nf_conntrack_max, the kernel drops packets that would create new entries and logs nf_conntrack: table full, dropping packet. Existing flows keep working, new ones fail, which looks like random timeouts. Short term I raise the limit and check memory. Then I find out why: a SYN flood or scan, a leak of half-open entries, or long default timeouts on a box with huge flow counts. Stateless high-volume traffic, such as a busy DNS server, can be exempted in the raw table.

    Go deeper

    Confirm

    Output
    $ dmesg | grep conntrack
nf_conntrack: nf_conntrack: table full, dropping packet
$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262144
net.netfilter.nf_conntrack_max = 262144

    conntrack -S shows per-CPU counters; a rising drop and early_drop confirm it.

    Short-term relief

    Shell
    sysctl -w net.netfilter.nf_conntrack_max=1048576
echo 262144 > /sys/module/nf_conntrack/parameters/hashsize

    Keep the hash table at a sensible fraction of the max (a quarter is a common choice) so lookups stay fast. Each entry costs kernel memory, so check free RAM before multiplying the limit.

    Find the cause

    Shell
    conntrack -L -o extended | awk '{print $3}' | sort | uniq -c   # entries per protocol
conntrack -L -p tcp --state SYN_SENT | wc -l                   # SYNs never answered
conntrack -L -p tcp --state SYN_RECV | wc -l                   # half-open, answered once
conntrack -L -p tcp | awk '{print $5}' | sort | uniq -c | sort -rn | head   # top sources

    Structural fixes

    • Shorter timeouts where safe, for example nf_conntrack_tcp_timeout_established well below 5 days.
    • Exempt stateless traffic you do not need to track: iptables -t raw -A PREROUTING -p udp --dport 53 -j CT --notrack and the matching OUTPUT rule. These packets are then UNTRACKED and need their own explicit ACCEPT rules in filter.
    • Rate limiting or SYN cookies upstream for floods.

    Traps

    • Raising nf_conntrack_max without checking memory or the hash size.
    • Calling it a network outage. Established traffic still flows; only new flows fail.
    How did you do?Own page
  3. How does Docker's default networking work on a Linux host, and what does it do to your firewall?

    Say this first

    Each container gets its own network namespace with an eth0 that is one end of a veth pair. The other end is attached to the docker0 bridge, which holds the gateway 172.17.0.1/16. Docker enables ip_forward, adds a MASQUERADE rule so containers reach the outside, and adds DNAT rules in a DOCKER chain for every published port. Because published ports are DNATed, their traffic goes through FORWARD, not INPUT, so host firewall rules on INPUT (ufw's default) do not protect them. Your own rules belong in DOCKER-USER.

    Go deeper

    What Docker installs (excerpt)

    Output
    $ sudo iptables -t nat -S
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:80

    That last rule is docker run -p 8080:80. In the filter table Docker sets the FORWARD policy to DROP and jumps to DOCKER-USER before its own chains. The chain layout has changed between releases, but DOCKER-USER is the documented, stable place for your rules.

    Restricting a published port

    Shell
    iptables -I DOCKER-USER -i ens192 ! -s 10.0.50.0/24 -p tcp -m conntrack --ctorigdstport 8080 -j DROP

    --ctorigdstport matches the port before DNAT, because by the time FORWARD runs the destination port is already 80. Or publish only on loopback with -p 127.0.0.1:8080:80.

    User-defined networks

    docker network create makes a br-<id> bridge with its own subnet and MASQUERADE rule, and containers on it get Docker's embedded DNS at 127.0.0.11 for name resolution between containers.

    Traps

    • Trusting ufw or INPUT rules to protect published container ports.
    • Running nft flush ruleset or a full iptables-restore on a Docker host.
    How did you do?Own page
  4. A server has a 2 x 10G LACP bond but one big transfer never goes above 10 Gbps, and all traffic to the gateway uses one link. Why?

    Say this first

    LACP balances by flow, not by packet, so one TCP connection always rides one member link and is capped at 10G. On top of that the default xmit_hash_policy is layer2, which hashes only MAC addresses. Every routed packet has the gateway's MAC as destination, so with layer2 all off-subnet traffic picks the same link. layer3+4 hashes IPs and ports and spreads many flows across both links. Inbound balance is decided by the switch's own hash.

    Go deeper

    See the policy and the per-link load

    Output
    $ grep -E 'Mode|Hash' /proc/net/bonding/bond0
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
$ ip -s link show ens1f0 | grep -A1 TX
$ ip -s link show ens1f1 | grep -A1 TX

    One member's TX counters climbing and the other idle is the symptom.

    Change it

    Shell
    ip link set bond0 type bond xmit_hash_policy layer3+4

    Netplan: transmit-hash-policy: layer3+4 under the bond's parameters. The kernel documentation notes layer3+4 is not strictly 802.3ad compliant because fragments of one packet may take different links, which is harmless in practice.

    The switch side

    Return traffic is balanced by the switch. On a Catalyst, show etherchannel load-balance shows the method; port-channel load-balance src-dst-ip (or a mixed IP and port method where supported) avoids the same trap on the way in.

    What does give more than 10G to one transfer

    Several parallel flows (multi-stream copies, iperf3 -P 4), a faster single link, or mode 0 at the cost of TCP reordering. LACP never splits one flow.

    Traps

    • Promising 20G for a single flow on a 2 x 10G bond.
    • Leaving layer2 hashing on a server whose traffic all goes through one gateway.
    How did you do?Own page
  5. Using one Linux host, build two isolated hosts that can talk to each other and reach the internet. Walk me through it.

    Say this first

    Two namespaces as the hosts, a bridge in the root namespace as their switch and gateway, one veth pair per namespace with one end on the bridge. Address the namespaces in one subnet, put the gateway address on the bridge, add a default route in each namespace. For the internet: turn on ip_forward and MASQUERADE the subnet out the uplink. That is exactly what Docker does with docker0, and building it by hand shows you understand every piece.

    Go deeper

    The build

    Shell
    ip netns add red
ip netns add blue
ip link add br0 type bridge
ip addr add 10.50.0.1/24 dev br0
ip link set br0 up

for ns in red blue; do
  ip link add veth-$ns type veth peer name veth-$ns-br
  ip link set veth-$ns netns $ns
  ip link set veth-$ns-br master br0
  ip link set veth-$ns-br up
  ip -n $ns link set lo up
  ip -n $ns link set veth-$ns up
done

ip -n red addr add 10.50.0.10/24 dev veth-red
ip -n blue addr add 10.50.0.20/24 dev veth-blue
ip -n red route add default via 10.50.0.1
ip -n blue route add default via 10.50.0.1

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.50.0.0/24 ! -o br0 -j MASQUERADE

    Test in order

    1. ip netns exec red ping -c 2 10.50.0.20: bridging works (TTL 64, nothing routed).
    2. ip netns exec red ping -c 2 10.50.0.1: the gateway answers.
    3. ip netns exec red ping -c 2 198.51.100.1: forwarding and NAT work. conntrack -L -p icmp shows the translation to the host's uplink address.

    What usually breaks

    • A FORWARD policy of DROP, typical on a host with Docker. Allow -i br0 out and established traffic back.
    • br_netfilter loaded with bridge-nf-call-iptables=1: then even red-to-blue bridged traffic must pass FORWARD.
    • DNS inside the namespace: ip netns exec uses /etc/netns/red/resolv.conf if it exists, else the host's, which may point at 127.0.0.53, a stub that does not exist inside the namespace.

    Traps

    • Forgetting lo and one veth end, then debugging routing that is fine.
    • Writing MASQUERADE without ! -o br0 and translating the traffic between the namespaces too.
    How did you do?Book 3.5 ImplementationOwn page
  6. A capture on a Linux server shows TCP segments of 20 to 60 KB and checksum errors, yet the MTU is 1500. Is something broken?

    Say this first

    No. tcpdump sees packets where the kernel hands them to the driver, not on the wire. On transmit, TSO and GSO let the kernel pass one large segment that the NIC (or the last software step) cuts into MTU-sized frames, and checksum offload means the NIC fills in the checksum after tcpdump saw it, hence "incorrect". On receive, GRO merges consecutive segments before the capture point. The wire carries normal 1500-byte frames. To see the real frames, capture on a SPAN port or switch the offloads off for the test.

    Go deeper

    What it looks like

    Output
    10:02:31.114502 IP 10.0.10.21.443 > 10.0.10.30.51022: Flags [P.], seq 1:28961, ack 518, win 501, length 28960

    28960 bytes is 20 segments of 1448. With -v, outgoing packets show cksum 0x2e4f (incorrect -> 0x8a1b).

    Check the offloads

    Output
    $ ethtool -k ens192 | grep -E 'checksumming|segmentation-offload|receive-offload'
rx-checksumming: on
tx-checksumming: on
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off

    See wire-sized packets for a test

    Shell
    ethtool -K ens192 tso off gso off gro off
# capture, then put them back
ethtool -K ens192 tso on gso on gro on

    This costs CPU and changes behaviour slightly, so do it briefly and not on a loaded production box.

    Traps

    • Opening a ticket for "giant packets" or "bad checksums" seen in a host capture.
    • Disabling offloads permanently to make captures look tidy.
    How did you do?Own page