Octet

Using one Linux host, build two isolated hosts that can talk to each other and reach the internet. Walk me through it.

Show the answer

Say this first

Two namespaces as the hosts, a bridge in the root namespace as their switch and gateway, one veth pair per namespace with one end on the bridge. Address the namespaces in one subnet, put the gateway address on the bridge, add a default route in each namespace. For the internet: turn on ip_forward and MASQUERADE the subnet out the uplink. That is exactly what Docker does with docker0, and building it by hand shows you understand every piece.

Go deeper

The build

Shell
ip netns add red
ip netns add blue
ip link add br0 type bridge
ip addr add 10.50.0.1/24 dev br0
ip link set br0 up

for ns in red blue; do
  ip link add veth-$ns type veth peer name veth-$ns-br
  ip link set veth-$ns netns $ns
  ip link set veth-$ns-br master br0
  ip link set veth-$ns-br up
  ip -n $ns link set lo up
  ip -n $ns link set veth-$ns up
done

ip -n red addr add 10.50.0.10/24 dev veth-red
ip -n blue addr add 10.50.0.20/24 dev veth-blue
ip -n red route add default via 10.50.0.1
ip -n blue route add default via 10.50.0.1

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.50.0.0/24 ! -o br0 -j MASQUERADE

Test in order

  1. ip netns exec red ping -c 2 10.50.0.20: bridging works (TTL 64, nothing routed).
  2. ip netns exec red ping -c 2 10.50.0.1: the gateway answers.
  3. ip netns exec red ping -c 2 198.51.100.1: forwarding and NAT work. conntrack -L -p icmp shows the translation to the host's uplink address.

What usually breaks

  • A FORWARD policy of DROP, typical on a host with Docker. Allow -i br0 out and established traffic back.
  • br_netfilter loaded with bridge-nf-call-iptables=1: then even red-to-blue bridged traffic must pass FORWARD.
  • DNS inside the namespace: ip netns exec uses /etc/netns/red/resolv.conf if it exists, else the host's, which may point at 127.0.0.53, a stub that does not exist inside the namespace.

Traps

  • Forgetting lo and one veth end, then debugging routing that is fine.
  • Writing MASQUERADE without ! -o br0 and translating the traffic between the namespaces too.
How did you do?Book 3.5 Implementation