Make a dual-homed Linux server answer on both ISP links with policy routing

LinuxExpert

SRV1 has two internet uplinks. The main table's default route uses ISP1, so connections arriving on ISP2 get replies through ISP1 with ISP2's address as source, and ISP1 drops them. Add a routing table for ISP2 and rules so that replies leave the way the request came in, and send one service account's outbound traffic over ISP2.

Uplink	Interface	Address	Gateway	Role
ISP1	`ens192`	198.51.100.10/29	198.51.100.9	Default route in table main
ISP2	`ens224`	203.0.113.10/29	203.0.113.9	Table 200, named `isp2`

The backup agent runs as user backup (UID 1001) and must leave through ISP2. A test client on the internet is 192.0.2.50.

Tasks

Give table 200 the name isp2.
Fill table isp2 with ISP2's connected subnet and a default route via 203.0.113.9.
Add a rule so that traffic sourced from 203.0.113.10 uses table isp2.
Add a rule so that all traffic from UID 1001 uses table isp2.
Make the source rule and table survive a reboot with netplan.

Solution

1Name the tableSRV1
SRV1 on Linux
```
mkdir -p /etc/iproute2/rt_tables.d
echo '200 isp2' > /etc/iproute2/rt_tables.d/isp2.conf
```
Optional. Every command also accepts the number 200. The name only makes output readable.
2Fill the isp2 tableSRV1
SRV1 on Linux
```
ip route add 203.0.113.8/29 dev ens224 src 203.0.113.10 table isp2
ip route add default via 203.0.113.9 dev ens224 table isp2
```
The connected route keeps traffic to ISP2's own subnet on the wire instead of sending it to the gateway.
3Source rule for symmetric repliesSRV1
SRV1 on Linux
```
ip rule add from 203.0.113.10 lookup isp2 priority 1000
```
Replies to connections that arrived on ISP2 carry 203.0.113.10 as source, so they match this rule. It also keeps strict rp_filter happy, because the reverse path check uses the local address as source.
4Per-user rule for the backup agentSRV1
SRV1 on Linux
```
ip rule add uidrange 1001-1001 lookup isp2 priority 1100
```
The lookup happens when the socket connects, so the kernel also picks 203.0.113.10 as the source address. Marking packets in mangle OUTPUT would also work, but the source is already chosen by then and you would need SNAT on ens224.

5Persist the table and source rule with netplanSRV1

SRV1 on YAML

# /etc/netplan/01-uplinks.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    ens192:
      addresses: [198.51.100.10/29]
      routes:
        - to: default
          via: 198.51.100.9
    ens224:
      addresses: [203.0.113.10/29]
      routes:
        - to: default
          via: 203.0.113.9
          table: 200
        - to: 203.0.113.8/29
          scope: link
          table: 200
      routing-policy:
        - from: 203.0.113.10
          table: 200
          priority: 1000

Apply with netplan try. Netplan takes table numbers, not names. Its routing-policy has no user selector, so the uidrange rule needs its own small systemd unit that runs after the network is up.

Verify

SRV1 output

SRV1# ip rule
0:      from all lookup local
1000:   from 203.0.113.10 lookup isp2
1100:   from all uidrange 1001-1001 lookup isp2
32766:  from all lookup main
32767:  from all lookup default

Both new rules sit between local (0) and main (32766), so they are consulted before the main table.

SRV1 output

SRV1# ip route show table isp2
default via 203.0.113.9 dev ens224
203.0.113.8/29 dev ens224 scope link src 203.0.113.10

The table has its own default route and the connected subnet.

SRV1 output

SRV1# ip route get 192.0.2.50 from 203.0.113.10; ip route get 192.0.2.50 uid 1001; ip route get 192.0.2.50
192.0.2.50 from 203.0.113.10 via 203.0.113.9 dev ens224 table isp2 uid 0
    cache
192.0.2.50 via 203.0.113.9 dev ens224 table isp2 src 203.0.113.10 uid 1001
    cache
192.0.2.50 via 198.51.100.9 dev ens192 src 198.51.100.10 uid 0
    cache

Traffic from 203.0.113.10 and from UID 1001 uses ISP2 (and UID 1001 gets 203.0.113.10 as source). Everything else still uses ISP1.

SRV1 output

SRV1# tcpdump -i ens224 -nn 'tcp port 443 and host 192.0.2.50'
11:05:42.318201 IP 192.0.2.50.53122 > 203.0.113.10.443: Flags [S], seq 901223411, win 64240, options [mss 1460,sackOK,TS val 51120 ecr 0,nop,wscale 7], length 0
11:05:42.318255 IP 203.0.113.10.443 > 192.0.2.50.53122: Flags [S.], seq 4412091877, ack 901223412, win 65160, options [mss 1460,sackOK,TS val 88213 ecr 51120,nop,wscale 7], length 0

The SYN from the test client arrives on ens224 and the SYN-ACK leaves on ens224. Before the fix, the SYN-ACK left on ens192.

Mistakes people make

Putting a rule at a priority above 32766. Main is consulted first and the rule never matters.
Leaving the connected route out of table isp2, so traffic to ISP2's own subnet hairpins through the gateway.
Using fwmark rules for local traffic without SNAT. The source address was already picked from ISP1.
Adding a second default route to main with a higher metric and expecting it to handle replies. Metrics only choose between routes for the same destination.
Lowering rp_filter instead of fixing the routing. The upstream ISP still drops the spoofed-looking replies.

On other platforms

The same rule and table with NetworkManager on a RHEL family host.

Linux

nmcli con mod ens224 ipv4.routes "0.0.0.0/0 203.0.113.9 table=200, 203.0.113.8/29 table=200" \
  ipv4.routing-rules "priority 1000 from 203.0.113.10 table 200"
nmcli con up ens224

Say it in the interview

"Linux routes by destination, so replies to ISP2 traffic followed the main default out ISP1 with the wrong source. I gave ISP2 its own table and a source rule, so anything from its address uses its gateway, and a uidrange rule moves the backup user's traffic over too. ip route get with from and uid proves each path before any real traffic does."

Book 3.3 Internet (IP)