Octet

Say this first

The reply is routed by destination only, so it follows the main table's default route out the first ISP, but with the second ISP's address as source. The first ISP drops it as spoofed (BCP 38 filtering), or the client's firewall sees a reply from the wrong path. The fix is source-based policy routing: a second table with the second ISP's default route, and a rule that sends anything sourced from the second address to that table. Replies then leave the way the request came in.

Go deeper

Setup

ens192 198.51.100.10/29, gateway 198.51.100.9 (ISP1, main default route)
ens224 203.0.113.10/29, gateway 203.0.113.9 (ISP2)

The fix

Shell

echo '200 isp2' > /etc/iproute2/rt_tables.d/isp2.conf
ip route add 203.0.113.8/29 dev ens224 src 203.0.113.10 table isp2
ip route add default via 203.0.113.9 dev ens224 table isp2
ip rule add from 203.0.113.10 lookup isp2 priority 1000

Output

$ ip route show table isp2
default via 203.0.113.9 dev ens224
203.0.113.8/29 dev ens224 scope link src 203.0.113.10
$ ip route get 192.0.2.50 from 203.0.113.10
192.0.2.50 from 203.0.113.10 via 203.0.113.9 dev ens224 table isp2 uid 0
    cache

The connected route in table isp2 keeps traffic to ISP2's own subnet on the wire instead of via the gateway.

Persisting it

Netplan has routing-policy and a table key on routes; NetworkManager has ipv4.routing-rules and ipv4.route-table; systemd-networkd has [RoutingPolicyRule] sections. Plain ip commands vanish on reboot.

Why strict rp_filter is now happy too

The reverse path check looks up the route back to the client using the local address as the source, so the from 203.0.113.10 rule matches and the route points out ens224, the interface the packet came in on.

They will ask next

The box is a router doing port forwarding to an internal server. Does the source rule still work?
No. The internal server's reply arrives on the LAN interface with the server's private source, so from 203.0.113.10 never matches. Mark the connection when it arrives and restore the mark on the reply: iptables -t mangle -A PREROUTING -i ens224 -m conntrack --ctstate NEW -j CONNMARK --set-mark 2, iptables -t mangle -A PREROUTING -i ens256 -j CONNMARK --restore-mark, then ip rule add fwmark 2 lookup isp2. Conntrack reverses the DNAT on the way out, so the reply leaves ens224 with 203.0.113.10 as its source.
How would you send one application's outbound traffic out ISP2?
Run it as its own user and add ip rule add uidrange 1001-1001 lookup isp2. The lookup happens at connect time, so the source address is chosen from ens224 as well. Marking packets in mangle OUTPUT also works but the source was already chosen, so you then need SNAT on ens224.

Traps

Adding a second default route to main with a higher metric and expecting it to handle inbound replies.
Fixing it by setting rp_filter to 0. That hides the symptom and the ISP still drops the spoofed-looking reply.

How did you do?