Octet

Some flows between two sites fail or crawl while others work fine. What do you suspect?

Show the answer

Say this first

A fault on one member of a set of parallel paths. ECMP and port channels hash each flow onto one link by its addresses and often its ports, so a bad member only hurts the flows that land on it: one user's share fails while a colleague's works, or a connection works on retry because a new source port hashes differently. I suspect a member with errors, a member up but not forwarding, an MTU or ACL difference on one path, or one next hop missing a route. I find which path a failing flow takes and test that member.

Go deeper

Why only some flows

Load sharing is per flow, not per packet, to keep packets in order. The hash uses source and destination IP and often layer 4 ports. A failing flow keeps failing until its ports change.

Finding the bad member

  • Which path does a failing pair take? show ip cef exact-route 10.10.0.25 10.30.0.15 on IOS for ECMP. For port channels, show etherchannel load-balance shows the hash inputs, and most platforms have a command that computes the member for a given flow.
  • Per-member health: show interfaces on each member, show etherchannel summary for flags such as suspended or standalone.
  • Compare members: light levels, MTU, ACLs, and allowed VLANs on channel members configured by hand.
  • Unidirectional fiber: a link that transmits but cannot receive in one direction. UDLD detects it.
  • In a maintenance window, shut the suspect member and see whether the failing flows recover.

Hash polarization

If several layers of routers use the same hash inputs and algorithm, flows that arrive on one branch all hash the same way again at the next layer, leaving links idle. Different hash inputs or seeds per layer fix it. That is uneven load, not failure, but it turns up in the same investigation.

Traps

  • Testing with a ping from the router, which may hash onto a different member than the failing application flow.
How did you do?