What do the states in ip neigh mean, such as REACHABLE, STALE, DELAY and FAILED?
Show the answer
Say this first
ip neigh is the ARP table for IPv4 and the NDP cache for IPv6, with a state machine. REACHABLE: the
mapping was confirmed recently. STALE: the confirmation is old, but the entry is still used. DELAY:
the entry was used while stale and the kernel waits a few seconds for an upper layer, such as a TCP ACK,
to confirm it. PROBE: it is sending unicast ARP requests. FAILED: nobody answered. INCOMPLETE:
first resolution in progress. PERMANENT is a static entry.
Go deeper
Example
$ ip neigh show dev ens192
10.0.10.1 lladdr 00:1c:7f:3a:11:02 REACHABLE
10.0.10.30 lladdr 00:50:56:a1:9c:44 STALE
10.0.10.31 lladdr 00:50:56:a1:47:e0 DELAY
10.0.10.44 FAILED
10.0.10.5 lladdr 00:50:56:a1:00:05 PERMANENTThe lifecycle and default timers
- No entry: the kernel broadcasts a request, entry is
INCOMPLETE(3 tries,mcast_solicit, 1 s apart). - Reply received:
REACHABLEfor a random time between 15 and 45 seconds (base_reachable_time_ms30000, randomised by half either way). - Timer expires:
STALE. Still used to send traffic, nothing is sent on the wire yet. - A packet is sent using a stale entry:
DELAYfor 5 seconds (delay_first_probe_time). If TCP sees an ACK from that neighbour in that window, it confirms the entry and it returns toREACHABLE. - No confirmation:
PROBE, 3 unicast requests (ucast_solicit). No answer:FAILED.
So STALE is normal and healthy. FAILED and INCOMPLETE are the states that mean trouble.
Fixing entries
ip neigh flush dev ens192 # clear dynamic entries on one interface
ip neigh del 10.0.10.30 dev ens192
ip neigh replace 10.0.10.5 lladdr 00:50:56:a1:00:05 dev ens192 nud permanent
ip -s neigh show 10.0.10.30 # adds used/confirmed/updated timersThey will ask next
You moved an IP to a new server and some clients still reach the old one. What do you do?
Their ARP caches still point at the old MAC. Send gratuitous ARP from the new server:
arping -U -c 3 -I ens192 10.0.10.30(iputils arping). Clustering tools such as keepalived do this on failover. If a firewall or router ignores gratuitous ARP, clear its entry by hand.The kernel logs "neighbour table overflow". What happened?
The cache hit
gc_thresh3(1024 entries by default, withgc_thresh1128 andgc_thresh2512). It happens on hosts that sit on very large subnets or on Kubernetes nodes. Raise the three thresholds with sysctl undernet.ipv4.neigh.default..What does the router flag mean in an IPv6 entry?
The neighbour advertised itself as a router (the R flag in its NA). It appears for the default gateway's link-local address in
ip -6 neigh.
Traps
- Calling STALE an error. It is the normal resting state of an entry nobody has confirmed lately.
- Using
arp -nand missing IPv6 entries.ip neighshows both families.