A router's CPU is at 99 percent. How do you find out why?
Show the answer
Say this first
show processes cpu sorted first, and show processes cpu history to see whether it is constant or
spiky. On classic IOS the header splits the 5-second figure, for example 99%/85%: the second number is
time spent at interrupt level, so a high value means packets are being switched in software. If one
process tops the list, its name points the way: IP Input means packets punted for process switching,
ARP Input an ARP storm, BGP Router routing churn, SNMP ENGINE heavy polling. Then I find the punted
traffic and stop it.
Go deeper
R1# show processes cpu sorted
CPU utilization for five seconds: 99%/85%; one minute: 96%; five minutes: 94%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
89 9873211 41920334 235 12.47% 10.31% 9.88% 0 IP Input85 percent at interrupt level plus IP Input near the top: a lot of traffic is not being forwarded in the
fast path.
Reading the clues
| Clue | Likely cause | Next step |
|---|---|---|
IP Input high | packets punted: TTL expiring in a loop, IP options, traffic to the router itself, ACL lines with log | show interfaces stats for process-switched counts, show ip traffic |
ARP Input high | ARP storm or scan, or a static route pointing out an Ethernet interface with no next hop | show ip arp summary, broadcast counters |
BGP Router, OSPF processes | churn from a flapping peer or link | show ip bgp summary, show ip ospf statistics |
SNMP ENGINE | a poller walking a big table too often | find the poller, slow it down |
Exec, Virtual Exec | heavy commands, or debug left running | show users, undebug all |
IOS-XE platforms
On ISR 4000, ASR 1000 and Catalyst 9000, forwarding happens in a separate data plane (the QFP on the
routers, the UADP ASIC on the switches), and show processes cpu sorted covers the IOSd control plane. show processes cpu platform sorted shows the underlying Linux processes.
High CPU there usually means punted traffic, and the platform punt statistics show which cause is hot.
Protect the control plane
- CoPP to cap punted traffic per class.
- Remove
logfrom busy ACL lines, fix loops, and point static routes at a next hop, not just an interface. - Never leave a debug running on a busy production device.
They will ask next
Does high CPU on a switch affect forwarding?
Hardware-switched traffic usually keeps flowing, but anything that needs the CPU suffers: routing hellos, BPDUs, LACP, ARP and management access. Lost BPDUs or hellos then cause loops or adjacency flaps, so high CPU can grow into an outage.
Traps
- Running
debug ip packeton a router that is already at 99 percent CPU.