How do you troubleshoot an intermittent problem you cannot reproduce on demand?
Show the answer
Say this first
I stop guessing and set traps. First I look for a pattern: time of day, which users, which path, how long each event lasts. Then I make sure the next event leaves evidence: NTP and millisecond timestamps, syslog to a server, continuous probes such as IP SLA or mtr, short polling intervals on the suspect interfaces, and a ring-buffer capture or an EEM script that collects show output the moment it happens. Typical causes are a marginal link, flapping, peak-time congestion, a duplicate IP, STP topology changes and failing hardware.
Go deeper
Find the pattern
- Same time every day: backups, scheduled jobs, peak-hour congestion.
- Every few minutes or hours: a timer. Examples are ARP aging (4 hours on Cisco) against MAC aging (5 minutes) causing unicast flooding, VPN rekeys, and DHCP lease renewals.
- Only some users or flows: one bad member of a port channel or ECMP group, a duplicate IP, wireless roaming.
- After an event: an STP topology change.
show spanning-tree detailshows the last change and the port it came from.
Make the next occurrence leave evidence
- NTP on everything,
service timestamps log datetime msec, logs to a syslog server. - Continuous probes: IP SLA with a track object, or
mtr -rwc 600 10.30.0.15from a host. - A ring-buffer capture that keeps the last 1 GB:
tcpdump -i eth0 -w /var/tmp/cap.pcap -C 100 -W 10 host 10.30.0.15. - An EEM applet that saves show output when a track object goes down or a syslog pattern appears.
Typical culprits
| Clue | Cause |
|---|---|
| CRC slowly climbing | marginal cable or optic |
Bursts of %LINK-3-UPDOWN | flapping link, bad cable, NIC power saving |
| STP topology change count rising | a non-edge port going up and down somewhere |
%IP-4-DUPADDR or ARP entries changing MAC | duplicate IP |
| Only at peak hours | congestion, output drops |
| Only some flows | a bad LAG or ECMP member |
They will ask next
The user says it happens "randomly". How do you get a usable timeline?
Ask them to note the exact time and what they were doing the next few times, or give them a small script that logs a timestamped ping or curl. Then match those times with syslog, probes and graphs.
Why do STP topology changes cause short slowdowns?
A topology change makes switches age out or flush MAC entries, so traffic floods until the tables relearn. A host port without PortFast causes a topology change every time a PC boots or sleeps.
Traps
- Changing things between occurrences without writing it down, so nobody knows what fixed it.