Catch an intermittent WAN drop with IP SLA, tracking and EEM

Tasks

1. Synchronize R1's clock with NTP and log with millisecond timestamps.
2. Create IP SLA 10, an ICMP echo to 10.255.0.1 sourced from Gi0/0/0 every 5 seconds, running forever.
3. Track the reachability of IP SLA 10 with track object 10.
4. Write an EEM applet that runs when track 10 goes down and appends the clock, interface, route, IP SLA and log output to flash:wan-evidence.txt.
5. Make the applet send its own syslog message so the drop shows up in the central log server.
6. Verify the probe, the track and the evidence file after the next drop.

Solution

1. 1Accurate time and useful logsR1

   ntp server 10.1.0.123
   service timestamps log datetime msec show-timezone
   service timestamps debug datetime msec show-timezone
   logging buffered 64000 informational

   Without NTP, you cannot line your timestamps up with the carrier's or with R2's. A log timestamp that starts with * means the clock is not authoritative.

2. 2The probeR1

   ip sla 10
    icmp-echo 10.255.0.1 source-interface GigabitEthernet0/0/0
    threshold 1000
    timeout 2000
    frequency 5
   ip sla schedule 10 life forever start-time now

   Keep threshold at or below timeout, and timeout below the frequency, all in milliseconds except the frequency, which is in seconds. A probe every 5 seconds catches a 10 second drop. Without the ip sla schedule line the operation exists but never runs. If you only need an alert without EEM, ip sla reaction-configuration 10 react timeout threshold-type immediate action-type trapOnly with ip sla logging traps logs each timeout.

3. 3Track the probeR1

   track 10 ip sla 10 reachability

   No delay here on purpose. You want the evidence at the first failure, not after the link has been down for a while.

4. 4Save evidence when the track goes downR1

   event manager applet WAN-EVIDENCE
    event track 10 state down
    action 1.0 cli command "enable"
    action 2.0 cli command "show clock | append flash:wan-evidence.txt"
    action 3.0 cli command "show interfaces GigabitEthernet0/0/0 | append flash:wan-evidence.txt"
    action 4.0 cli command "show ip route 10.255.0.1 | append flash:wan-evidence.txt"
    action 5.0 cli command "show ip sla statistics 10 | append flash:wan-evidence.txt"
    action 6.0 cli command "show logging | append flash:wan-evidence.txt"
    action 7.0 syslog priority warnings msg "WAN probe to 10.255.0.1 failed, evidence appended to flash:wan-evidence.txt"

   The applet fires on the transition to down, once per drop, not every 5 seconds while it stays down. EEM's CLI session starts in user EXEC, so enable comes first. If the router uses AAA command authorization, add event manager session cli username <user> so the applet's commands are authorized.

Verify

R1# show ntp status | include Clock is
Clock is synchronized, stratum 3, reference is 10.1.0.123

R1# show ip sla statistics 10
IPSLAs Latest Operation Statistics

IPSLA operation id: 10
        Latest RTT: 9 milliseconds
Latest operation start time: 14:23:04 UTC Tue Nov 10 2026
Latest operation return code: OK
Number of successes: 1720
Number of failures: 6
Operation time to live: Forever

R1# show track 10
Track 10
  IP SLA 10 reachability
  Reachability is Up
    7 changes, last change 00:41:12
  Latest operation return code: OK
  Latest RTT (millisecs) 9

R1# show logging | include TRACK|HA_EM
Nov 10 13:41:52.317 UTC: %TRACK-6-STATE: 10 ip sla 10 reachability Up -> Down
Nov 10 13:41:53.104 UTC: %HA_EM-4-LOG: WAN-EVIDENCE: WAN probe to 10.255.0.1 failed, evidence appended to flash:wan-evidence.txt
Nov 10 13:42:02.322 UTC: %TRACK-6-STATE: 10 ip sla 10 reachability Down -> Up

R1# more flash:wan-evidence.txt
13:41:52.598 UTC Tue Nov 10 2026
GigabitEthernet0/0/0 is up, line protocol is up
  Hardware is ISR4331-3x1GE, address is 7c21.0e44.9a10 (bia 7c21.0e44.9a10)
  Description: WAN-CARRIER-CKT-88213
  Internet address is 10.255.0.2/30
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 7/255
...
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
...
Routing entry for 10.255.0.0/30
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via GigabitEthernet0/0/0
      Route metric is 0, traffic share count is 1
...

Mistakes people make

• Forgetting ip sla schedule, so the probe never runs and the track never changes.
• Setting timeout or threshold larger than the frequency allows. Keep threshold at or below timeout, and timeout below the probe interval.
• Leaving out enable as the first EEM action, or running AAA command authorization without event manager session cli username. The applet then fails silently.
• Running without NTP and millisecond timestamps. Evidence that cannot be matched with the carrier's clock is easy to dismiss.
• Probing a carrier device that rate-limits ICMP. Probe a device you own at the far end, and add a second probe further into HQ to tell the circuit apart from HQ.
• Appending to the same file forever. Check dir flash: now and then and archive the file once you have enough drops.

On other platforms

ping -D -O -i 1 10.255.0.1 | tee -a /var/log/wan-ping.log