Troubleshooting

0 of 40 nailed

Core16

Walk me through how you troubleshoot a network problem.

Say this first

I define the problem before I touch anything: what exactly fails, for whom, since when, and what changed. Then I gather facts, rank the likely causes, and test one hypothesis at a time. I pick the method to fit the symptom: bottom-up for physical symptoms, top-down when one application fails, divide and conquer from a ping when I have no strong lead, or follow the path hop by hop. I verify the fix from the user's side and write down the root cause, not just what I changed.

Go deeper

The loop

Define the problem. Turn "the network is slow" into "users on floor 3 see 5-second page loads to the CRM since 09:10".
Gather facts. Scope, timing, recent changes, syslog, monitoring graphs, one reproducible example.
List likely causes. Rank them by how likely they are and how cheap they are to test.
Test one hypothesis at a time. Change one thing, observe, and revert it if it made no difference.
Fix and verify from the user's point of view, not only from the router.
Document. Timeline, cause, fix, and the follow-up actions that stop it happening again.

Picking a method

Method	Start at	Best when
Bottom-up	Layer 1	new install, link lights, interface errors
Top-down	The application	one app fails while others work
Divide and conquer	Layer 3, usually a ping	no strong lead; one test rules out half the stack
Follow the path	The source, hop by hop	multi-hop paths, firewalls, NAT, routing
Compare with working	A known-good device or config	one site or port fails, a similar one works
Swap components	Suspect hardware	cable, optic or NIC is the likely culprit

What a senior answer adds

Read before you write. Capture show output before any change, keep a rollback ready, and tell the incident lead before touching production config. Time-box each hypothesis so you do not sink an hour into a guess.

They will ask next

How do you avoid making an outage worse while you troubleshoot?
Read-only commands first. Save the current state before changes. One change at a time, each with a rollback. On a remote device I use a safety net such as reload in 10 or an archived config and configure replace, so a mistake that cuts my access undoes itself.
When do you escalate?
When the fault is outside my access or skill, when the impact is growing faster than I can restore it, or when my time box runs out. I escalate with facts: what fails, the scope, what I tested and what each test showed.

Traps

Jumping to a fix before the problem is defined.
Changing several things at once, so nobody learns which one fixed it.

How did you do?Book 1.3 Architecture Own page

What is the difference between top-down, bottom-up and divide-and-conquer troubleshooting?
Say this first
All three use the OSI layers as a checklist. Bottom-up starts at the physical layer and works up: thorough but slow, good for new installs or cabling suspicions. Top-down starts at the application and works down: good when one app fails and others work. Divide and conquer starts in the middle, usually with a ping at layer 3. If it works, the lower layers on that path are fine and I go up; if it fails, I go down.

Go deeper
Bottom-up
Link lights and show interfaces status, then counters, speed and duplex, then VLAN, trunk and STP, then IP and routing, then transport and the application. Nothing is skipped, which is why it wastes time on a problem that turns out to be DNS.

Top-down
Start with what the user does. Does the page load? Does DNS resolve? Does TCP 443 connect? Then go down to routing and links. Works well when the user's report is specific.

Divide and conquer
Pick the layer that splits the stack, usually layer 3. A successful ping proves the physical layer, the data link and IP routing in both directions along that path, for ICMP. Most experienced engineers start here because one command rules out half the possibilities.

The limit of a passing ping
It does not prove that a TCP port is allowed, that large packets pass, or that the application takes the same path. Policy-based routing, proxies, load balancers and per-protocol firewall rules all break that assumption.
They will ask next
Which one do you use most?
Divide and conquer. A ping and a traceroute from the right source take seconds and cut the problem in half. The moment I see a down link or climbing error counters, I switch to bottom-up.

Where does the OSI model stop helping?
Real faults cross layers. An MTU problem looks like an application hang, and a duplex mismatch looks like slowness. I use the layers as a checklist, not as a diagnosis.
Traps
Saying a successful ping proves the application path works.
How did you do?Book 1.3 Architecture Own page
A user calls and says "the network is down". What do you ask before you touch anything?
Say this first
I narrow the scope. What exactly fails: one app, one site, everything? Who else is affected? Since when? All the time or on and off? Wired, wireless or VPN? What changed, on their side and ours? Then I get one concrete example I can reproduce: source host, destination, time and the exact error. Those answers usually point to a layer and a location before I log in to anything.
Go deeper
The questions and what each answer points to

What fails exactly? "Down" often means one website, email, Wi-Fi or the VPN.

Who is affected? One user points at the host, cable or port. A floor points at a switch, an uplink or a VLAN. A whole site points at the WAN, the edge, or a shared service like DNS or DHCP.

Since when? Line it up against the change log, monitoring and syslog.

Constant or intermittent? Intermittent points at errors, flapping, congestion or a duplicate IP.

What changed? A new laptop, a desk move, a patch, our own change window.

The exact error text. "Name not resolved" and "connection timed out" send you to different layers.

Then pin one example
Source IP, destination name and IP, port, time. Every later test uses the same example, so results can be compared and handed over.
They will ask next
The user cannot give you details. What next?
Get the facts from the host and the network directly: IP, gateway and DNS from the endpoint tool or ipconfig /all, then the switch port from the MAC address, then logs and monitoring for that port and VLAN.
Traps
Taking "the network is down" literally. Most of those tickets are one app, DNS, or a single host.
How did you do?Own page

Give me an example of a fault at each layer and the command that finds it.

Say this first

Layer 1: a dirty fiber or bad cable, seen as CRC errors in show interfaces or low light in show interfaces transceiver. Layer 2: a port in the wrong VLAN or a trunk not carrying it, seen with show interfaces status and show interfaces trunk. Layer 3: a missing route or wrong gateway, seen with show ip route and ping. Layer 4: an ACL or firewall dropping a port, seen with ACL hit counts and nc -vz. Application: DNS or the service, seen with nslookup, dig and curl -v.

Go deeper

Layer	Example fault	First command
1 Physical	dirty fiber, damaged cable, wrong optic type	`show interfaces`, `show interfaces transceiver detail`
2 Data link	wrong access VLAN, VLAN missing on trunk, STP blocking, duplex mismatch	`show interfaces status`, `show interfaces trunk`, `show spanning-tree vlan 10`
3 Network	missing route, wrong mask or gateway, duplicate IP	`show ip route`, `show ip arp`, `ping`, `traceroute`
4 Transport	ACL or firewall dropping TCP 443, MSS too large	`show access-lists`, `nc -vz 10.30.0.15 443`, a capture
5 to 7	DNS, expired TLS certificate, app error	`nslookup`, `dig`, `curl -v`, `openssl s_client -connect host:443`

Layers 5 and 6 rarely matter on their own in practice. TLS is the usual one: an expired certificate or a server that does not send its intermediate certificate.

They will ask next

Where does a duplex mismatch show up?
Layers 1 and 2. The half-duplex side counts late collisions; the full-duplex side counts CRC errors and runts. Ping looks fine, but throughput collapses under load.

Traps

Getting stuck arguing which layer ARP belongs to. Say it maps layer 3 addresses to layer 2 and move on.

How did you do?Book 1.3 Architecture Own page

How does ping work, and what can a failed ping tell you?

Say this first

Ping sends ICMP echo requests (type 8) and waits for echo replies (type 0). A reply proves two-way IP reachability for ICMP and gives round-trip time and loss. A failure is only useful once you read why: a timeout means something dropped the request or the reply, destination unreachable (type 3) names the router that could not deliver, and TTL exceeded (type 11) points at a loop. On Cisco, ! is a reply, . is a timeout and U is an unreachable.

Go deeper

What travels

Echo request is ICMP type 8 code 0, carrying an identifier, a sequence number and a data payload. Echo reply is type 0. ICMP is IP protocol 1. Because the reply has to come back, ping tests both directions.

Defaults worth knowing

Cisco IOS: 5 echoes, 100-byte datagrams (the size includes the IP header), 2-second timeout.
Windows: 4 echoes with 32 bytes of data.
Linux: runs until you stop it, 56 bytes of data (64 bytes of ICMP with the header).

Cisco ping characters

Char	Meaning
`!`	reply received
`.`	timed out waiting for a reply
`U`	destination unreachable received
`M`	could not fragment (DF set, packet too big)
`Q`	source quench (obsolete)
`&`	packet lifetime exceeded
`?`	unknown packet type

Why the first ping from a router often shows `.!!!!`

The router has no ARP entry yet for the next hop or the host. It drops the packet that triggered the ARP request, so the first echo times out and the rest succeed. 80 percent on the first try is normal.

Reading failures

Timeouts: no route somewhere, a filter, or a host that drops ICMP. Many Windows installs block inbound echo requests in the host firewall.
Unreachable from a router: that router has no route, or got no ARP answer for the host.
TTL expired in transit: a routing loop.

They will ask next

A host fails ping but users can browse to it. Why?
Its host firewall drops echo requests, or an ACL in the path allows TCP but not ICMP. Test the real port instead: Test-NetConnection 10.30.0.15 -Port 443 on Windows, nc -vz 10.30.0.15 443 on Linux.
On Windows you see "Reply from 10.1.1.20 Destination host unreachable", and 10.1.1.20 is your own address. What does that mean?
Your own host generated the unreachable because it could not ARP for the destination or the gateway. Look at layer 2: the VLAN, the port, or whether the target is powered on.

Traps

Treating a failed ping as proof a host is down. Plenty of hosts and firewalls drop ICMP.
Forgetting that a ping also tests the return path.

How did you do?Book 3.3 Internet (IP)Own page

Ping works but the application does not. Where do you look next?

Say this first

Ping only proves ICMP gets there and back. Next I test what the application really uses: does the name resolve to the address I pinged, does TCP connect on the right port, and does the service answer. That is nslookup, then nc -vz or Test-NetConnection -Port, then curl -v. The usual causes are DNS, an ACL or firewall that permits ICMP but not the port, MTU, a proxy or load balancer, or the service itself.

Go deeper

Carry on up the stack

Name. Does the client resolve the name to the IP you pinged? Split DNS and stale records are common.
Port. nc -vz 10.30.0.15 443 on Linux or Test-NetConnection 10.30.0.15 -Port 443 on Windows.
Large packets. If TCP connects but the page hangs, suspect MTU and test with ping -M do -s 1472.
Application. curl -v https://app.example.com/ shows resolution, connect, TLS and the HTTP status.
Path. Policy-based routing, a proxy or a load balancer can send TCP 443 a different way from ICMP.

Read the TCP result

What you see	Likely meaning
SYN, then nothing	dropped by a firewall or ACL, broken return route, or host down
SYN, then RST	host reached but nothing listens, or a firewall set to reject
Handshake completes, then a hang	MTU, a middlebox, or the application

They will ask next

What does "connection refused" prove?
Something answered with a TCP RST, so routing works in both directions to that point. Either nothing listens on that port or a device in the path resets on the host's behalf. In a capture, the RST's TTL often shows whether it came from the host or from a firewall closer to you.

Traps

Closing the ticket because "ping works".

How did you do?Own page

How does traceroute work?

Say this first

It sends probes with TTL 1, then 2, then 3, and so on. Each router that decrements the TTL to zero drops the probe and returns ICMP time exceeded (type 11), which reveals its address. The destination answers differently: ICMP port unreachable for a UDP probe, or an echo reply for an ICMP probe, and that ends the trace. Linux, macOS and Cisco IOS send UDP to high ports starting at 33434. Windows tracert sends ICMP echo. Three probes per hop by default.

Go deeper

Step by step

The first probe leaves with TTL 1. The first router decrements it to 0, drops it, and sends ICMP type 11 code 0 (TTL exceeded in transit) back from its own address, usually the interface facing you.
The probe with TTL 2 expires at the second router, and so on.
The destination gets a probe with TTL still above zero. For UDP to an unused high port it answers ICMP type 3 code 3 (port unreachable). For ICMP echo it sends an echo reply. Either one ends the trace.

Variants

Tool	Probe	End signal
Linux and macOS `traceroute`	UDP, port 33434 and up	port unreachable
`traceroute -I`	ICMP echo	echo reply
`traceroute -T -p 443` (Linux, root)	TCP SYN	SYN-ACK or RST
Windows `tracert`	ICMP echo	echo reply
Cisco IOS `traceroute`	UDP, port 33434 and up	port unreachable

Reading a line

Each hop shows three round-trip times, one per probe. The time includes how long that router's CPU took to build the ICMP message. Cisco marks * for a timeout and codes such as !H (host unreachable), !N (network unreachable) and !A (administratively prohibited).

They will ask next

Why does Unix traceroute use high UDP ports?
So nothing is likely to be listening at the destination, which makes the host answer with port unreachable and tells traceroute it has arrived. The port changes per probe so replies can be matched.
A firewall blocks UDP probes. What do you use instead?
ICMP with traceroute -I or Windows tracert, or TCP to an allowed port with traceroute -T -p 443. A TCP trace to the service port is subject to the same firewall rules as the application.
How does traceroute match a reply to its probe?
An ICMP error quotes the original IP header and at least the first 8 bytes of the original payload, which hold the UDP ports or the ICMP identifier and sequence. traceroute matches on those.

Traps

Saying traceroute shows the return path. It shows the forward path only.
Saying Windows tracert uses UDP. It uses ICMP echo.

How did you do?Book 3.3 Internet (IP)Own page

Which show commands do you run first on a Cisco device, and why?

Say this first

show ip interface brief for interface and protocol state. show logging for what changed and when. show interfaces on the suspect ports for errors and drops. show version for uptime and the last reload reason. show ip route <destination> for the path, show ip arp and show mac address-table to find the host, and show cdp neighbors or show lldp neighbors to confirm what connects where. In a minute they tell me whether the box restarted, a link is down or erroring, or a route is missing.

Go deeper

Command	What it answers
`show ip interface brief`	which interfaces are up/up, administratively down, or up/down
`show logging`	link flaps, neighbor changes, err-disable events, duplicate addresses, with times
`show version`	uptime, last reload reason, software version
`show interfaces Gi1/0/48`	errors, drops, load, speed and duplex, when counters were cleared
`show interfaces status`	per-port VLAN, speed, duplex, and err-disabled ports on a switch
`show ip route 10.30.0.15`	which route matches and the next hop
`show ip arp 10.10.0.57`, `show mac address-table address <mac>`	where a host lives
`show cdp neighbors`, `show lldp neighbors`	what is plugged into each port
`show processes cpu sorted`	whether the device itself is struggling

Habits that save time

terminal length 0 before collecting output, so it does not page.
Timestamps only help if they are right: service timestamps log datetime msec localtime and NTP.
Save what you collect before you change anything. It is your baseline and your evidence.

Other platforms

Junos: show interfaces terse, show log messages, show route 10.30.0.15. NX-OS: show interface brief, show logging logfile. Arista EOS is close to IOS: show ip interface brief, show logging.

They will ask next

Why look at show version early?
Uptime and the last reload reason tell you whether the device crashed or was reloaded. A crash moves the investigation to software, power or hardware, and a crashinfo file may be waiting on flash.

Traps

Reading logs without synced clocks. You cannot line events up across devices.

How did you do?Own page

Walk me through `show interfaces` output. What do you look at?

Say this first

The first line gives interface and line protocol state. Then speed and duplex, the input and output rates against the bandwidth, and the counters: input errors, CRC, runts, giants, overruns, output drops, collisions, late collisions and interface resets. I check when counters were last cleared, because a big number collected over months means little. What matters is which counters are climbing now.

Go deeper

A port with a problem

Output

SW1# show interfaces GigabitEthernet1/0/24
GigabitEthernet1/0/24 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 7c21.0e4f.2a18 (bia 7c21.0e4f.2a18)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 12/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters 00:15:12
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1432
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 48210000 bits/sec, 5120 packets/sec
  5 minute output rate 12730000 bits/sec, 3890 packets/sec
     4523110 packets input, 4980112245 bytes, 0 no buffer
     Received 1203 broadcasts (884 multicasts)
     27 runts, 0 giants, 0 throttles
     418 input errors, 391 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 884 multicast, 0 pause input
     0 input packets with dribble condition detected
     3500211 packets output, 1120340099 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

In 15 minutes: 391 CRC errors and 27 runts (together the 418 input errors), so frames arrive damaged. That is a physical problem on the receive side of this port. Separately, 1432 output drops at a low average load suggest bursts toward this port.

Field by field

Field	Meaning
`up, line protocol is up`	layers 1 and 2 working; `administratively down` means shutdown
`txload`, `rxload`	load out of 255, from the 5-minute average
`Full-duplex, 1000Mb/s`	negotiated or set values; half duplex on a modern link is a red flag
input errors	the sum of CRC, frame, runts, giants, overruns, ignored and similar
CRC	frames that failed the FCS check: cabling, optics, duplex
runts, giants	frames below 64 bytes or above the maximum size
overrun, ignored, no buffer	good frames the hardware could not take in time
Total output drops	the egress queue was full: congestion or bursts
collisions, late collision	only on half duplex; late collisions mean duplex mismatch or an over-long cable
interface resets	the interface was reset, for example after a flap or a keepalive problem

How I use it

Note the numbers, wait a few minutes and look again, or clear counters GigabitEthernet1/0/24 and watch. Rates are 5-minute averages by default; load-interval 30 on the interface makes them react faster.

They will ask next

What does "up, line protocol is down" mean?
The physical layer has a signal but the data link layer is not working. On serial links it is usually an encapsulation mismatch or keepalive failure. On a switch, read the reason in brackets: a SPAN destination, for example, shows line protocol is down (monitoring).
Why set load-interval 30?
The default 5-minute average smooths bursts away. 30 seconds is the shortest interval and reacts faster, but even that hides millisecond microbursts.

Traps

Quoting a large error count without checking "Last clearing" first.

How did you do?Book 2.4 Error Detection Own page

An interface shows CRC errors that keep climbing. What does it mean and what do you do?
Say this first
A CRC error is a frame whose frame check sequence did not match: the bits were damaged in transit. It is counted on the receiving side, so the cause is the cable, connector, optic or the far end's transmitter feeding this port, or a duplex mismatch. I confirm it is still climbing, check light levels on both ends, look at the far side, then clean or replace the patch lead, optic or port one at a time and clear counters to prove the fix.
Go deeper
What the counter means
Ethernet's 4-byte FCS is a CRC-32 over the frame. The receiver recomputes it, and on a mismatch it drops the frame and counts CRC and input errors. The sender's counters stay clean, so the side reporting CRC is the receiver of the bad direction.

Likely causes, most common first

Dirty or damaged fiber connector, bad patch lead, a bend that is too tight.

A failing or wrong optic, or low receive power. Check show interfaces transceiver detail.

Copper: damaged cable, poor termination, a run beyond 100 m, interference near power cabling.

Duplex mismatch: the full-duplex side counts CRC and runts.

A faulty port or NIC at the far end.

Cut-through switches
A cut-through switch starts forwarding before it has the FCS, so it passes a corrupt frame on and the next device counts the error. On cut-through fabrics, find the first port that counts the frame as received with errors. Some platforms count forwarded bad frames separately; Nexus calls them stomped CRC.

What I do

Confirm it is active: note the count and check again, or clear counters and watch.

Check transceiver light levels on both ends of a fiber link.

Check the far end's counters for the reverse direction.

Change one thing at a time: clean and reseat, swap the patch lead, swap the optic, move ports.

Clear counters and confirm CRC stays at zero under real load.

A healthy modern link runs with essentially zero CRC errors. Any steady increase is a fault.
They will ask next
The CRC count is high but not increasing. Do you care?
Check when the counters were last cleared. A historic count from a lead replaced months ago does not matter. Clear the counters and look for new errors.

What do CRC errors do to TCP traffic?
The damaged frame is dropped, so TCP sees loss, retransmits and cuts its sending rate. Even 0.1 percent loss noticeably lowers throughput on a long-RTT path.
Traps
Looking for the fault on the sending side's counters. The receiver counts CRC.
Replacing the optic, the lead and the port at once, so nobody knows which part was bad.
How did you do?Book 2.4 Error Detection Own page

Which Wireshark filters do you use most, and how is a capture filter different from a display filter?

Say this first

A capture filter uses BPF syntax, like tcpdump, and decides what gets saved: smaller files, but whatever it excludes is gone. A display filter uses Wireshark's own syntax and only hides packets already captured. My everyday display filters are ip.addr == 10.10.0.25, tcp.port == 443, tcp.analysis.flags for retransmissions and window problems, tcp.flags.reset == 1, dns, icmp, and tcp.flags.syn == 1 && tcp.flags.ack == 0 to see connection attempts.

Go deeper

Two syntaxes

	Capture filter (BPF)	Display filter
Example	`host 10.10.0.25 and tcp port 443`	`ip.addr == 10.10.0.25 && tcp.port == 443`
Applies	while capturing	after capturing
Also used by	tcpdump, dumpcap	tshark with `-Y`

Display filters worth knowing by heart

Filter	Shows
`ip.src == 10.10.0.25 && ip.dst == 10.30.0.15`	one direction only
`tcp.analysis.flags`	anything flagged: retransmissions, dup ACKs, zero window, lost segments
`tcp.analysis.retransmission`	retransmissions only
`tcp.analysis.zero_window`	a receiver advertising a zero window
`tcp.flags.syn == 1 && tcp.flags.ack == 0`	new connection attempts
`tcp.flags.reset == 1`	resets
`tcp.stream eq 7`	one conversation; also Follow TCP Stream
`dns.flags.rcode != 0`	DNS errors such as NXDOMAIN and SERVFAIL
`icmp.type == 3`	destination unreachable, including fragmentation needed (code 4)
`arp.duplicate-address-detected`	two MAC addresses claiming one IP
`frame.time_delta_displayed > 1`	gaps of more than a second between displayed packets

Habits

Use Statistics, Conversations to find the heavy talkers, and Expert Information for a quick list of problems. Set the time display to "seconds since previous displayed packet" when hunting for delays.

They will ask next

How do you capture on a Linux server with no GUI?
tcpdump -ni eth0 -w /tmp/app.pcap host 10.30.0.15 and tcp port 443, then open the file in Wireshark. -n skips name lookups, and -w writes raw packets instead of text.
Where do you capture to prove who dropped a packet?
At both ends at the same time, with synced clocks. A packet that leaves A and never arrives at B was lost in between. Then move the capture points closer together: divide and conquer with captures.

Traps

A capture filter so tight that ARP, DNS and ICMP errors, the evidence you needed, were never saved.

How did you do?Book 5.2 Reliable Byte Stream (TCP)Own page

How do you find which switch port a device with a given IP address is plugged into?

Say this first

On the gateway, show ip arp <ip> gives the MAC address. On the switch, show mac address-table address <mac> gives the port. If that port is an uplink, show cdp neighbors <port> detail or LLDP names the next switch, and I repeat until the MAC appears on an access port.

Go deeper

Walking it

Output

DSW1# show ip arp 10.10.0.57
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.0.57              3   3c52.8211.4a07  ARPA   Vlan10

DSW1# show mac address-table address 3c52.8211.4a07
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    3c52.8211.4a07    DYNAMIC     Gi1/0/1
Total Mac Addresses for this criterion: 1

DSW1# show cdp neighbors GigabitEthernet1/0/1 detail
-------------------------
Device ID: ASW3
Entry address(es):
  IP address: 10.99.0.13
Platform: cisco C9200L-48P-4G,  Capabilities: Switch IGMP
Interface: GigabitEthernet1/0/1,  Port ID (outgoing port): GigabitEthernet1/1/1

ASW3# show mac address-table address 3c52.8211.4a07
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    3c52.8211.4a07    DYNAMIC     Gi1/0/17
Total Mac Addresses for this criterion: 1

The host is on ASW3 Gi1/0/17.

What else the table tells you

The same MAC jumping between ports: a VM moving, or a loop (look for MAC flapping logs).
Many MACs on one access port: an unmanaged switch, a phone with a PC behind it, or a hypervisor.

They will ask next

The ARP entry exists but the MAC is in no switch table. Why?
Cisco MAC table entries age out after 300 seconds by default, while ARP entries last 4 hours. A quiet host keeps its ARP entry but drops out of the MAC table. Ping it from the gateway to refresh the entry.

Traps

Stopping at an uplink port. A MAC learned on a trunk only tells you which way to walk.

How did you do?Book 3.2 Switched Ethernet Own page

Users say "the network is slow". How do you approach it?

Say this first

"Slow" needs a number and a scope first: which application, which users and sites, since when, and slow how (page loads, file copies, call quality, logins). Then I split it: one app slow for everyone points at the app or its server, one site or path points at a link, congestion or errors, and everything slow points at a shared service like DNS or the internet edge. I check utilization, errors and drops along the path, latency and loss with mtr, and then capture to separate network time from server time.

Go deeper

Turn the complaint into a measurement

What is slow, for whom, and when? Compare to a baseline: "the CRM normally loads in 1 second, now 6".
Every app or one? Every user or one site? Wired and wireless?
Constant, at certain times of day, or random?

Where the time usually goes

Symptom	Look at
Everything slow from one site	WAN or internet link utilization, drops, errors at that site
One app slow for everyone	the app server, its database, its load balancer
Only wireless users	channel utilization, retries, signal, AP load
Slow to start, then fine	DNS, authentication, proxy discovery, connection setup
Big transfers slow, small ones fine	window size and RTT, MTU, a policer, loss
Slow at 9 a.m. or during backups	congestion; check the graphs

Tools

Monitoring: interface utilization and drops, NetFlow or IPFIX for top talkers.
Path: mtr or pathping for latency and loss per hop.
show interfaces along the path: errors, output drops, rate against bandwidth.
A capture at the client: handshake time compared with time to the first response byte.

They will ask next

The WAN link is at 95 percent every morning. What do you do?
Find out what fills it with NetFlow, then decide: schedule or move bulk traffic, apply QoS so voice and interactive apps stay protected, or add bandwidth if the growth is real. QoS does not create capacity; it decides who waits.

Traps

Buying bandwidth before measuring. If the real cause is a slow server, a bigger circuit changes nothing.

How did you do?Own page

How do you tell a DNS problem from a network problem?

Say this first

I test by address and by name separately. If ping 198.51.100.10 works but the same test by name fails, the path is fine and resolution is broken. Then I run nslookup against the configured resolver and against another known-good one. If the other one answers, the configured resolver is the problem. If none answer, I check reachability to port 53. I also check that the name resolves to the right address, because a wrong answer looks exactly like a network outage.

Go deeper

The split

Test	If it works	If it fails
`ping 198.51.100.10`	the IP path is fine	a network problem: routing, filtering
`ping app.example.com`	resolution and path are fine	DNS, or the name points at a wrong address
`nslookup app.example.com`	shows the server used and the answer	timed out: server unreachable; NXDOMAIN: name does not exist there; SERVFAIL: the resolver could not get an answer

Common DNS faults

Clients point at a wrong or dead resolver, from a bad DHCP option 6 or a static setting.
A firewall allows UDP 53 but blocks TCP 53, so large answers fail. Resolvers retry over TCP when a UDP answer is truncated, and zone transfers always use TCP.
Split DNS: an internal name resolves to the external address, or VPN users get the wrong view.
Stale records after a migration, cached for the record's TTL.
Negative caching: an NXDOMAIN stays cached for the zone's negative TTL after a record was briefly missing.
Search suffixes: a short name like crm resolves only when the suffix list is right.

Host-side checks

Windows: ipconfig /all for the DNS servers in use, ipconfig /flushdns, Resolve-DnsName. Linux: resolvectl status with systemd-resolved, /etc/resolv.conf, and dig.

They will ask next

The name resolves and ping by name fails, but ping to the server's real IP works. What is wrong?
The name resolves to the wrong address: a stale record, a cached old answer, a hosts file entry, or split DNS returning the external address. Compare the answer with the expected address and check the TTL and the hosts file.

Traps

Saying DNS only uses UDP. Truncated answers retry over TCP 53, and blocking it breaks large responses.

How did you do?Book 9.3 Infrastructure Applications Own page

A PC can reach its gateway but not a server in another subnet. What do you check?
Say this first
Since the gateway answers, the PC's link, VLAN and local addressing work. Next I check the PC's mask and gateway (a wrong mask can make the server look local), the route to the server's subnet on the gateway, the route back from the server side to the PC's subnet, ACLs and firewalls on the way, NAT, and the server's own gateway and firewall. Traceroute from the PC shows where it stops, and a ping from the gateway sourced from the PC's subnet tests the return path.
Go deeper
Order of checks

PC settings. ipconfig /all: mask and gateway. A /16 typed instead of a /24 makes the PC ARP for the server directly instead of sending to the gateway.

Forward route. show ip route 10.30.0.15 on the gateway. Missing, or the wrong next hop?

Traceroute from the PC. The last hop that answers is where to look next.

Return route. ping 10.30.0.15 source Vlan10 on the gateway. If a plain ping works and the sourced one fails, the far side has no route back to the PC's subnet.

Filters. ACLs on the SVI and WAN interfaces (show ip interface Vlan10 | include access list), firewall policies and NAT rules.

Server side. Its gateway, its host firewall, and on multi-homed servers, which NIC replies leave from.

A classic
A new subnet works locally, but nobody added it to the routing protocol or the firewall. Traffic leaves on the default route, and the far side has no route back.
They will ask next
How does a wrong subnet mask on the PC break remote access?
The PC uses its mask to decide what is local. With a mask that is too wide, it believes the server is on its own subnet and sends ARP requests for it instead of using the gateway. Nothing answers, unless proxy ARP on the router hides the mistake.
Traps
Testing only the forward route. Missing return routes are just as common.
How did you do?Book 3.3 Internet (IP)Own page

What does good change control look like, and why does it matter to a network engineer?

Say this first

Most outages follow a change, so change control is how changes stay safe and traceable. A good change has a clear reason, the exact commands, an impact and risk statement, pre-checks and post-checks, a rollback plan with a trigger, a maintenance window, a peer review, and a communication plan. Afterwards the record says what was done and what was seen. When something breaks, the change log is the first place I look.

Go deeper

What a change record contains

Part	Example
Reason	add VLAN 30 for new cameras
Scope and impact	DSW1, DSW2, ASW1 to ASW4; no outage expected
Exact steps	the commands, per device, in order
Pre-checks	`show interfaces trunk`, `show spanning-tree root`, saved to a file
Post-checks	the same commands compared, plus a test from a camera
Rollback	the exact undo commands and the trigger, such as any uplink down for 2 minutes
Window and approval	Saturday 22:00 to 23:00, approved, peer reviewed
Communication	service desk and stakeholders told before and after

Practices that prevent outages

Peer review of the actual commands, not a summary.
Change types: standard (pre-approved, low risk, repeatable), normal (reviewed), emergency (reviewed afterwards).
Safety nets for remote work: reload in, an archived config with configure replace, Junos commit confirmed.
One change per window where possible, so cause and effect stay clear.
Automatic config backups and a diff after every change.

Emergency changes

During an outage you may change things without prior approval. You still record them, review them afterwards, and bring the documented state back in line with the running config.

They will ask next

How does reload in protect a remote change?
It schedules a reload, for example in 15 minutes. If the change cuts you off, the device reboots into the saved startup config, which does not contain your unsaved change. If the change works, you run reload cancel and then save. It costs a full reboot if it fires, so configure replace with an archived config is gentler where the platform supports it.
What makes a good rollback plan?
Exact commands, tested where possible, a clear trigger that says when to use it, and a deadline for the decision, so nobody is still debugging at 2 a.m. after the window has closed.

Traps

Treating change control as paperwork. Its value is in the pre-checks, the rollback and the record you use when something breaks.

How did you do?Own page

Working17

What is the follow-the-path method, and where do comparing with a working setup and swapping parts fit in?
Say this first
Follow the path means tracing the real route a packet takes, device by device, and checking at each hop that it arrives and leaves correctly: ARP, MAC table, route, ACL, NAT, firewall policy. Then I do the same for the return path, which can differ. It finds the one bad rule or route that a layer-by-layer check misses. Comparing with a working site or port shows the difference fast. Swapping a cable or optic is quick when the evidence points at hardware, but I record what I swapped and the counters before and after.
Go deeper
Follow the path, both directions
For a host in VLAN 10 reaching a server at another site:

Host: IP, mask, gateway, DNS, ARP entry for the gateway.

Access switch: MAC learned on the right port and VLAN, trunk carries the VLAN.

Gateway: ARP entry for the host, route to the server subnet.

Each router: show ip route 10.30.0.15, and show ip cef exact-route 10.10.0.25 10.30.0.15 when ECMP is involved. ACLs and NAT on the in and out interfaces.

Firewall: which policy matched, NAT, the session table.

Reverse it. The reply may take another router or another firewall.

Compare with a working one
Diff the broken device's config against a working peer or its last backup. Compare behavior too: a working host's ARP entry, route and capture next to the broken host's.

Swap components
Move the user to another port, replace the patch lead or the optic. Fast, and fine when the counters already point at hardware. The risk: the fault vanishes and you cannot tell whether the part or the reseat fixed it. Note the counters before and after.
They will ask next
What does show ip cef exact-route tell you?
Which next hop and outgoing interface CEF uses for one source and destination pair. With ECMP, a broken path can hurt only some flows, and this shows which path a failing pair actually takes.
Traps
Checking only the forward path. Return traffic through a different firewall is a classic cause.
How did you do?Book 3.3 Internet (IP)Own page
What is an extended ping on a Cisco router, and when do you use it?
Say this first
Extended ping lets me set the source address or interface, the repeat count, datagram size, timeout, the DF bit, and a sweep of sizes. I use the source option to test the return route for a specific subnet, because a ping from the WAN interface can work while one sourced from the LAN fails. I use DF with a size sweep to find the path MTU, and a high repeat count to measure loss on a suspect link.
Go deeper
Why the source matters
A router sources its ping from the exit interface by default. The far end may have a route back to that WAN /30 but not to the user subnet. Sourcing from the LAN interface or a loopback tests the path that users actually use.

Cisco IOS
R1# ping 10.20.0.10 source GigabitEthernet0/0/1 repeat 100 R1# ping 10.20.0.10 source Loopback0 size 1500 df-bit
The interactive form
Type ping alone in privileged mode. It prompts for protocol, target, repeat count, datagram size and timeout, then "Extended commands". Answer y to get the source address or interface, type of service, DF bit, data pattern, and "Sweep range of sizes".

Finding the path MTU
Set DF and sweep from 1400 to 1500. Sizes above the path MTU come back as M (a router sent "fragmentation needed") or . (the message was filtered). Equivalent host tests:

Shell
ping -M do -s 1472 10.20.0.10 # Linux: 1472 data + 8 ICMP + 20 IP = 1500 ping -f -l 1472 10.20.0.10 # Windows
Measuring loss
ping 10.20.0.10 repeat 1000 timeout 1 gives a loss percentage worth quoting. Five pings do not.
They will ask next
The router can ping a server, but users still cannot reach it. What do you try?
Ping again sourced from the users' gateway interface. If that fails, the server side has no route back to the user subnet, or a firewall permits the router's WAN address but not the user range.

Why might ping 10.20.0.10 size 1500 df-bit fail on a path that looks healthy?
Cisco's size is the whole IP datagram, so 1500 should pass on a clean Ethernet path. If it fails, something in the path has a smaller MTU, such as a GRE or IPsec tunnel, or PPPoE at 1492.
Traps
Mixing up Cisco's size (the whole IP datagram) with Linux and Windows sizes (ICMP data only, so 1472 fills 1500).
How did you do?Book 3.3 Internet (IP)Own page

Why do some traceroute hops show asterisks, and does that mean packets are lost there?

Say this first

An asterisk means no reply came back for that probe before the timeout. It usually is not forwarding loss. Routers rate-limit ICMP generation, drop it with control plane policing, filter it, or have no route back to your source. If later hops and the destination answer normally, traffic passed through that hop fine. Loss is real only when it starts at a hop and continues to every later hop and the destination.

Go deeper

Common causes of `* * *` at one hop

The router rate-limits or deprioritizes ICMP it generates.
CoPP or an ACL drops packets punted to the CPU, which includes TTL-expired probes.
The router has no route back to your source, common inside provider cores and MPLS networks.
A firewall in the path drops the probes or the ICMP replies.

Reading the pattern

Pattern	Meaning
One hop all `*`, later hops fine	that router does not answer; ignore it
`*` mixed with times at one hop	ICMP rate limiting at that router
Every hop `*` from hop N to the end	a real break at or after hop N, or the rest of the path filters that probe type
Only the last hop `*`	the destination drops that probe type; try `-I` or `-T -p 443`

mtr and pathping

Both send many probes and show loss per hop. The same rule applies: loss at an intermediate hop that does not carry through to the destination is ICMP rate limiting at that router.

They will ask next

mtr shows 30 percent loss at hop 4 and 0 percent at the destination. Is there a problem?
Not at hop 4. That router forwards your traffic fine and just answers your probes slowly or rarely. Real loss carries through to the end, so the destination's figure is the one that counts.
Why does a firewall sometimes never appear in a traceroute?
Some firewalls do not decrement TTL by default. A Cisco ASA, for example, stays invisible unless configured to decrement TTL. Others drop the probes or the ICMP replies.

Traps

Opening an ISP ticket for "loss at hop 5" when the destination shows none.

How did you do?Own page

What does a routing loop look like in traceroute, and how do you confirm it?
Say this first
The same two or more addresses repeat in a cycle until the trace hits its maximum TTL (30 by default) without reaching the destination. A ping shows "TTL expired in transit". I confirm it with show ip route <destination> on each router in the cycle: each points at the other. Common causes are two static routes pointing at each other, a summary with no Null0 discard route, or mutual redistribution at two points.
Go deeper
What it looks like
Output
R3# traceroute 172.16.9.10 Type escape sequence to abort. Tracing the route to 172.16.9.10 VRF info: (vrf in name/id, vrf out name/id) 1 10.0.34.4 1 msec 1 msec 1 msec 2 10.0.34.3 1 msec 1 msec 1 msec 3 10.0.34.4 2 msec 1 msec 2 msec 4 10.0.34.3 2 msec 2 msec 2 msec 5 10.0.34.4 2 msec 2 msec 2 msec
It carries on alternating until hop 30.

Confirm on both routers
Output
R3# show ip route 172.16.9.10 Routing entry for 0.0.0.0/0, supernet Known via "static", distance 1, metric 0, candidate default path Routing Descriptor Blocks: * 10.0.34.4 Route metric is 0, traffic share count is 1 R4# show ip route 172.16.9.10 Routing entry for 172.16.0.0/16 Known via "static", distance 1, metric 0 Routing Descriptor Blocks: * 10.0.34.3 Route metric is 0, traffic share count is 1
R4 sends the whole 172.16.0.0/16 to R3. R3 no longer has the specific subnet, so it follows its default route back to R4. Each packet bounces until its TTL runs out, and the link between them fills up.

Fixes

Correct the wrong route, or put back the missing one.

On the router that owns a block, add a discard route so unknown subnets inside it are dropped locally: ip route 172.16.0.0 255.255.0.0 Null0.

With redistribution, tag routes and filter them at the second redistribution point.
They will ask next
How is a layer 2 loop different?
Ethernet frames have no TTL, so a switching loop circulates broadcasts forever. You see MAC flapping logs, links at 100 percent and high CPU. traceroute does not show it, because the loop is inside one IP hop.

Can loops be temporary?
Yes. Micro-loops can last a second or so during convergence while routers update their tables at different times. A loop that stays is a configuration problem.
Traps
Calling a trace that ends in * * * a loop. A loop shows repeating addresses, not silence.
How did you do?Book 3.4 Routing Own page

What are runts and giants, and what usually causes them?

Say this first

A runt is a frame shorter than the Ethernet minimum of 64 bytes. A giant is longer than the port accepts: 1518 bytes untagged, or 1522 with one 802.1Q tag, on a standard 1500-byte port. Runts usually point at collisions, a duplex mismatch or a failing NIC. Giants usually mean an MTU mismatch: a neighbor sends jumbo frames or extra tags to a port that is not configured for them, and those frames are dropped.

Go deeper

Runts

The minimum frame is 64 bytes including the FCS, which is why a minimum payload is 46 bytes.
Causes: collision fragments on half-duplex links, a duplex mismatch where the half-duplex side aborts frames mid-transmission, or a failing NIC or transceiver.

Giants

The standard maximum is 1518 bytes: 1500 MTU, 14-byte header, 4-byte FCS. 1522 with an 802.1Q tag.
Causes: a server or neighbor sending 9000-byte jumbo frames to a 1500 port, extra encapsulation such as QinQ or MPLS labels without raising the MTU, or a faulty NIC.
Fix: make the MTU consistent across the whole layer 2 path. The command is platform-specific: a global system mtu on many Catalyst models, a per-interface mtu on NX-OS, EOS and Junos.

Reading them with other counters

Pattern	Likely cause
Runts and CRC on one side, late collisions on the other	duplex mismatch
Giants climbing on a port facing a server or a provider	MTU mismatch, jumbo frames or extra tags
Runts on a full-duplex link with no collisions anywhere	failing NIC or optic

They will ask next

Why is 64 bytes the minimum Ethernet frame?
Half-duplex Ethernet needed a sender to still be transmitting when a collision from the far end of a maximum-length segment came back, so it could detect it. At 10 Mb/s that slot time is 512 bit times, which is 64 bytes. The minimum stayed for compatibility.
Is a 1522-byte frame on a trunk a giant?
No. A switch expects the 4-byte tag on trunks and accepts 1522. Giants are frames larger than the port is set to take, and they are dropped.

Traps

Saying the maximum Ethernet frame is 1500 bytes. 1500 is the MTU, the payload; the frame is 1518 untagged.

How did you do?Own page

What is the difference between input errors, overruns, input queue drops and output drops?

Say this first

Input errors are frames that arrived damaged or malformed: CRC, runts, giants, framing. Overruns and ignored mean the frame was fine, but the receiving hardware or its buffers could not take it in time. Input queue drops on a router mean packets punted to the CPU arrived faster than it could process them. Output drops mean the egress queue was full: more traffic was aimed at the port than it could send, which is congestion, very often in short bursts.

Go deeper

Counter	Where	Points to
CRC, frame, runts, giants	receive, the frame is bad	cabling, optics, duplex, MTU
overrun	receive, the hardware FIFO filled before the frame moved to memory	the receiving device cannot keep up with line-rate bursts
ignored, no buffer	receive, no free buffer	bursts, buffer shortage
Input queue drops (third number in `Input queue`)	packets for the router's CPU	too much punted traffic: control plane, ARP, packets not switched in hardware
Total output drops	transmit, egress queue full	congestion: speed mismatch such as 10G into 1G, many-to-one, microbursts
output errors, underruns	transmit hardware	rare; hardware

Fault or capacity?

Output drops with no input errors anywhere on the path are capacity, not cabling. The answers are QoS (decide what gets dropped), more bandwidth, spreading load across links, or more buffer where the platform allows it.

Where the drops really are

show interfaces gives a total. Per-queue drops live in show policy-map interface on routers with a QoS policy, and in platform-specific queue commands on switches.

They will ask next

A 1G server port shows output drops, but its 5-minute rate is 150 Mb/s. How?
Microbursts. Several 10G uplinks can send to the same 1G port within a few milliseconds and overflow its buffer, while the 5-minute average stays low.
How do you reduce output drops without buying bandwidth?
Apply QoS so the drops fall on bulk traffic instead of voice and critical apps, give the queue more buffer where the platform supports it (on Catalyst 9000, qos queue-softmax-multiplier 1200 is a common change), or spread the load over more links.

Traps

Sending someone to replace a cable because of output drops. Output drops are a capacity or burst problem.

How did you do?Own page

What are the symptoms of a duplex mismatch, and which side shows which counters?
Say this first
The link stays up and ping looks fine, but throughput collapses under load. The half-duplex side stops when it receives while transmitting, so it counts collisions and late collisions. The full-duplex side transmits whenever it wants and receives the frames the other side aborted, so it counts CRC errors and runts. The usual cause is one side hard-set to 100/full and the other left on auto, which then falls back to half duplex.

Go deeper
Why it happens
Autonegotiation only works when both ends take part. A port hard-set to 100/full does not negotiate. The auto side can still sense the speed from the signal, but it cannot learn the duplex, so it falls back to half duplex.

Counters
Side Counters that climb
Half duplex collisions, late collisions
Full duplex CRC, runts, input errors
A late collision is one detected after the first 64 bytes. On a correct half-duplex segment it never happens, so late collisions mean a duplex mismatch or a cable beyond specification.

Symptoms
Light traffic such as ping works. Bulk transfers drop to a fraction of the link speed because every lost frame costs a TCP retransmission. When both ends run CDP, Cisco logs %CDP-4-DUPLEX_MISMATCH.

Fix
Set both ends to auto, which is the preferred choice, or hard-set both ends identically. Never one of each.
They will ask next
Can this happen on gigabit copper?
Rarely, because 1000BASE-T requires autonegotiation. It is mostly a 10 and 100 Mb/s problem, common with old printers, building systems and hard-set server NICs.
Traps
Hard-setting only your side to "fix" negotiation. Hard-setting one side is exactly what creates a mismatch.
How did you do?Own page

Side	Counters that climb
Half duplex	collisions, late collisions
Full duplex	CRC, runts, input errors

What is the difference between SPAN, RSPAN and ERSPAN, and when would you use each?

Say this first

All three copy traffic from source ports or VLANs to an analyzer. Local SPAN sends the copy to another port on the same switch. RSPAN carries the copy across layer 2 trunks in a dedicated remote-span VLAN to a port on another switch. ERSPAN wraps the copy in GRE and routes it over IP to a collector anywhere. I use SPAN when I can plug in at the switch, RSPAN inside one layer 2 campus, and ERSPAN across routed networks or into a central capture appliance.

Go deeper

	Local SPAN	RSPAN	ERSPAN
Copy goes to	a port on the same switch	a port on another switch	an IP address
Transport	inside the switch	a `remote-span` VLAN on trunks	GRE over IP
Needs	nothing extra	the RSPAN VLAN allowed on every trunk in between	IP reachability and MTU headroom
Typical use	quick capture in a closet	central analyzer in a campus	data center, remote sites, capture appliances

Cisco IOS

SW1(config)# monitor session 1 source interface GigabitEthernet1/0/5 both
SW1(config)# monitor session 1 destination interface GigabitEthernet1/0/48

Limits to know before you trust a capture

SPAN is best effort. Mirroring both directions of a busy 1G port into one 1G destination can exceed the destination's speed, and the extra copies are dropped silently. Gaps in the capture may be SPAN loss, not network loss.
Errored frames are usually not copied. A frame that fails CRC is dropped on ingress, so SPAN will not show physical-layer damage. Counters show it.
The destination port is no longer a normal port. By default it does not forward normal traffic.
Platforms limit the number of sessions and sometimes the source types.
A network TAP copies at the physical layer and does not drop under load, which makes it better evidence than SPAN for a dispute.

They will ask next

Your SPAN capture shows retransmissions but not the original segments. Is the network dropping them?
Not necessarily. Check output drops on the SPAN destination port, and capture at both ends of the flow if you can. SPAN may have dropped the copies.
Why mirror only rx when the source is a VLAN?
A frame entering one port in VLAN 10 and leaving another port in VLAN 10 is copied twice with both, once on ingress and once on egress. rx gives you each frame once.

Traps

Treating a SPAN capture as lossless proof.
Forgetting to allow the RSPAN VLAN on every trunk between source and destination.

How did you do?Own page

Wireshark shows lots of TCP retransmissions. How do you work out where the loss is?

Say this first

A retransmission means the sender believed a segment was lost. I check which side retransmits and what else Wireshark flags: duplicate ACKs, "previous segment not captured". Then I capture at both ends. If the original left the sender and never reached the receiver, the loss is in between, and I narrow it with capture points and interface counters along the path. If it arrived, the ACK was lost on the way back, or the capture point itself missed it.

Go deeper

Wireshark labels

Label	Meaning
`[TCP Retransmission]`	sent again after the retransmission timer expired
`[TCP Fast Retransmission]`	sent again after 3 duplicate ACKs, without waiting for the timer
`[TCP Dup ACK 21#3]`	the third duplicate of the ACK in frame 21: the receiver is missing a segment
`[TCP Previous segment not captured]`	a gap in sequence numbers, lost in the network or by the capture
`[TCP Spurious Retransmission]`	resent data the receiver had already acknowledged
`[TCP Out-Of-Order]`	arrived with a lower sequence number than expected

Locating the loss

A capture on the sender proves the original left. A capture on the receiver proves whether it arrived.
Match the same segment in both files by sequence number or IP ID.
Lost in between: walk the path. CRC errors on receiving interfaces, output drops on egress ports, policer drops in show policy-map interface.
Arrived but no ACK came back: look at the return path, which may be a different path.

Read the pattern

Pattern	Likely cause
Steady loss at any load	a link with errors
Bursts of loss at high throughput	congestion or a policer
Only full-size segments lost	MTU

Why a little loss hurts

Single-flow TCP throughput falls roughly with the square root of the loss rate (the Mathis estimate). On a 50 ms path, 1 percent loss can hold one flow to a few Mb/s whatever the link speed.

They will ask next

How do you tell congestion loss from a bad link?
Congestion loss comes in bursts when throughput is high and lines up with output or policer drops on some device. Link errors happen at any load and line up with CRC or input errors on a receiving port.
What is the difference between a retransmission and a fast retransmission?
A timer-based retransmission waits for the RTO, which RFC 6298 starts at 1 second and doubles on each timeout. A fast retransmission is triggered by 3 duplicate ACKs, usually within one round trip, and costs far less time.

Traps

Blaming the network for spurious retransmissions caused by a slow receiver or a capture point that dropped packets.

How did you do?Book 5.2 Reliable Byte Stream (TCP)Book 6.3 TCP Congestion Control Own page

What does a TCP zero window mean in a capture, and whose problem is it?
Say this first
Every ACK carries the receiver's window: how many more bytes it can buffer. A zero window means the receiving application is not reading from its socket fast enough, its buffer is full, and the sender must stop. The sender then sends window probes until the window opens again. It points at the receiving host or application (CPU, disk, a stalled thread), not at the network. Filter with tcp.analysis.zero_window.
Go deeper
What it looks like
A client at 10.10.0.25 copies a file to a server at 10.30.0.15 over SMB:

Output
No. Time Source Destination Protocol Info 2201 8.4410 10.30.0.15 10.10.0.25 TCP 445 > 50114 [ACK] Seq=1 Ack=5120001 Win=1460 Len=0 2202 8.4412 10.10.0.25 10.30.0.15 TCP [TCP Window Full] 50114 > 445 [ACK] Seq=5120001 Ack=1 Win=8192 Len=1460 2203 8.4790 10.30.0.15 10.10.0.25 TCP [TCP ZeroWindow] 445 > 50114 [ACK] Seq=1 Ack=5121461 Win=0 Len=0 2204 8.7795 10.10.0.25 10.30.0.15 TCP [TCP ZeroWindowProbe] 50114 > 445 [ACK] Seq=5121461 Ack=1 Win=8192 Len=1 2205 8.7797 10.30.0.15 10.10.0.25 TCP [TCP ZeroWindowProbeAck] [TCP ZeroWindow] 445 > 50114 [ACK] Seq=1 Ack=5121461 Win=0 Len=0 2206 9.3801 10.10.0.25 10.30.0.15 TCP [TCP ZeroWindowProbe] 50114 > 445 [ACK] Seq=5121461 Ack=1 Win=8192 Len=1 2207 9.3803 10.30.0.15 10.10.0.25 TCP [TCP ZeroWindowProbeAck] [TCP ZeroWindow] 445 > 50114 [ACK] Seq=1 Ack=5121461 Win=0 Len=0 2210 11.2050 10.30.0.15 10.10.0.25 TCP [TCP Window Update] 445 > 50114 [ACK] Seq=1 Ack=5121461 Win=262144 Len=0
The server answers each probe in 0.2 ms, so the network is fast. For almost 3 seconds the server simply will not take more data. The probe byte is not accepted while the window is zero, so the ACK number stays at 5121461. That stall is on the server: its application or disk is not keeping up.

Window Full and Zero Window
[TCP Window Full] marks the sender's segment that fills the receiver's advertised window. [TCP ZeroWindow] is the receiver advertising zero. Both point at the receiver as the bottleneck, while retransmissions point at the path.

The window scaling trap
The window field is 16 bits, so 65,535 at most. Window scaling, agreed in the SYN and SYN-ACK, multiplies it. A capture that starts mid-connection cannot know the scale factor and shows windows that look tiny. Capture from the handshake. A middlebox that strips the scale option caps each flow at 64 KB per round trip.
They will ask next
The client advertises a zero window during a download. What does that suggest?
The client host cannot keep up: a slow disk, antivirus scanning every write, high CPU, or a small receive buffer. The network delivers faster than the host consumes.

Is a small non-zero window a problem too?
It can be. Throughput is capped at window divided by round-trip time, so on a long path even a healthy receiver with a small window makes transfers slow.
Traps
Calling a zero window "network congestion". Congestion shows as loss and retransmissions, not as a full receive buffer.
How did you do?Book 5.2 Reliable Byte Stream (TCP)Own page

How do you prove whether a slowness problem is the network or the application?

Say this first

I split one transaction into its parts with a capture near the client. The TCP handshake (SYN to SYN-ACK) is almost pure network round trip. The time from the request to the first byte of the response, minus one round trip, is server think time. If the handshake takes 2 ms and the response 4 seconds, the network is not the problem. If the handshake is slow, or I see retransmissions and stalls in the data, the network is involved. I bring numbers, not opinions.

Go deeper

One request, timed

Output

t=0.000   client  SYN
t=0.002   server  SYN-ACK          <- network round trip about 2 ms
t=0.002   client  ACK
t=0.003   TLS handshake begins, finished by t=0.009
t=0.010   client  HTTP GET /report
t=0.012   server  ACK              <- request delivered
t=4.215   server  first response byte   <- about 4.2 s of server time

Wireshark helpers

tcp.analysis.ack_rtt shows how quickly segments were acknowledged.
For clear-text HTTP, the http.time field gives time from request to response.
Statistics, TCP Stream Graphs show steady flow versus stalls.

What points where

Evidence	Points to
Fast handshake, long wait for the response	the server or its backend
Retransmissions and duplicate ACKs	loss on the path
Zero window from the server	the server cannot consume data
Slow handshake that also shows in ping	latency or congestion on the path
Long gaps before the client's next request	the client

Timing from the command line

Shell

curl -o /dev/null -s -w 'dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total}\n' https://app.example.com/report

The values are cumulative from the start, so connect minus DNS time is roughly one round trip.

They will ask next

The app team insists it is the network. How do you handle it?
With a shared capture and plain numbers: "the handshake is 2 ms, the server took 4.2 seconds to answer this request". Offer to capture on the server side as well. Stay neutral; the goal is the fix.
Does a fast handshake rule out the network completely?
No. Large responses can still suffer from loss, MTU or a small window. Check the data phase for retransmissions and stalls, not just the setup.

Traps

Pinging the server and stopping there. ICMP tells you nothing about server think time.

How did you do?Book 5.2 Reliable Byte Stream (TCP)Own page

How do you troubleshoot an intermittent problem you cannot reproduce on demand?

Say this first

I stop guessing and set traps. First I look for a pattern: time of day, which users, which path, how long each event lasts. Then I make sure the next event leaves evidence: NTP and millisecond timestamps, syslog to a server, continuous probes such as IP SLA or mtr, short polling intervals on the suspect interfaces, and a ring-buffer capture or an EEM script that collects show output the moment it happens. Typical causes are a marginal link, flapping, peak-time congestion, a duplicate IP, STP topology changes and failing hardware.

Go deeper

Find the pattern

Same time every day: backups, scheduled jobs, peak-hour congestion.
Every few minutes or hours: a timer. Examples are ARP aging (4 hours on Cisco) against MAC aging (5 minutes) causing unicast flooding, VPN rekeys, and DHCP lease renewals.
Only some users or flows: one bad member of a port channel or ECMP group, a duplicate IP, wireless roaming.
After an event: an STP topology change. show spanning-tree detail shows the last change and the port it came from.

Make the next occurrence leave evidence

NTP on everything, service timestamps log datetime msec, logs to a syslog server.
Continuous probes: IP SLA with a track object, or mtr -rwc 600 10.30.0.15 from a host.
A ring-buffer capture that keeps the last 1 GB: tcpdump -i eth0 -w /var/tmp/cap.pcap -C 100 -W 10 host 10.30.0.15.
An EEM applet that saves show output when a track object goes down or a syslog pattern appears.

Typical culprits

Clue	Cause
CRC slowly climbing	marginal cable or optic
Bursts of `%LINK-3-UPDOWN`	flapping link, bad cable, NIC power saving
STP topology change count rising	a non-edge port going up and down somewhere
`%IP-4-DUPADDR` or ARP entries changing MAC	duplicate IP
Only at peak hours	congestion, output drops
Only some flows	a bad LAG or ECMP member

They will ask next

The user says it happens "randomly". How do you get a usable timeline?
Ask them to note the exact time and what they were doing the next few times, or give them a small script that logs a timestamped ping or curl. Then match those times with syslog, probes and graphs.
Why do STP topology changes cause short slowdowns?
A topology change makes switches age out or flush MAC entries, so traffic floods until the tables relearn. A host port without PortFast causes a topology change every time a PC boots or sleeps.

Traps

Changing things between occurrences without writing it down, so nobody knows what fixed it.

How did you do?Own page

What is asymmetric routing, and why does it break stateful firewalls?

Say this first

Asymmetric routing means a flow goes out one path and comes back another. Routers do not mind, since they forward each packet on its own. A stateful firewall does: it creates a session when it sees the SYN and only passes return traffic that matches. If the SYN goes through firewall A and the SYN-ACK returns through firewall B, B has no session and drops it. Symptoms: TCP connections time out while ping may work, and the firewall logs drops for packets with no connection.

Go deeper

How it happens

Two internet or WAN links where outbound follows one default route and inbound arrives on the other.
Two data centers, each with its own firewall, advertising the same prefix.
HSRP active on one switch while the firewall's route back points at the other.
A new link or metric change that moves only one direction.

Who cares about asymmetry

Device	Effect
Router	none; each packet is routed on its own
Stateful firewall	drops return or mid-flow packets of sessions it never saw
NAT device	return traffic arrives where no translation exists
IPS, some load balancers	see half a conversation and cannot track it

A Cisco ASA logs these as %ASA-6-106015: Deny TCP (no connection) messages.

Detecting it

Traceroute from each end toward the other, and compare.
show ip route for the destination on the outbound side and for the source on the return side.
Capture on both paths: the SYN on one, the SYN-ACK on the other.

Fixing it

Make the paths symmetric: BGP local preference and AS path prepending or MED, IGP metrics, an FHRP aligned with the firewall's route, or source NAT on the firewall so replies must come back to it. A firewall cluster that shares state across both paths also works. TCP state bypass gives up inspection for that traffic, so it is a documented exception, not a fix.

They will ask next

Is asymmetric routing always a problem?
No. The internet is full of it and routers handle it fine. It breaks only devices that keep per-flow state: firewalls, NAT, IPS and some load balancers.
How does source NAT fix asymmetry?
The firewall rewrites the source to its own address, so the server replies to an address that only routes back through that firewall. The cost is that the server no longer sees the real client IP.

Traps

Disabling stateful inspection globally to "fix" it.

How did you do?Book 3.3 Internet (IP)Own page

What are the classic symptoms of an MTU problem, and how do you prove it?
Say this first
Small things work and big things hang. Ping works, the TCP handshake completes and SSH logs in, but web pages stall partway, file copies freeze, or a VPN connects and then passes almost nothing. Full-size packets are being dropped somewhere, usually at a tunnel or PPPoE link, and the ICMP message that should tell the sender is filtered. I prove it with DF-bit pings of increasing size and fix it with a consistent MTU, TCP MSS clamping, and by letting ICMP fragmentation-needed through.
Go deeper
Symptoms

The handshake works, then the connection stalls at the first full-size segment, often the server's certificate during TLS.

Some websites load and others hang.

File copies, replication or backups stall over a VPN.

It works over one path and fails over another, such as a backup tunnel.

Prove it
Shell
ping -M do -s 1472 10.30.0.15 # 1500 bytes total, passes on a clean path ping -M do -s 1372 10.30.0.15 # 1400 bytes total
Windows: ping -f -l 1472. Cisco: ping 10.30.0.15 size 1500 df-bit. Bisect to the largest size that passes. Linux tracepath reports the path MTU hop by hop when ICMP is allowed. In a capture, look for the same full-size segment retransmitted again and again, with no ICMP type 3 code 4 coming back.

Common MTU budgets
Path IP MTU
Ethernet 1500
PPPoE 1492
GRE (24 bytes overhead) 1476
IPsec tunnel mode roughly 1400 to 1440, depending on cipher and NAT-T
GRE over IPsec commonly set to 1400
Fixes

Set the tunnel MTU and clamp TCP MSS: ip mtu 1400 and ip tcp adjust-mss 1360 on the tunnel.

Permit ICMP type 3 code 4, and ICMPv6 type 2 (packet too big), through every firewall.

Keep jumbo MTU consistent end to end in the data center.

MSS clamping fixes TCP only. Large UDP still needs a correct MTU or fragmentation.
They will ask next
How does MSS clamping fix it?
The router rewrites the MSS option in SYN packets passing through, so both ends agree to send segments that fit the tunnel. No oversized packets are created, so none are dropped.

Why 1360 with an IP MTU of 1400?
1400 minus 20 bytes of IPv4 header minus 20 bytes of TCP header. For IPv6 subtract 60, giving 1340.
Traps
Ruling out MTU because ping works. Default pings are tiny.
How did you do?Book 3.3 Internet (IP)Book 5.2 Reliable Byte Stream (TCP)Own page

Path	IP MTU
Ethernet	1500
PPPoE	1492
GRE (24 bytes overhead)	1476
IPsec tunnel mode	roughly 1400 to 1440, depending on cipher and NAT-T
GRE over IPsec	commonly set to 1400

Walk me through troubleshooting name resolution with dig and nslookup.

Say this first

I query the resolver the client uses, then a different resolver, then the authoritative server. dig name @resolver shows the status (NOERROR, NXDOMAIN, SERVFAIL, REFUSED), the answer, its remaining TTL and which server replied. dig +trace name walks from the root down, which shows where a delegation breaks. dig name @authoritative +norecurse shows the source of truth. Comparing the three tells me whether the fault is the resolver, a cache, or the zone itself.

Go deeper

A healthy answer

Output

$ dig app.example.com @10.0.0.53

; <<>> DiG 9.18.28 <<>> app.example.com @10.0.0.53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41872
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;app.example.com.               IN      A

;; ANSWER SECTION:
app.example.com.        247     IN      A       203.0.113.40

;; Query time: 3 msec
;; SERVER: 10.0.0.53#53(10.0.0.53) (UDP)
;; MSG SIZE  rcvd: 60

Read the status, the flags (aa means an authoritative answer, ra means recursion is available), the TTL (a cached answer counts down), and the SERVER line.

Statuses

Status	Meaning
NOERROR with an answer	resolved
NOERROR with ANSWER: 0	the name exists, but not with that record type
NXDOMAIN	the name does not exist
SERVFAIL	the resolver could not get an answer: upstream unreachable, broken delegation, DNSSEC failure
REFUSED	the server will not answer you, often recursion not allowed for your source
`connection timed out; no servers could be reached`	the server is down or port 53 is blocked

Useful options

dig +trace app.example.com follows the delegation from the root and shows where it breaks.
dig app.example.com @ns1.example.com +norecurse asks the authoritative server directly.
dig -x 203.0.113.40 for the PTR record, dig example.com MX, dig example.com SOA.
dig +tcp app.example.com tests DNS over TCP 53.

nslookup equivalents

nslookup app.example.com 10.0.0.53 and nslookup -type=mx example.com. "Non-authoritative answer" only means the answer came from a resolver's cache, not from the zone's own server.

They will ask next

nslookup prints "Server UnKnown". Is DNS broken?
No. nslookup does a reverse lookup on the DNS server's own address to show its name, and "UnKnown" only means that address has no PTR record. The answers can still be correct.
Why does a record change not show up for users?
Resolvers keep the old answer until its TTL expires, and clients and browsers cache too. Lower the TTL well before a planned change, then flush the caches you control.

Traps

Reading "Non-authoritative answer" as an error.

How did you do?Book 9.3 Infrastructure Applications Own page

How do you find and confirm a duplicate IP address?
Say this first
The symptom is intermittent: a host works, drops, then works again, because other devices' ARP caches flip between two MAC addresses. To confirm, I watch show ip arp <ip> on the gateway change MAC, look for the Windows address conflict warning, or use arping -D or the Wireshark filter arp.duplicate-address-detected. If someone takes the router's own address, IOS logs %IP-4-DUPADDR. I trace both MACs to switch ports, find the device that should not have the address, and prevent repeats with DHCP exclusions, IPAM, and DHCP snooping with DAI and IP Source Guard.
Go deeper
Confirm it
Output
DSW1# show ip arp 10.20.0.40 Protocol Address Age (min) Hardware Addr Type Interface Internet 10.20.0.40 0 0c4a.21d3.9e10 ARPA Vlan20 DSW1# clear ip arp 10.20.0.40 DSW1# show ip arp 10.20.0.40 Protocol Address Age (min) Hardware Addr Type Interface Internet 10.20.0.40 0 b8ca.3a5f.0c22 ARPA Vlan20
Two MAC addresses answer for one IP. Trace each one with show mac address-table address and CDP or LLDP until you reach two access ports. On Linux, arping -D -I eth0 10.20.0.40 gets a reply if another host holds the address.

When the conflict is with the router's own interface address, the log names the culprit: %IP-4-DUPADDR: Duplicate address 10.20.0.1 on Vlan20, sourced by b8ca.3a5f.0c22.

Common causes

A static IP set on a device inside the DHCP pool: printers, cameras, a test laptop.

A cloned VM booted with the same static address.

Someone used the gateway address as a host address.

A rogue DHCP server handing out overlapping addresses.

Prevent it

DHCP exclusions for static ranges, and an IPAM record for every static address.

DHCP snooping with Dynamic ARP Inspection and IP Source Guard on access ports, so only leased or statically bound addresses are accepted.

Conflict detection on the DHCP server. Cisco IOS DHCP pings an address twice before offering it by default; on Windows DHCP it is a setting that is off by default.
They will ask next
Does a duplicate IP always show up in the router's log?
No. %IP-4-DUPADDR fires only when a device claims one of the router's own addresses. Two hosts fighting over a client address show up as an ARP entry that keeps changing MAC, or in a capture.

What role does gratuitous ARP play?
A host sends a gratuitous ARP when its interface comes up, to announce its address and detect conflicts. It also updates everyone's ARP caches, which is exactly why the entry flips between two devices.
Traps
Giving the victim a new address and never finding the device that took the old one.
How did you do?Book 3.3 Internet (IP)Own page
How do you write a root cause analysis after an outage?
Say this first
A good RCA is factual and blameless. It has a short summary, the impact (who, what, how long), a timeline from detection to recovery taken from logs, the root cause and contributing factors, how it was detected and fixed, and corrective actions with owners and due dates. I keep asking why until I reach a cause we can fix in the process or design, not just "an engineer made a typo". The purpose is to stop it happening again.
Go deeper
Structure

Summary. Two or three sentences a manager can repeat.

Impact. Services, users, sites, duration, business effect.

Timeline. Times from logs, not memory: first symptom, detection, escalation, mitigation, recovery.

Root cause. The technical cause, and why it was possible.

Contributing factors. Monitoring that did not alert, a skipped review, an undocumented dependency.

What went well. A fast rollback, clear communication.

Corrective actions. Specific, each with an owner and a date.

Five whys, worked

The branch lost all routes. Why? The OSPF adjacency to the hub went down.

Why? Hellos were dropped by a new inbound ACL on the WAN interface.

Why? The ACL was written to block one host and ended in the implicit deny.

Why was it not caught? No peer review and no post-check of OSPF neighbors.

Why? The change was filed as a standard change, which skips review.

Corrective actions: an ACL template that permits routing protocols explicitly, mandatory OSPF neighbor post-checks for WAN ACL changes, and ACL changes reclassified as normal changes.

Blameless is not the same as no accountability
People act on the information and tools they have. Asking how the process let the mistake through leads to fixes. Naming a person teaches everyone to hide mistakes.
They will ask next
What if you cannot find the root cause?
Say so. Record the evidence, the hypotheses ruled out, and what you are adding to catch it next time, such as more logging, probes or a standing capture. A wrong root cause is worse than an honest "not yet known".

How is the root cause different from the trigger?
The trigger is the event that set it off, such as a fiber cut. The root cause is why that event caused an outage, for example both "diverse" circuits running in the same duct. Fixing only the trigger leaves the weakness in place.
Traps
Writing "human error" as the root cause. That is where the investigation starts, not where it ends.
How did you do?Own page

Expert7

Why can a later hop in a traceroute show lower latency than an earlier one?
Say this first
Each hop's time measures something different: the round trip to that router plus the time its control plane took to build the ICMP reply. Transit traffic is forwarded in hardware, but a TTL-expired probe is punted to the CPU and answered at low priority, so a busy router looks slow while the hops behind it look fast. Each router also sends its reply back by its own routing, which may be a different path. Only an increase that persists to every later hop and the destination is real latency.
Go deeper
The reasons

ICMP generation is slow path. The probe is punted to the CPU, which builds the time exceeded at low priority and often behind CoPP. That delay is added to that hop's line only.

Different return paths. Hop 5 may reply over a congested link while hop 6 replies over a clean one.

MPLS cores. A P router that expires the TTL often has no route to your source. Commonly it sends the time exceeded onward along the label switched path to the egress PE, which routes it back. Every core hop then shows about the same time: the trip to the far edge and back.

Real latency looks different
A jump that stays is distance or congestion. Light in fiber covers about 200 km per millisecond, so each 100 km of fiber adds about 1 ms of round trip. A step from 8 ms to 80 ms that holds for every later hop is usually an ocean crossing, not a fault.

How to read it
Judge by the destination's RTT and the trend across hops. A hop whose high time is not carried by the hops after it is a slow responder, not a slow link.
They will ask next
How would you measure the latency users actually feel to a service?
Measure with the application's own protocol: the TCP handshake time in a capture, curl -w timing, or a TCP-based probe. ICMP to routers on the way is a weak proxy.
Traps
Blaming a router because its own hop shows 200 ms while the destination answers in 30 ms.
How did you do?Own page
A single file copy over a 1 Gb/s WAN with 80 ms of latency only reaches about 6.5 Mb/s. Why?
Say this first
One TCP flow can send at most one window of data per round trip. To fill 1 Gb/s at 80 ms, the window has to hold the bandwidth delay product: 1 Gb/s x 0.08 s = 80 Mb, which is 10 MB. If window scaling is off or stripped, the window stops at 65,535 bytes, and 65,535 x 8 / 0.08 s is about 6.5 Mb/s. The link is fine; the flow is window limited. The fixes are window scaling with bigger socket buffers, parallel streams, or WAN optimization.
Go deeper
The formula
Maximum throughput of one flow = window / RTT. The window needed to fill a link = bandwidth x RTT.

Window RTT Maximum per flow
64 KB 80 ms about 6.5 Mb/s
64 KB 10 ms about 52 Mb/s
4 MB 80 ms about 420 Mb/s
10 MB 80 ms about 1 Gb/s
What keeps the window small

Window scaling disabled, or stripped by a middlebox. The scale option is only sent in the SYN and SYN-ACK, so a device that rewrites those can cap every flow.

Host buffer limits: the Linux net.ipv4.tcp_rmem maximum, or Windows receive window auto-tuning set to disabled.

Application buffers, for example an old transfer tool with a fixed buffer size.

Loss: after a loss, congestion control cuts the congestion window, and on a long path it takes many round trips to grow back. CUBIC and BBR cope better than Reno.

Proving it

Check the handshake: Wireshark shows WS= in the SYN and SYN-ACK info when scaling is offered.

Statistics, TCP Stream Graphs, Window Scaling: bytes in flight sitting flat at the receive window means the window is the limit.

iperf3 -c 10.30.0.15 -P 8 with parallel streams shows whether the capacity exists.
They will ask next
Why do parallel streams help?
Each flow has its own window, so eight flows carry eight windows per round trip. Many backup and transfer tools do this. It also limits the damage from one loss to one flow.

Would a 10 Gb/s circuit help?
No. A window-limited flow gets the same throughput on 1 or 10 Gb/s. Only a bigger window or a shorter round trip helps.
Traps
Upgrading the circuit for a transfer that is limited by its window.
How did you do?Book 1.5 Performance Book 5.2 Reliable Byte Stream (TCP)Own page
What is a PMTUD black hole, and how do you find and fix one?
Say this first
Path MTU discovery relies on routers sending ICMP "fragmentation needed" (IPv4 type 3 code 4) or "packet too big" (ICMPv6 type 2) back to the sender when a packet is too large to forward without fragmenting. If a firewall drops that ICMP, or the router does not send it, the sender never learns, keeps sending full-size packets, and they vanish. Connections hang after the handshake. I find it by bisecting with DF pings and fix it by permitting that ICMP, clamping MSS, or setting the right MTU where the tunnel starts.
Go deeper
How PMTUD works

TCP stacks send segments with DF set.

A router whose next link is too small drops the packet and returns ICMP type 3 code 4, including the next-hop MTU (RFC 1191).

The sender lowers its path MTU for that destination and resends smaller segments.

IPv6 routers never fragment. Every too-big packet is dropped with ICMPv6 type 2 (RFC 8201), so filtering that message breaks IPv6 paths outright.

Ways it breaks

A firewall blocks all ICMP "for security".

The router that drops the packet has no ip unreachables on that interface, or rate-limits ICMP hard.

The ICMP comes from a private address inside a provider, and the sender's edge filters it.

ECMP or an anycast load balancer delivers the ICMP to a different server than the one holding the connection.

Finding it

DF ping bisect from the client side.

A capture at the sender shows full-size segments retransmitted with no ICMP arriving. A capture at the tunnel router shows whether the ICMP is generated and leaves.

Fixing it

Permit ICMP type 3 code 4 and ICMPv6 type 2 everywhere. Rate-limit if you must; never block.

Clamp TCP MSS at the tunnel: ip tcp adjust-mss 1360.

Hosts can use packetization layer PMTUD (RFC 4821), which probes using TCP itself. On Linux, net.ipv4.tcp_mtu_probing=1 turns it on when a black hole is detected.
They will ask next
Why not just clear the DF bit on the router?
Then the router fragments, which works but costs CPU on the router and the receiver, and one lost fragment loses the whole packet. It is a workaround for a broken path, and it does nothing for IPv6.

Does MSS clamping help UDP?
No. MSS is a TCP option. Large UDP, such as some VPN protocols or big DNS answers, still depends on a correct MTU or fragmentation.
Traps
Blocking all ICMP as a security measure. It breaks PMTUD, and IPv6 cannot work properly without ICMPv6.
How did you do?Book 3.3 Internet (IP)Book 4.2 IP Version 6 Own page
An interface averages 20 percent utilization but its output drops keep rising. Explain it and how you would prove it.
Say this first
Averages hide bursts. A 30-second or 5-minute counter averages millions of packets, while a microburst lasts milliseconds. When several fast ports send to one egress port at once, or a 10G port sends toward a 1G port, the egress buffer fills within a few milliseconds and drops, while the average stays low. I prove it with fine-grained data: platform buffer or microburst monitoring, per-queue drop counters, or a timestamped capture showing line-rate trains. Fixes are QoS and buffer tuning, removing the speed mismatch, or spreading the load.
Go deeper
The arithmetic
A 1G port drains 125 KB per millisecond. A 10G port can deliver 1.25 MB toward it in that same millisecond. If the queue's buffer is a few hundred KB, a burst of a couple of milliseconds overflows it. Over 30 seconds, the port still averages a small number.

Typical sources

Speed mismatch: a 10G or 25G server sending to a 1G client.

Incast: many servers answering one request at the same moment, as with storage or distributed databases.

Backups and replication jobs that all start on the hour.

Proving it

Platform tools built for this, such as Arista LANZ or micro-burst monitoring on Nexus 9000, or streaming telemetry at short intervals.

Per-queue drop counters, which show which class is dropping.

A capture with precise timestamps at the receiver: trains of back-to-back packets, then loss and retransmissions.

Fixes

QoS, so bursts land in a queue with room and drops fall on bulk traffic.

More buffer per queue where supported (on Catalyst 9000, qos queue-softmax-multiplier 1200).

Remove the speed mismatch by upgrading the egress port.

Spread incast across more links or pace it in the application.
They will ask next
Why does a deeper buffer not always help?
Deep buffers absorb bursts but add queuing delay, which hurts latency-sensitive traffic. Storage and bulk transfer like deep buffers; voice and trading prefer short queues.
Traps
Concluding "the link is only 20 percent used, so it cannot be congestion".
How did you do?Book 6.2 Queuing Disciplines Own page
Some flows between two sites fail or crawl while others work fine. What do you suspect?
Say this first
A fault on one member of a set of parallel paths. ECMP and port channels hash each flow onto one link by its addresses and often its ports, so a bad member only hurts the flows that land on it: one user's share fails while a colleague's works, or a connection works on retry because a new source port hashes differently. I suspect a member with errors, a member up but not forwarding, an MTU or ACL difference on one path, or one next hop missing a route. I find which path a failing flow takes and test that member.
Go deeper
Why only some flows
Load sharing is per flow, not per packet, to keep packets in order. The hash uses source and destination IP and often layer 4 ports. A failing flow keeps failing until its ports change.

Finding the bad member

Which path does a failing pair take? show ip cef exact-route 10.10.0.25 10.30.0.15 on IOS for ECMP. For port channels, show etherchannel load-balance shows the hash inputs, and most platforms have a command that computes the member for a given flow.

Per-member health: show interfaces on each member, show etherchannel summary for flags such as suspended or standalone.

Compare members: light levels, MTU, ACLs, and allowed VLANs on channel members configured by hand.

Unidirectional fiber: a link that transmits but cannot receive in one direction. UDLD detects it.

In a maintenance window, shut the suspect member and see whether the failing flows recover.

Hash polarization
If several layers of routers use the same hash inputs and algorithm, flows that arrive on one branch all hash the same way again at the next layer, leaving links idle. Different hash inputs or seeds per layer fix it. That is uneven load, not failure, but it turns up in the same investigation.
They will ask next
Why not balance per packet for a perfect spread?
Packets of one flow would take paths with different delays and arrive out of order. TCP treats reordering as possible loss, sends duplicate ACKs and slows down.
Traps
Testing with a ping from the router, which may hash onto a different member than the failing application flow.
How did you do?Own page

Window	RTT	Maximum per flow
64 KB	80 ms	about 6.5 Mb/s
64 KB	10 ms	about 52 Mb/s
4 MB	80 ms	about 420 Mb/s
10 MB	80 ms	about 1 Gb/s

A router's CPU is at 99 percent. How do you find out why?

Say this first

show processes cpu sorted first, and show processes cpu history to see whether it is constant or spiky. On classic IOS the header splits the 5-second figure, for example 99%/85%: the second number is time spent at interrupt level, so a high value means packets are being switched in software. If one process tops the list, its name points the way: IP Input means packets punted for process switching, ARP Input an ARP storm, BGP Router routing churn, SNMP ENGINE heavy polling. Then I find the punted traffic and stop it.

Go deeper

Output

R1# show processes cpu sorted
CPU utilization for five seconds: 99%/85%; one minute: 96%; five minutes: 94%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  89     9873211    41920334        235 12.47% 10.31%  9.88%   0 IP Input

85 percent at interrupt level plus IP Input near the top: a lot of traffic is not being forwarded in the fast path.

Reading the clues

Clue	Likely cause	Next step
`IP Input` high	packets punted: TTL expiring in a loop, IP options, traffic to the router itself, ACL lines with `log`	`show interfaces stats` for process-switched counts, `show ip traffic`
`ARP Input` high	ARP storm or scan, or a static route pointing out an Ethernet interface with no next hop	`show ip arp summary`, broadcast counters
`BGP Router`, OSPF processes	churn from a flapping peer or link	`show ip bgp summary`, `show ip ospf statistics`
`SNMP ENGINE`	a poller walking a big table too often	find the poller, slow it down
`Exec`, `Virtual Exec`	heavy commands, or `debug` left running	`show users`, `undebug all`

IOS-XE platforms

On ISR 4000, ASR 1000 and Catalyst 9000, forwarding happens in a separate data plane (the QFP on the routers, the UADP ASIC on the switches), and show processes cpu sorted covers the IOSd control plane. show processes cpu platform sorted shows the underlying Linux processes. High CPU there usually means punted traffic, and the platform punt statistics show which cause is hot.

Protect the control plane

CoPP to cap punted traffic per class.
Remove log from busy ACL lines, fix loops, and point static routes at a next hop, not just an interface.
Never leave a debug running on a busy production device.

They will ask next

Does high CPU on a switch affect forwarding?
Hardware-switched traffic usually keeps flowing, but anything that needs the CPU suffers: routing hellos, BPDUs, LACP, ARP and management access. Lost BPDUs or hellos then cause loops or adjacency flaps, so high CPU can grow into an outage.

Traps

Running debug ip packet on a router that is already at 99 percent CPU.

How did you do?Own page

You are the senior engineer on a major outage. How do you run the incident?

Say this first

Restore service first and find the root cause later. I make sure there is one incident lead, one channel and a fixed update rhythm, so engineers work while someone else handles stakeholders. I scope the impact, check what changed recently, and go for the fastest safe mitigation: roll back, fail over, or route around the fault. Every action is timestamped in the incident log as it happens. When service is back, I confirm it with users and monitoring, then hand over to the RCA.

Go deeper

Roles

Role	Job
Incident lead	decides, sets priorities, owns the timeline
Communications	updates stakeholders on a fixed schedule, for example every 30 minutes
Engineers	investigate and carry out agreed actions
Scribe	records actions and times; in a small team the lead may do this

One person can hold two roles in a small team, but nobody should debug and run communications at once.

The first 15 minutes

Declare the incident and its severity.
Scope it: which services, sites and users, and is it growing?
What changed? The change log, recent deployments, provider notices.
Pick the quickest safe mitigation. A rollback you understand beats a fix you are guessing at.
Give each action a named owner and a time to report back.

Rules that stop it getting worse

One change at a time, announced in the channel before it is made.
Preserve evidence when you can: save show output and logs before a reload.
Time-box each hypothesis. If it produces no evidence in 15 minutes, move on.
Open vendor and provider cases early. They cost little and take time to staff.

Closing it

Confirm recovery from the user side, watch for recurrence, send the all-clear, and book the RCA while memories are fresh.

They will ask next

A senior manager joins the bridge and starts directing engineers. What do you do?
Politely steer them to the incident lead or the communications person, give them the current status and the time of the next update, and keep the engineers focused. Clear structure protects the people doing the fix.
When do you reload a device during an outage?
When the evidence points at a device-level fault and a reload is the fastest safe way to restore service, after capturing show output, logs and any crashinfo. A reload without evidence may cure the symptom and destroy the only clue.

Traps

Everyone debugging in parallel and changing things without telling each other.

How did you do?Own page

Core16

Say this first

Go deeper

The loop

Picking a method

What a senior answer adds

They will ask next

Traps

Say this first

Go deeper

Bottom-up

Top-down

Divide and conquer

The limit of a passing ping

They will ask next

Traps

Say this first

Go deeper

The questions and what each answer points to

Then pin one example

They will ask next

Traps

Say this first

Go deeper

They will ask next

Traps

Say this first

Go deeper

What travels

Defaults worth knowing

Cisco ping characters

Why the first ping from a router often shows .!!!!

Reading failures

They will ask next

Traps

Say this first

Go deeper

Carry on up the stack

Read the TCP result

They will ask next

Traps

Say this first

Go deeper

Step by step

Variants

Reading a line

They will ask next

Traps

Say this first

Go deeper

Habits that save time

Other platforms

They will ask next

Traps

Say this first

Go deeper

A port with a problem

Field by field

How I use it

They will ask next

Traps

Say this first

Go deeper

What the counter means

Likely causes, most common first

Cut-through switches

What I do

They will ask next

Traps

Say this first

Go deeper

Two syntaxes

Display filters worth knowing by heart

Habits

They will ask next

Traps

Say this first

Go deeper

Walking it

What else the table tells you

Why the first ping from a router often shows `.!!!!`

Common causes of `* * *` at one hop