How do you write a root cause analysis after an outage?
Show the answer
Say this first
A good RCA is factual and blameless. It has a short summary, the impact (who, what, how long), a timeline from detection to recovery taken from logs, the root cause and contributing factors, how it was detected and fixed, and corrective actions with owners and due dates. I keep asking why until I reach a cause we can fix in the process or design, not just "an engineer made a typo". The purpose is to stop it happening again.
Go deeper
Structure
- Summary. Two or three sentences a manager can repeat.
- Impact. Services, users, sites, duration, business effect.
- Timeline. Times from logs, not memory: first symptom, detection, escalation, mitigation, recovery.
- Root cause. The technical cause, and why it was possible.
- Contributing factors. Monitoring that did not alert, a skipped review, an undocumented dependency.
- What went well. A fast rollback, clear communication.
- Corrective actions. Specific, each with an owner and a date.
Five whys, worked
- The branch lost all routes. Why? The OSPF adjacency to the hub went down.
- Why? Hellos were dropped by a new inbound ACL on the WAN interface.
- Why? The ACL was written to block one host and ended in the implicit deny.
- Why was it not caught? No peer review and no post-check of OSPF neighbors.
- Why? The change was filed as a standard change, which skips review.
Corrective actions: an ACL template that permits routing protocols explicitly, mandatory OSPF neighbor post-checks for WAN ACL changes, and ACL changes reclassified as normal changes.
Blameless is not the same as no accountability
People act on the information and tools they have. Asking how the process let the mistake through leads to fixes. Naming a person teaches everyone to hide mistakes.
They will ask next
What if you cannot find the root cause?
Say so. Record the evidence, the hypotheses ruled out, and what you are adding to catch it next time, such as more logging, probes or a standing capture. A wrong root cause is worse than an honest "not yet known".
How is the root cause different from the trigger?
The trigger is the event that set it off, such as a fiber cut. The root cause is why that event caused an outage, for example both "diverse" circuits running in the same duct. Fixing only the trigger leaves the weakness in place.
Traps
- Writing "human error" as the root cause. That is where the investigation starts, not where it ends.