You are the senior engineer on a major outage. How do you run the incident?
Show the answer
Say this first
Restore service first and find the root cause later. I make sure there is one incident lead, one channel and a fixed update rhythm, so engineers work while someone else handles stakeholders. I scope the impact, check what changed recently, and go for the fastest safe mitigation: roll back, fail over, or route around the fault. Every action is timestamped in the incident log as it happens. When service is back, I confirm it with users and monitoring, then hand over to the RCA.
Go deeper
Roles
| Role | Job |
|---|---|
| Incident lead | decides, sets priorities, owns the timeline |
| Communications | updates stakeholders on a fixed schedule, for example every 30 minutes |
| Engineers | investigate and carry out agreed actions |
| Scribe | records actions and times; in a small team the lead may do this |
One person can hold two roles in a small team, but nobody should debug and run communications at once.
The first 15 minutes
- Declare the incident and its severity.
- Scope it: which services, sites and users, and is it growing?
- What changed? The change log, recent deployments, provider notices.
- Pick the quickest safe mitigation. A rollback you understand beats a fix you are guessing at.
- Give each action a named owner and a time to report back.
Rules that stop it getting worse
- One change at a time, announced in the channel before it is made.
- Preserve evidence when you can: save show output and logs before a reload.
- Time-box each hypothesis. If it produces no evidence in 15 minutes, move on.
- Open vendor and provider cases early. They cost little and take time to staff.
Closing it
Confirm recovery from the user side, watch for recurrence, send the all-clear, and book the RCA while memories are fresh.
They will ask next
A senior manager joins the bridge and starts directing engineers. What do you do?
Politely steer them to the incident lead or the communications person, give them the current status and the time of the next update, and keep the engineers focused. Clear structure protects the people doing the fix.
When do you reload a device during an outage?
When the evidence points at a device-level fault and a reload is the fastest safe way to restore service, after capturing show output, logs and any crashinfo. A reload without evidence may cure the symptom and destroy the only clue.
Traps
- Everyone debugging in parallel and changing things without telling each other.