What is a failure domain, and how do you design to limit the blast radius of a failure or a bad change?
Show the answer
Say this first
A failure domain is everything affected when one thing breaks. I keep them small with layer 3 boundaries between blocks, summarization and stubs so routing events stay local, independent control planes for things that must not fail together, and staged rollouts so a bad change hits a few devices before it can hit all of them. The tradeoff is more boundaries to maintain, more devices and sometimes less optimal routing.
Go deeper
Kinds of failure domain
- Layer 2: a VLAN and its spanning tree. A loop or storm affects everything in it.
- Routing: an OSPF area floods LSAs to every router in it; an EIGRP query travels until it hits a summary or a stub boundary.
- Control plane: a stack, a virtual pair or a chassis runs one software instance.
- Shared services: one DNS or RADIUS cluster, an SD-WAN controller, a cloud region.
- Physical: a room, a power feed, a conduit, a building.
- Change: an automation job that pushes to every device at once has a blast radius equal to the whole inventory.
How I bound them
- Routed boundaries at distribution (or at access), small VLANs, no VLANs spanning closets or sites.
- Summaries at area and block edges; stub areas and EIGRP stubs.
- Two independent devices instead of one virtual device where a shared bug would be unacceptable.
- Services that fail independently: several RADIUS servers in different places, anycast DNS, and a defined behavior when all of them are unreachable (for example a critical VLAN for 802.1X).
- Changes in waves: one lab, one site, one region, then everything, with automated checks between waves.
The cost
Each boundary is a summary to maintain, possible suboptimal paths, extra devices and more config. The goal is not the smallest possible domain, but domains sized to the damage the business can tolerate.
They will ask next
How can summarization cause an outage instead of preventing one?
A summary hides the state of the subnets inside it. If two distribution switches both advertise the same summary and one loses a link to part of the block, it may still attract traffic for that part and drop it at its discard route. Fix with a link between the summarizing pair, or by designing so each summarizer can reach every subnet it advertises.
Traps
- Talking only about hardware. Changes and automation are the most common way a whole network fails at once.