Network design

Access layer

• Ports for PCs, phones, APs, printers and cameras. Port count and PoE budget drive the hardware.
• Edge security: 802.1X or MAB, port security, BPDU Guard, DHCP snooping, dynamic ARP inspection.
• QoS classification and marking. The trust boundary sits here: trust the IP phone, re-mark the PC.
• Usually two uplinks, one to each distribution switch.

Distribution layer

• Aggregates one building or one group of closets. A distribution pair plus its access switches is called a distribution block.
• The layer 2 to layer 3 boundary when access is layer 2: SVIs, HSRP or VRRP, STP root for its VLANs.
• Policy that belongs between groups: ACLs, QoS policing, route filtering.
• Summarizes the block's subnets toward the core, so a flapping access subnet does not ripple out.

Core layer

• Connects distribution blocks, the data center, the WAN edge and the internet edge.
• Only fast, simple, redundant layer 3 forwarding. No endpoints, no layer 2 loops, no policy that needs regular tuning.
• It must never be the reason anything is slow or down, so changes here are rare and careful.

What the model buys you

Each layer has a clear job, so you know where a feature or a problem lives. Blocks are copied, not redesigned, when you grow. Failures stay inside a block because the boundaries are layer 3 with summaries.

Real life

It is a model, not an order to buy three tiers of boxes. Small sites merge core and distribution (collapsed core). Many modern campuses push layer 3 down to the access switch (routed access). The roles stay the same even when the boxes merge.

What goes wrong in a flat network

• One broadcast domain. ARP, DHCP, mDNS and other broadcast or flooded traffic reach every host. Unknown unicast floods everywhere whenever a MAC entry ages out.
• One spanning tree. A loop anywhere becomes a storm everywhere. The tree grows with every switch you daisy chain.
• No natural place for policy. Without layer 3 boundaries there is nowhere to filter between groups.
• Hard to troubleshoot. Every problem could be anywhere.

What hierarchy gives you

• Failure domains. Layer 3 between blocks stops a storm or loop at the block edge.
• Scale by copying. A new building is one more distribution block built from the same template.
• Summarization. Each block gets a contiguous range advertised as one route. A flapping subnet inside the block does not cause recalculation across the network.
• Cost placement. Expensive features and fast links go where they are needed, not on every switch.
• Predictability. Deterministic paths make capacity planning and troubleshooting easier.

Why it works for small sites

In one building the distribution blocks are just a few closets. A dedicated core would only connect one distribution pair to itself. Merging the layers saves two switches, the links between them and one layer of routing, with no real loss of function.

When to split out a core

• Many distribution blocks. Without a core, n blocks need n(n-1)/2 block-to-block connections to be fully meshed: 4 blocks need 6, 8 blocks need 28. With a core each block connects only to the core, so you need n.
• Several buildings. Fiber from every closet in every building to one pair gets long and expensive, and that pair becomes a site-wide failure domain.
• Port density. The pair runs out of 10G or 25G ports for access uplinks.
• Change risk. Every change to the collapsed pair touches everything. A separate core can stay untouched while you work on one block.

How I build the pair

• Two independent switches with a routed link between them, SVIs with HSRP or VRRP, and the STP root aligned with the active gateway.
• Or a virtual pair (StackWise Virtual on Catalyst 9000, VSS on older Catalyst 6500 and 4500) so each access switch connects with one multichassis EtherChannel and nothing blocks.

Why this matters in the interview

When you are asked to design anything, naming the blocks first shows structure. Then you size and secure each one. The blocks also map to failure domains: an internet edge problem should not take down internal routing.

Distance

• Structured cabling allows a 100 m channel for twisted pair: up to 90 m of fixed cable plus patch cords. That decides where the intermediate distribution frames (IDFs, the closets) go.
• Closet to main distribution frame (MDF) runs use fiber. Multimode is fine for short runs inside a building (10GBASE-SR reaches about 300 m on OM3 and 400 m on OM4). Use single-mode for long runs and between buildings.
• Run redundant uplinks through different risers or conduits. Two fibers in one conduit fail together.

Power

• PoE per port at the switch: 802.3af up to 15.4 W, 802.3at (PoE+) 30 W, 802.3bt 60 W (Type 3) or 90 W (Type 4). Newer APs, PTZ cameras and some lighting can need more than PoE+.
• Size the power supplies for total PoE draw plus the switch itself, and decide whether PoE must survive one supply failing.
• UPS runtime decides how long phones, APs and badge readers stay up in a power cut.

Space and cooling

A PoE switch turns much of its draw into heat in the closet, and the powered devices dissipate the rest. Closets need ventilation or cooling, rack space for future switches and a tidy patching layout.

Doing it in your head

Unavailability is 1 minus availability. 99.99 percent means 0.01 percent down, which is 1 in 10,000. 525,600 divided by 10,000 is about 52.6 minutes. Remember "about 9 hours, 53 minutes, 5 minutes" for three, four and five nines.

What to ask when someone quotes a number

• Measured on what: a device, a link, or the service the user sees?
• Does planned maintenance count? Many SLAs exclude scheduled windows, which changes everything.
• Over what period? A monthly SLA can be met while a single bad month ruins the yearly figure.

The network's part in DR

• Capacity at the DR site and WAN paths so branches can reach it.
• An address strategy: advertise the same prefixes from the DR site, or re-address and change DNS with short TTLs, or use global load balancing.
• Configuration backups, IPAM and diagrams stored outside the primary site.
• Regular DR tests. A plan that has never been run is a hope, not a plan.

Backups are part of DR, not HA. Restoring a config from backup can take an hour; a standby device takes seconds.

How to talk about it

Name the SPOF, its impact, and what you would do: remove it (add a second one with independent fate), or accept it and reduce recovery time (spare on the shelf, support contract, runbook, out-of-band access).

Rules

• Never use public space you do not own "because it was free". Those destinations become unreachable.
• 100.64.0.0/10 (RFC 6598) is shared address space for carrier-grade NAT, meant for providers, not a spare private range for the LAN.
• Allocate hierarchically so every site and function summarizes. Record every allocation in an IPAM tool from day one.
• Plan for mergers: a random choice inside 10.0.0.0/8 is more likely to overlap an acquired company's network than a plan that skips the obvious ranges.

IPv6

Get a global prefix, provider-independent from the RIR or from the ISP. Give each site a /48 (or at least a /56) and each VLAN a /64, which SLAAC needs. Unique local addresses (fd00::/8) exist for internal-only use, but most designs use global addresses with firewalls instead of IPv6 NAT.

Example for one closet

Closet 2 gets users at 10.1.12.0/24, keeping 10.1.11.0/24 free so closet 1 can grow into 10.1.10.0/23.

Quick sizes

Rules of thumb

• Keep VLAN numbers consistent per function across sites so people recognize them.
• Do not put users in VLAN 1. Use an unused VLAN as native on trunks.
• Wireless with a controller usually maps each SSID to one larger VLAN at the controller, or to a group of VLANs, instead of per-closet VLANs.
• Guest traffic goes straight to the internet through the firewall, never into internal subnets.

In-band management

• A dedicated management VLAN or VRF, with access restricted to jump hosts and monitoring servers.
• SSH only, AAA with TACACS+ or RADIUS and a local fallback account, SNMPv3, syslog and NTP.
• Restrict who can reach the device itself:

ip access-list standard MGMT-HOSTS
 permit 10.250.0.0 0.0.0.255
!
line vty 0 15
 access-class MGMT-HOSTS in vrf-also
 transport input ssh

vrf-also makes the access class apply to SSH arriving in a VRF, such as the management VRF.

Out-of-band management

• Management ports: GigabitEthernet0/0 in the Mgmt-vrf VRF on Catalyst 9000, mgmt0 in VRF management on NX-OS, Management1 on Arista EOS. Cable them to a separate management switch.
• Console servers connected to every console port, reachable over a separate circuit or LTE. This is what saves you when a bad change cuts off in-band access.
• Secure it as carefully as production: it is a direct path to every device.

Why it matters

When a change breaks routing, in-band access goes with it. Out-of-band access turns a site visit or a long outage into a quick rollback.

The static route's blind spot

A static route stays in the table as long as its exit interface is up. If the failure is beyond the next hop, for example inside the ISP, the route stays and traffic is black-holed. Fix it with IP SLA and object tracking, or use a routing protocol.

Floating static as a backup

A floating static has an administrative distance higher than the primary route, so it is installed only when the primary disappears. A branch that learns its default by OSPF over MPLS (AD 110) can keep a backup over an internet tunnel:

ip route 0.0.0.0 0.0.0.0 Tunnel100 250

When dynamic routing is worth it

• More than one path to the same place.
• More than a handful of subnets or sites.
• Frequent adds and moves, where manual routes would drift.
• When you need fast, automatic failover without scripting.

Building blocks

• Edge: router or SD-WAN edge, often with the firewall built in. Encrypted tunnels to the hubs or the cloud.
• LAN: a PoE switch or small stack, the gateway SVIs on the edge or on a layer 3 switch.
• Wireless: cloud-managed APs, or controller APs with local switching so traffic does not hairpin to a central controller.
• Services: DHCP local on the edge or relayed to central servers, DNS forwarding, NTP.

Decisions that matter

• Internet breakout: local breakout for SaaS and internet is now common; backhaul everything only if central inspection is mandatory and the WAN can carry it.
• Survivability: what still works if the WAN fails? Local DHCP, cached DNS, voice survivability (SRST on Cisco) so phones can still call locally and out a local line if one exists.
• Standardization: same VLAN IDs, same address layout, same config template. Zero-touch provisioning so a non-specialist can install the hardware.
• Security: guest goes straight to the internet; edge ports use 802.1X or MAB like the campus.

Worked examples

Things that change the real ratio

• Failures. Design for the case where one uplink is down. In a layer 2 design without multichassis EtherChannel, STP blocks one uplink all the time, so the second row is your normal state.
• Traffic pattern. Office traffic is bursty and mostly north-south to the internet or data center. Storage, backups and AI training traffic are sustained and east-west, so those fabrics aim for 1:1.
• Microbursts. A 5-minute average can show 15 percent while the port drops packets in bursts a few milliseconds long, especially when many senders hit one receiver. Check output drops, not just averages.

The 20:1 and 4:1 figures are rules of thumb from older campus design guides. Validate them against your own traffic.

Business

• What does the business do here, and which services must never stop?
• Headcount now and in three years? New sites, hybrid work, acquisitions?
• What does an hour of outage cost, and who decides on risk?

Technical

• Applications: SaaS, on-premises, data center or cloud? Voice and video? Latency-sensitive systems?
• Devices: laptops, phones, IoT, cameras, production equipment. How many need PoE?
• Availability target per service, and security or compliance rules (card payments, health data, segmentation requirements).

Constraints

• Budget for capital and running costs, timeline, existing equipment and contracts.
• The building: floors, closets, cabling, power. Which carriers serve the address.
• The team: skills, size, vendor preference, managed service or in-house.

In an interview

Ask three or four sharp questions, then say "I will assume the following; tell me if any are wrong" and continue. Interviewers grade the structure as much as the final design.

Keeping it alive

• Make documentation updates a closing step of every change, checked in review.
• Use a source of truth (NetBox or similar) for devices, IPs, cables and circuits, and generate diagrams and configs from it where you can.
• Back up configs automatically with a tool that keeps history.
• Keep an offline copy reachable during an outage. Docs on a server behind the broken network do not help.

A third option is layer 2 access to a virtual distribution pair (StackWise Virtual or VSS). The uplinks form one multichassis EtherChannel, nothing blocks, and the pair is one gateway. It keeps layer 2 flexibility but puts both distribution switches in one control plane.

Routed access uplink on IOS-XE

interface TenGigabitEthernet1/1/1
 description To DSW1
 no switchport
 ip address 10.1.255.1 255.255.255.254
 ip ospf network point-to-point
!
interface Vlan10
 description USERS-FLOOR1
 ip address 10.1.10.1 255.255.255.0
 ip helper-address 10.2.0.10
!
router ospf 1
 router-id 10.1.0.11
 passive-interface default
 no passive-interface TenGigabitEthernet1/1/1
 no passive-interface TenGigabitEthernet1/1/2
 network 10.1.0.0 0.0.255.255 area 10

The second uplink, Te1/1/2 to DSW2, is configured the same way with its own /31. Area 10 is summarized at the distribution ABRs. With EIGRP the access switch would be eigrp stub.

Why no FHRP is needed

If the access switch dies, its hosts are down anyway because each host has one cable. So the gateway on the access switch adds no new single point of failure.

Reasons to stay layer 2

Apps or devices that need the same subnet in several closets, an existing large layer 2 design, access switches without a layer 3 license, or a team that is not ready to run routing at the edge.

How redundancy hurts

• New failure modes. A virtual pair can go dual-active when its links fail. A unidirectional link can stop BPDUs and make a blocked STP port forward, creating a loop. An MLAG peer link failure can isolate the wrong switch.
• Untested paths. Backup paths often fail when needed because nobody exercised them: an expired license on the standby firewall, a missing VLAN on the second uplink.
• Shared fate. Two routers running the same software version share the same bug. Two circuits in one duct fail together. Redundancy that is not independent is not redundancy.
• Operators. More boxes and more protocols mean longer troubleshooting and more chances to make a mistake. Human error is a leading cause of outages.

How I balance it

• Quantify: what does an hour of outage cost, and how much does each option reduce expected downtime?
• Prefer layer 3 and ECMP over layer 2 tricks, and open standards the team understands.
• Two independent paths, not three. The third rarely changes the math but always adds work.
• Make failover automatic and fast, and test it in maintenance windows.
• Standardize so every site fails the same way.

A quick number to quote

Two independent components at 99.9 percent in parallel give about 99.9999 percent. A third adds almost nothing on paper, and in practice the limit becomes software bugs and human error, which extra boxes do not fix.

Series

A = A1 x A2 x A3. Three components at 99.9 percent: 0.999 x 0.999 x 0.999 = 0.997, about 99.7 percent or roughly 26 hours a year. Every box in a single path lowers the total.

Parallel

A = 1 - (1 - A1)(1 - A2). Two links at 99.5 percent: 1 - (0.005)(0.005) = 1 - 0.000025 = 99.9975 percent, about 13 minutes a year instead of about 44 hours for one link.

Combine them

Those two links terminating on one router at 99.95 percent: 0.999975 x 0.9995 = about 99.9475 percent, roughly 4.6 hours a year. The single router now dominates. That is the point of the exercise: find the series element that limits you.

Single component

A = MTBF / (MTBF + MTTR). MTBF 50,000 hours with a 4 hour repair gives 50,000 / 50,004, about 99.992 percent. With a 24 hour repair it drops to about 99.95 percent. MTTR is the lever you control: spares on site, support contracts, out-of-band access, runbooks.

Where the math lies

• It assumes independent failures. Two circuits in one duct, two routers on one UPS or one software bug on both boxes break the parallel formula.
• It assumes failover is instant and always works. A 40 second reconvergence counts as downtime.
• It ignores human error and changes, which cause a large share of real outages.

Method

1. List critical services with the business: payments, voice, the ERP, the warehouse scanners.
2. Trace each path from user to server and back: access port, uplinks, distribution, core, firewall, WAN or internet, data center, the server, and the return path.
3. At each hop ask what fails it (device, link, power, software, config) and what happens: does it fail over automatically, how fast, and has anyone ever tested it?
4. Check shared risk. Two links are redundant only if they do not share a conduit, a building entry, a carrier exchange, a line card or a PDU. Carriers call this a shared risk link group (SRLG). Ask carriers for route maps of the circuits, not just two circuit IDs.
5. Check what the path depends on: DNS, DHCP, RADIUS, NTP, certificates, the wireless controller, licensing, a cloud controller.
6. Check people and process: who can fix it at 3 a.m., where are the credentials, are configs backed up, are there spares and support contracts.

Output

A SPOF register: item, impact, likelihood, current recovery time, fix, cost and decision (fix now, plan, or accept with a named owner). Review it after every incident.

Prove it

Planned failover tests in a maintenance window: pull the primary link, power off the active firewall, fail the primary RADIUS server. Most hidden SPOFs are found by tests, not by drawings.

Company level

Branch level

Branch k (0 to 63) starts 16 x k further along in the third octet, rolling into the next second octet every 16 branches: branch 1 is 10.16.16.0/20, branch 15 is 10.16.240.0/20, branch 16 is 10.17.0.0/20. The layout inside branch 1:

What summarization gives you

Each branch advertises one /20. The WAN hubs advertise 10.16.0.0/14 toward the data centers and HQ, which then hold one route for every branch. Firewall rules and ACLs can say "all branches" in one line. A flapping branch subnet does not ripple through the network.

IPv6 alongside

A /48 per site and a /64 per VLAN. Writing the VLAN number into the fourth hextet keeps it readable: VLAN 10 at HQ could be 2001:db8:1:10::/64.

Design principles

• One IGP if you can. Every redistribution point is a place for loops and surprises.
• If you must redistribute, do it at as few points as possible, tag routes and filter on the tags so routes do not loop back.
• Fast convergence comes mostly from design and failure detection (routed links, BFD, summarization), not from which IGP you picked.
• Match the team: a protocol nobody on call understands is a risk, however elegant.

Options

The tracked default route on IOS-XE

ip sla 1
 icmp-echo 192.0.2.53 source-interface GigabitEthernet0/0/0
 frequency 10
ip sla schedule 1 life forever start-time now
!
track 1 ip sla 1 reachability
 delay up 60
!
ip route 0.0.0.0 0.0.0.0 198.51.100.1 track 1
ip route 0.0.0.0 0.0.0.0 203.0.113.1 250
ip route 192.0.2.53 255.255.255.255 198.51.100.1

• The probe target is beyond the ISP, not just its next hop, so it catches failures inside ISP1.
• The host route keeps the probe on ISP1 even after failover.
• delay up 60 waits a minute before moving back, so a flapping link does not bounce traffic.

NAT per ISP

ip access-list standard NAT-INSIDE
 permit 10.0.0.0 0.255.255.255
route-map NAT-ISP1 permit 10
 match ip address NAT-INSIDE
 match interface GigabitEthernet0/0/0
route-map NAT-ISP2 permit 10
 match ip address NAT-INSIDE
 match interface GigabitEthernet0/0/1
ip nat inside source route-map NAT-ISP1 interface GigabitEthernet0/0/0 overload
ip nat inside source route-map NAT-ISP2 interface GigabitEthernet0/0/1 overload

match interface picks the translation by exit interface. After failover new sessions leave with the ISP2 address; existing sessions break because their public source address changes.

Other things to ask for

Physically diverse circuits (different building entries and exchanges), the firewall pair in HA, and where remote access VPN and published services terminate.

Questions that decide the tier

• What does an hour of outage at this site cost? Can staff work offline?
• Are there local systems (point of sale, production lines, cameras) that need the WAN?
• What can the carriers actually deliver at that address? Sometimes only one fiber provider exists.

Design details

• Real diversity: two carriers can share the same last-mile duct. Ask for the path, not just a second circuit ID.
• SD-WAN: measures loss, latency and jitter on every path and steers applications to the path that meets their policy, so two internet links can replace an MPLS circuit for many sites.
• Two edge devices: need a first hop redundancy protocol (VRRP or HSRP) or routing on the LAN side, and each device owns one circuit.
• LTE: watch data caps and costs; consider sending only critical traffic over it.

Measure

• Interface counters by SNMP or streaming telemetry at 1 to 5 minute intervals. Look at the busy hour and the 95th percentile, not the monthly average.
• Output drops and discards, which show congestion that utilization graphs smooth over.
• NetFlow or IPFIX to know who and what uses the link, so you can tell growth from one rogue backup job.

Plan for failure

In a pair of links that share load, each must stay under about 50 percent at peak, or the survivor cannot carry both when one fails. The same applies to firewall and router pairs.

Forecast

• Trend the busy-hour figures monthly and project when you will cross the trigger.
• Add known events: a new office, a cloud migration, a video platform rollout.
• Work back from lead times. New circuits commonly take weeks to months, and hardware can too.

Capacity beyond bandwidth

Say it in business terms

"At the current trend this link reaches 70 percent at peak in five months. A circuit upgrade takes about three months. We should order it now."

1. Questions first

How many floors and how large? Growth in three years? Desk phones or softphones? Cameras, badge readers, meeting room AV? Applications in SaaS, a data center or local servers? Connection to HQ? Downtime cost? Budget and in-house skills? Then state assumptions: two floors, growth to 250 people, mostly laptops on Wi-Fi, desk phones in some areas, SaaS heavy, a link to the HQ data center.

2. Physical

One closet per floor within the 100 m copper limit, fiber between closets and the main room on two paths, UPS per closet sized for the PoE load, a small comms room for the core and edge.

3. Switching

• Ports: about 200 desks, 30 meeting room and AV ports, printers, about 14 APs, 20 cameras, plus 20 percent spare: roughly 330 ports, so seven or eight 48-port PoE+ switches, a stack of four per floor. Multigigabit ports for the APs.
• Each stack has two 10G uplinks from different members, bundled into one EtherChannel to the core pair. 192 x 1G down against 20G up is 9.6:1, well inside campus norms.
• Collapsed core: two layer 3 switches, either a StackWise Virtual pair or two independent switches with VRRP or HSRP. The core hosts the SVIs and connects to the firewalls.

4. VLANs and addressing (site block 10.20.0.0/16)

5. Wireless

With 200 people carrying two or three devices each, plan for 400 to 600 wireless clients, not all active at once. A rule of thumb of 25 to 35 active clients per AP gives about 12 to 16 APs before meeting rooms are counted; a predictive survey sets the number and a validation survey checks it. 5 GHz and 6 GHz for corporate devices, 20 or 40 MHz channels on 5 GHz, WPA3-Enterprise with 802.1X, a guest SSID with a portal. Keep it to three or four SSIDs, since each one costs airtime.

6. Internet and WAN

Two internet circuits from different carriers with diverse building entries, an active/standby firewall pair, tracked default routes (or BGP if they have their own address space), and IPsec or SD-WAN to HQ and the cloud. SaaS breaks out locally.

7. Security

802.1X on wired ports with MAB for printers and cameras, DHCP snooping, dynamic ARP inspection, BPDU Guard, storm control. Firewall policy between users, IoT and servers. Guest isolated.

8. Services and management

DHCP (central with relay, or on the core), DNS, NTP, RADIUS, syslog, SNMPv3 or telemetry, NetFlow. A management VRF, a console server with LTE for out-of-band access, config backups and IPAM from day one.

9. Tradeoffs

On a tight budget I would accept one ISP plus LTE backup, or a single firewall with a fast support contract. I would not cut the second core switch or the UPS, because either failure takes the whole office down.

Phases

1. Discovery: inventory, configs, every port mapped with CDP, LLDP and MAC tables, and hidden dependencies: static routes to old addresses, SNMP and syslog sources, firewall rules that name the old devices, RADIUS clients.
2. Design and test: new configs built from templates, tested in a lab, peer reviewed.
3. Method of procedure (MOP): numbered steps with exact commands, expected output, who does what, timings and checkpoints.
4. Pre-checks: capture interface states, routing neighbors and route counts, ARP and MAC counts, STP root and FHRP state, a ping matrix to key hosts, and application tests by their owners.
5. Window: go or no-go meeting, other changes frozen, console or out-of-band access confirmed, safety nets such as reload in or commit confirmed where they fit.
6. Rollback: triggers written in advance, for example "any critical application test fails and is not fixed within 15 minutes" or "not complete by 03:00". The rollback time is reserved inside the window.
7. Post-checks: the same captures, compared line by line, and application owners confirm.
8. Hypercare: extra monitoring and someone on site the first business morning.
9. Close: update diagrams, IPAM, monitoring and the documentation, and hold a short review.

Techniques that lower risk

• Build new beside old and move links in batches instead of one big bang.
• Drain traffic before touching a device, for example OSPF max-metric router-lsa or higher costs, so traffic has already moved when you cut.
• Lower DNS TTLs days before any change that moves service addresses.
• Pilot with a small, friendly group first.

What the interviewer is testing

Ownership, honesty, how fast you recover, how you communicate, and whether you fix the system instead of promising to "be more careful". Everyone senior has caused an outage. Claiming you never have is a red flag.

The model, step by step

• Situation: a planned evening change to add a VLAN for new conference room equipment on a distribution trunk serving two floors.
• Task: the change was mine. When it went wrong, so was the recovery and the explanation.
• Action: monitoring alarmed within a minute. Mine was the only change in progress, so I checked show interfaces trunk, saw the allowed list reduced to VLAN 40, and restored it from my pre-change output. I told the incident lead and service desk what had happened before anyone asked. The next day I wrote a blameless post-incident review with the timeline and root cause.
• Result: about six minutes of impact. We added two controls: trunk changes use reviewed command scripts, and a post-check compares the allowed VLAN list before and after. That class of error did not happen again.
• Lesson: the process allowed one mistyped word to take down two floors; a review and an automatic check removed that path.

Adapting it

Pick a real outage you were part of, keep it under two minutes, give numbers (duration, users, services), and spend most of the time on actions and what changed afterwards.

What the interviewer is testing

Whether you argue with data and requirements instead of opinions, whether you listen, and whether you can commit to a decision you did not prefer.

The model, step by step

• Situation: a design review for data center failover. My colleague proposed stretching the server VLAN between sites.
• Task: reach a decision that the network team and the application team could support.
• Action: I asked what the real requirement was and learned it covered only three applications. We agreed the criteria before arguing about solutions: failover time, risk, cost and effort to run. I tested both approaches in the lab and shared the results with my colleague before the meeting, so nobody was surprised in front of the architect.
• Result: two applications moved to DNS-based failover; the third got a narrow extension with storm control and a written plan for an interconnect failure. The relationship stayed good, and the criteria template is still used.
• Lesson: agreeing on criteria first turns a debate about opinions into a comparison.

If you lost the argument

That is a fine story too. Show that you raised the risk clearly, accepted the decision, helped make it work and documented the risk.

What the interviewer is testing

Calm under pressure, triage by business impact, restore first and diagnose second, clear communication, and not trying to do everything alone.

The model, step by step

• Situation: Saturday morning, the primary firewall crashed and rebooted, the standby took over but was missing a recent NAT rule. Card payments failed across 40 stores. Monitoring showed hundreds of alerts and managers were calling.
• Task: I was on call and became the technical lead.
• Action:
  1. Asked for an incident commander and a single bridge, so I could work while someone else handled updates and callers.
  2. Ranked by business impact: payments first, everything else waits.
  3. Filtered the alert storm to the payment path only.
  4. Looked for the fastest safe restore: the primary was healthy again after its reboot, so I failed back to it after checking its state.
  5. Gave a short update every 15 minutes through the incident commander, and kept a timeline.
• Result: payments restored 35 minutes after the first alert. The review found configuration sync between the pair had been failing for weeks. We added monitoring for sync state and a monthly failover test.

Principles to say out loud

Stop the bleeding before finding the root cause. One person coordinates, one person types. Fixed update intervals beat constant interruptions. Write the timeline as you go.

What the interviewer is testing

Whether you can speak the language of the business, give options instead of complaints, and get a clear decision, including a documented acceptance of risk if the answer is no.

The model, step by step

• Situation: single circuit and single firewall at headquarters; two outages the previous year.
• Task: get either funding or an explicit, recorded acceptance of the risk.
• Action: I asked finance what an hour of contact center downtime costs, added the outage history and the carrier's repair times, and wrote one page with three options: do nothing, add LTE backup, or add a second carrier and an HA firewall pair. Each had cost, expected downtime and what it still did not cover. I made a recommendation and kept the jargon out.
• Result: they chose the third option. A later fiber cut on the primary circuit had no user impact.
• Lesson: numbers and options get decisions; warnings get postponed.

If the answer had been no

Say so in your story if it happened. Record the risk acceptance with the owner's name, and set a date to revisit it.

1. Detection

• Direct routed links between switches. When the fiber or optic fails, the interface goes down and routing reacts at once. Hello timers are only a backstop.
• Where a carrier circuit or media converter hides the failure, run BFD. Three missed 50 ms packets means about 150 ms detection. Supported minimum intervals vary by platform.

2. Precomputed alternates

• Triangles, not squares. Each access switch connects to both distribution switches. The two paths are equal cost and both are in the forwarding table, so losing one is a local removal, not a recalculation. In a square the backup path has to be computed first.
• EIGRP feasible successors and OSPF loop-free alternates (where supported) give a backup next hop in advance on non-ECMP topologies.

3. Limit the work

• Summarize each distribution block toward the core. A failure inside the block changes nothing outside.
• EIGRP stub on access switches keeps queries from spreading. OSPF areas keep LSA flooding local.
• SPF throttle with a short initial wait so the first event is processed quickly.

4. Remove layer 2 from the path

Routed access takes STP and HSRP out of uplink failover. If you keep layer 2 access, use Rapid PVST+, a loop-free design and tuned FHRP timers.

5. Survive supervisor failures

NSF/SSO lets a dual-supervisor chassis keep forwarding while the standby takes over and neighbors help resynchronize. Aggressive hello or BFD timers toward that chassis can declare it dead before the switchover completes, so pick timers that let NSF work.

Example on IOS-XE

interface TenGigabitEthernet1/1/1
 no switchport
 ip address 10.1.255.0 255.255.255.254
 ip ospf network point-to-point
 bfd interval 50 min_rx 50 multiplier 3
!
router ospf 1
 bfd all-interfaces
 timers throttle spf 10 100 5000

Prove it

Send steady traffic (a traffic generator, or pings at a short interval) and pull a link, then a switch. Lost packets multiplied by the interval gives the real outage. Measure, do not assume.

Why stretched layer 2 is risky

• Shared failure domain. Broadcast, unknown unicast and STP events cross the interconnect. One looped cable can take out both data centers.
• Traffic tromboning. A VM moved to site B still uses its gateway in site A unless the gateway is present in both sites.
• Split brain. If the interconnect fails, both halves of a cluster may think they are primary, and the same IP may be active in two places.
• Bandwidth. Flooded traffic and replication compete with real traffic on the interconnect.

Options, best first

1. Remove the need: DNS or global load balancing to move users between sites, applications that replicate data themselves, automation that re-addresses during failover.
2. Routed interconnect, separate subnets per site, and advertise service addresses where they live.
3. If layer 2 is truly required, VXLAN with BGP EVPN between sites: distributed anycast gateway in both sites, ARP suppression, controlled flooding, no STP across the interconnect, only the VLANs that need it.

Questions I ask

• Which VLANs, how many hosts, and for how long?
• What should happen when the interconnect fails? Who is primary?
• Is this for live migration, or for disaster recovery where a short re-address would be fine?

How fast the budget goes

5.26 minutes a year is about 315 seconds. With default timers:

• 802.1D spanning tree reconverges in 30 to 50 seconds.
• OSPF on broadcast or point-to-point links declares a neighbor dead after 40 seconds if it relies on hellos (10 second hello, 40 second dead).
• BGP on Cisco waits 180 seconds (hold time) by default.
• A non-redundant switch reload for an upgrade often takes several minutes on its own.

A handful of failovers on default timers, or one unplanned reload, spends the whole year.

What it takes

• Redundancy with independent fate: separate power feeds and UPS, diverse fiber paths, different carriers, ideally different software trains on critical pairs.
• Sub-second detection and failover: routed links, BFD, ECMP, tuned FHRP where layer 2 remains.
• Hitless maintenance: rolling upgrades across independent pairs, or ISSU where it is proven.
• NSF/SSO on chassis with dual supervisors.
• Change discipline: peer review, pre and post checks, automatic rollback, staged rollouts.
• Monitoring that measures the service from the user's side, so you know the real number.

How I answer the business

"Which service, and what does an hour of it being down cost? Five nines for voice in the contact center may be worth it. Five nines for every printer is not. Here are two options with their availability, cost and what they still do not protect against."

Kinds of failure domain

• Layer 2: a VLAN and its spanning tree. A loop or storm affects everything in it.
• Routing: an OSPF area floods LSAs to every router in it; an EIGRP query travels until it hits a summary or a stub boundary.
• Control plane: a stack, a virtual pair or a chassis runs one software instance.
• Shared services: one DNS or RADIUS cluster, an SD-WAN controller, a cloud region.
• Physical: a room, a power feed, a conduit, a building.
• Change: an automation job that pushes to every device at once has a blast radius equal to the whole inventory.

How I bound them

• Routed boundaries at distribution (or at access), small VLANs, no VLANs spanning closets or sites.
• Summaries at area and block edges; stub areas and EIGRP stubs.
• Two independent devices instead of one virtual device where a shared bug would be unacceptable.
• Services that fail independently: several RADIUS servers in different places, anycast DNS, and a defined behavior when all of them are unreachable (for example a critical VLAN for 802.1X).
• Changes in waves: one lab, one site, one region, then everything, with automated checks between waves.

The cost

Each boundary is a summary to maintain, possible suboptimal paths, extra devices and more config. The goal is not the smallest possible domain, but domains sized to the damage the business can tolerate.

Before you start

• A public ASN and provider-independent space, or the ISP's permission (often a letter of authorization) to announce its space through another provider.
• At least a /24 for IPv4 and a /48 for IPv6. Longer prefixes are filtered by most networks.
• IRR route objects and RPKI ROAs for your prefixes, so both upstreams and their peers accept them.
• Decide what to receive: default only, default plus the ISPs' customer routes, or full tables (memory heavy).

Example on one router

Our AS 64500 with 198.51.100.0/24. ISP1 (AS 64501) is primary, ISP2 (AS 64502) is backup.

ip prefix-list OUR-PREFIX seq 10 permit 198.51.100.0/24
ip prefix-list DEFAULT-ONLY seq 10 permit 0.0.0.0/0
!
route-map ISP1-IN permit 10
 match ip address prefix-list DEFAULT-ONLY
 set local-preference 200
route-map ISP1-OUT permit 10
 match ip address prefix-list OUR-PREFIX
route-map ISP2-IN permit 10
 match ip address prefix-list DEFAULT-ONLY
route-map ISP2-OUT permit 10
 match ip address prefix-list OUR-PREFIX
 set as-path prepend 64500 64500 64500
!
router bgp 64500
 bgp log-neighbor-changes
 neighbor 192.0.2.1 remote-as 64501
 neighbor 203.0.113.1 remote-as 64502
 address-family ipv4
  network 198.51.100.0 mask 255.255.255.0
  neighbor 192.0.2.1 activate
  neighbor 192.0.2.1 route-map ISP1-IN in
  neighbor 192.0.2.1 route-map ISP1-OUT out
  neighbor 203.0.113.1 activate
  neighbor 203.0.113.1 route-map ISP2-IN in
  neighbor 203.0.113.1 route-map ISP2-OUT out
!
ip route 198.51.100.0 255.255.255.0 Null0

• The outbound route maps announce only our prefix, so we never become transit between the ISPs.
• The Null0 route lets network find an exact match in the RIB.
• With two edge routers, local preference carries across iBGP so both routers agree on the exit.

Why prepending is not enough

ISP2 itself usually gives routes from its customers a higher local preference than routes from its peers, and local preference beats AS path length. So ISP2 and its customers still send traffic to us over the backup link. To make ISP2 a true backup, use the community ISP2 publishes for "lower my local preference for this route".

More-specifics

Announcing a /23 to both ISPs and its two /24s only on the primary works everywhere, because longest match wins. It needs a block larger than /24, adds routes to the global table, and the RPKI ROAs must allow the longer prefixes or they become invalid.

What breaks first

Target design

• Campus: split out a core once there are several distribution blocks or buildings; routed access for new blocks.
• Addressing: a hierarchical plan by region and site with summaries at block and region edges, leaving room for acquisitions and cloud.
• Routing: one IGP with areas per building or region, BGP toward the WAN, cloud and internet.
• Edges: larger and diverse circuits, regional internet breakout, SD-WAN for branches, dedicated cloud connections (Direct Connect, ExpressRoute) once cloud traffic is steady.
• Operations: the team will not triple, so automation is the multiplier: a source of truth such as NetBox, templates, generated configs, compliance checks and CI for changes, monitoring that scales.

Phasing

1. Foundations that do not touch users: IPAM and source of truth, monitoring, the new address plan, standards and templates.
2. Build the new core in parallel and move one distribution block at a time.
3. Scale the internet edge and WAN, then retire what is left of the old design.

Each phase has its own rollback and measurable exit criteria.

Tradeoff to name

Building ahead of growth costs capital early; building just in time risks outages during the growth itself. I would build the structure (core, addressing, automation) early and add capacity as it is needed.

1. Discover

• ARP table on the gateway and MAC tables on the switches: what exists and where it plugs in.
• Compare DHCP leases with ARP entries. Addresses in ARP but not in DHCP are static: printers, servers, PLCs, cameras, anything with a hard-coded IP.
• NetFlow or firewall logs show who talks to whom, which becomes the future rule set.
• Find IP dependencies: licenses bound to addresses, ACLs, monitoring, partner VPNs.

2. Build beside the old network

New core or distribution with SVIs for new subnets from a fresh block that does not overlap the old /16. Trunk the legacy VLAN to the new core.

3. Move the gateway

In a short window, move the legacy gateway address to the new core, ideally as the FHRP virtual address. The new gateway announces itself with gratuitous ARP; hosts that miss it relearn when their ARP entry expires. Now old and new subnets route in one place.

4. Migrate in waves

Per closet: change access ports to the new VLAN and bounce each port so the client sees a link down and asks DHCP for a new lease. Check, then move the next closet. Phones and printers by type.

5. Static devices last

Re-address them with their owners, or leave them in the legacy subnet for now and shrink it later.

6. Security

Allow everything between the new segments at first, but log it. Build rules from what you see, then switch to deny by default per segment.

7. Retire

When the legacy ARP table holds only known stragglers, finish them and remove the VLAN.