Replace a distribution switch with a runbook, pre and post checks and a rollback

Cisco IOSWorking

Replace DSW1, one of a pair of distribution switches, with new hardware during a four-hour window and without a user-visible outage. Write the runbook the way a change board expects it: capture the pre-state, drain the switch, swap it, bring links up one at a time, compare the post-state with the pre-state, and decide go or rollback at a fixed time.

Item	DSW1 today
Router ID	`10.255.0.11`
Routed uplinks	Te1/0/1 `10.255.1.1/31` to CORE1, Te1/0/2 `10.255.1.3/31` to CORE2
Link to DSW2	Po1 (Te1/0/3 and Te1/0/4) trunk, OSPF over `Vlan900` `10.255.1.9/30`, DSW2 is `10.255.1.10`
OSPF	All three OSPF interfaces use `ip ospf network point-to-point`, so there is no DR or BDR
Access	Po11 to ASW1, Po12 to ASW2
HSRP	Active for VLANs 10 and 30 (priority 110), standby for 20 and 40 (priority 100). DSW2 mirrors it. Preempt on both
STP	Root for VLANs 10 and 30 (priority 24576). DSW2 is root for 20 and 40 and secondary (28672) for 10 and 30
Window	Saturday 22:00 to 02:00. Go or rollback decision at 00:30

The new DSW1 has been staged in the lab with the old configuration, plus the drain settings from the drain step, so it comes up without taking any role.

Tasks

Write the change header with impact, go and no-go criteria, rollback owner, decision time and comms plan.
Capture the pre-state on DSW1 and DSW2 to files, and copy DSW1's files off the box.
Put a safety net on every device you touch that stays in service.
Drain DSW1 so DSW2 carries everything before any cable moves.
Swap the hardware and bring the new DSW1's links up one group at a time, checking after each.
Return DSW1 to its normal roles.
Capture the post-state and compare it with the pre-state.
Make the go or rollback call at the decision time and close the change.

Solution

1Change header and comms plan

Text

CHANGE      CHG-2041  Replace DSW1 (building A distribution)
WINDOW      Saturday 22:00 to 02:00
IMPACT      None expected. Under 2 s of loss possible on VLANs 10 and 30 at each HSRP move
GO/NO-GO    21:45 bridge call: DSW2 healthy, new DSW1 staged and tested, rollback kit on site
DECISION    00:30 go or rollback. Rollback must be finished by 01:30
ROLLBACK    Old DSW1 stays racked below the new one, every cable labelled with both ends
COMMS       Service desk and stakeholders at start, every 30 minutes, at the decision, at close
ROLES       Implementer, reviewer on the bridge, rollback owner

2Capture the pre-stateDSW1

DSW1 on Cisco IOS

show ip interface brief | redirect flash:pre-ipint.txt
show ip route summary | redirect flash:pre-route.txt
show ip ospf neighbor | redirect flash:pre-ospf.txt
show standby brief | redirect flash:pre-hsrp.txt
show spanning-tree root | redirect flash:pre-stp.txt
show etherchannel summary | redirect flash:pre-po.txt
show mac address-table count | redirect flash:pre-mac.txt
show running-config | redirect flash:pre-run.txt
copy flash:pre-ospf.txt scp://netops@10.99.0.20/chg2041/dsw1-pre-ospf.txt

Copy every file off the box, not only the OSPF one shown: DSW1's flash leaves with the old hardware. Run the same set on DSW2, which stays, so you can see how the pair looked before and after.

3Safety net on the switches that stayDSW2
DSW2 on Cisco IOS
```
archive
 path flash:$h-cfg
 maximum 5
end
!
configure terminal revert timer 20
! make any change needed on DSW2 here, then check it
end
configure confirm
```
With an archive path set, configure terminal revert timer 20 rolls the running config back automatically after 20 minutes unless you enter configure confirm. If a change cuts you off, the switch fixes itself. Do not use reload in on DSW2: tonight it carries all the traffic.
4Drain DSW1DSW1
DSW1 on Cisco IOS
```
interface Vlan10
 standby 10 priority 90
interface Vlan30
 standby 30 priority 90
!
spanning-tree vlan 10,30 priority 32768
!
router ospf 1
 max-metric router-lsa include-stub
!
interface range TenGigabitEthernet1/0/1 - 4 , TenGigabitEthernet1/0/11 - 14
 shutdown
```
DSW2 preempts and becomes HSRP active for VLANs 10 and 30. Raising DSW1's bridge priority hands the root to DSW2's 28672 before any link drops. max-metric router-lsa steers routed traffic away from DSW1, and include-stub also raises the cost of its own subnets (the user VLANs it shares with DSW2), so the cores stop sending return traffic to it. Check show standby brief on DSW2 shows Active for all four groups before shutting the links.

5Bring up the new DSW1 one link group at a timeDSW1

DSW1 on Cisco IOS

interface range TenGigabitEthernet1/0/1 - 2
 no shutdown
! check: show ip ospf neighbor lists CORE1 and CORE2 in FULL
interface range TenGigabitEthernet1/0/3 - 4
 no shutdown
! check: Po1 is SU and DSW2 is FULL on Vlan900
interface range TenGigabitEthernet1/0/11 - 12
 no shutdown
! check: Po11 is SU and MAC addresses from ASW1 appear
interface range TenGigabitEthernet1/0/13 - 14
 no shutdown
! check: Po12 is SU and MAC addresses from ASW2 appear

The staged config still holds the drain settings, so the new switch joins without taking any role. One link group at a time means a problem shows up against the one change that caused it.

6Return DSW1 to its normal rolesDSW1
DSW1 on Cisco IOS
```
router ospf 1
 no max-metric router-lsa
!
spanning-tree vlan 10,30 priority 24576
!
interface Vlan10
 standby 10 priority 110
interface Vlan30
 standby 30 priority 110
```
DSW1 preempts back to HSRP active for VLANs 10 and 30 and becomes root for them again, so the HSRP gateway and the STP root stay on the same switch.
7Capture the post-state and compare
Linux
```
diff <(awk '{print $1,$3,$7}' dsw1-pre-ospf.txt) <(awk '{print $1,$3,$7}' dsw1-post-ospf.txt) && echo SAME
diff dsw1-pre-hsrp.txt dsw1-post-hsrp.txt && echo SAME
diff dsw1-pre-po.txt dsw1-post-po.txt && echo SAME
```
Capture the post files on DSW1 with the same commands as the pre-state (post- names), copy them to the server, then compare. For OSPF, compare only neighbor ID, state and interface: the dead timer changes every second and would make every line differ.

8Decision point and close

Text

00:30 DECISION
  GO if all true:
    OSPF neighbors match the pre-state (3 in FULL)
    HSRP roles match the pre-state
    STP root for VLANs 10 and 30 is DSW1 again
    Po1, Po11 and Po12 are SU with every member P
    MAC count within 10% of the pre-state
    Test call and test login pass from each access closet
  ROLLBACK if any check fails and cannot be fixed by 00:30:
    Shut the new DSW1's links, move the labelled cables to the old DSW1,
    power it on, confirm the pre-state, close as rolled back
CLOSE
  Attach the pre and post files to CHG-2041, notify stakeholders,
  watch monitoring until 08:00

Verify

DSW1 output

DSW1# show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
10.255.0.1        0   FULL/  -        00:00:35    10.255.1.0      TenGigabitEthernet1/0/1
10.255.0.2        0   FULL/  -        00:00:38    10.255.1.2      TenGigabitEthernet1/0/2
10.255.0.12       0   FULL/  -        00:00:33    10.255.1.10     Vlan900

Three neighbors in FULL, as before. FULL/ - with priority 0 is normal for point-to-point network type, which has no DR.

DSW1 output

DSW1# show standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Vl10        10   110 P Active  local           10.10.10.3      10.10.10.1
Vl20        20   100 P Standby 10.10.20.3      local           10.10.20.1
Vl30        30   110 P Active  local           10.10.30.3      10.10.30.1
Vl40        40   100 P Standby 10.10.40.3      local           10.10.40.1

Active for 10 and 30, standby for 20 and 40, exactly as in the pre-state file.

DSW1 output

DSW1# show spanning-tree summary | include Root bridge
Root bridge for: VLAN0010, VLAN0030

The STP root and the HSRP active gateway are on the same switch for each VLAN.

DSW1 output

DSW1# show etherchannel summary | begin Group
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP        Te1/0/3(P)  Te1/0/4(P)
11     Po11(SU)        LACP        Te1/0/11(P) Te1/0/12(P)
12     Po12(SU)        LACP        Te1/0/13(P) Te1/0/14(P)

Every bundle is layer 2 and in use (SU), with every member bundled (P).

SRV1 output

SRV1# bash compare.sh
SAME
SAME
SAME

On the file server 10.99.0.20 (SRV1), the three diff lines from the compare step, saved as compare.sh, find no difference in OSPF neighbors, HSRP roles or port channels.

Mistakes people make

Saving the pre-state only on the switch being removed. The files leave with the old hardware.
Staging the new switch with its final HSRP and STP priorities. It grabs the active and root roles the moment its links come up, before you have checked it.
Bringing every link up at once, so a fault cannot be tied to a single change.
No decision time. Without a fixed go or rollback point the window stretches into business hours.
Comparing outputs that contain timers or counters line by line, and calling the change failed because the dead timer moved.
Using reload in as the safety net on a switch that carries all the traffic during the change.

On other platforms

Junos rolls back on its own unless a second commit arrives within the timer.

Junos

configure
set interfaces xe-0/0/1 disable
commit confirmed 5
commit

EOS uses a configuration session with a commit timer, confirmed by committing the session again.

Arista EOS

configure session CHG2041
interface Ethernet1
   shutdown
commit timer 00:05:00
configure session CHG2041 commit

Say it in the interview

"Every cutover I run has the same shape. A header with impact, a go or no-go call, a rollback owner and a fixed decision time. Pre-state captured to files and copied off the box. A safety net, like a config revert timer, on anything that stays. I drain the switch first, so its partner carries everything, then bring the new one up drained and one link group at a time. Then I put the roles back and diff the post-state against the pre-state on the things that matter: neighbors, HSRP, STP roots and bundles. If it is not clean by the decision time, we roll back, no debate."