You need to push a change to 800 switches with automation. How do you make it safe?
Show the answer
Say this first
I treat it as a staged rollout. Pre-checks gate each device, a dry run diff is reviewed, and the change goes to one or two canary devices first, then a small batch, then the rest, stopping on the first failure. Where the platform has it I use confirmed commit so a change that cuts my access rolls itself back. Post-checks run after every batch, and I keep out-of-band access, a tested rollback and a kill switch.
Go deeper
Before
- Peer-reviewed merge request with a dry run diff for a sample of device types.
- Tested on a lab device with the same platform and release.
- Pre-checks: device reachable, healthy, config matches the expected baseline. Skip and report devices that fail.
- Rollback written and tested, not just "re-run the old playbook".
During
- name: Push new AAA config
hosts: access_switches
gather_facts: false
serial: [1, 5, "10%", "25%"]
max_fail_percentage: 0
tasks:
- name: Apply template
cisco.ios.ios_config:
src: aaa.j2
save_when: modifiedserial grows the batches, and max_fail_percentage: 0 stops the rollout as soon as any device in a
batch fails. forks limits how many devices are touched at once, which also protects TACACS servers.
Safety nets per platform
- Junos and NETCONF candidate:
commit confirmedwith a few minutes timeout. - IOS-XE with
archiveconfigured:configure terminal revert timerthenconfigure confirm. - Changes that touch management access (AAA, ACLs on vty, management VRF) go last and slowest.
After
Post-checks per batch, compare with pre-checks, then a final report: changed, unchanged, failed, skipped.
They will ask next
What would make you stop the rollout halfway?
Any failed post-check, any device that does not come back reachable, or a jump in alarms from monitoring. Stopping is cheap; the tooling should make it the default when anything is unexpected.
Traps
- Running against all 800 at once because "it worked in the lab".
- Changing AAA or management ACLs without a confirmed-commit or console fallback, and locking yourself out everywhere.