Design a CI/CD pipeline for network changes. What runs before and after the change?
Show the answer
Say this first
A merge request triggers lint and data validation, then renders configs and tests them offline (Batfish) or in a virtual lab (containerlab), then runs a dry run against production and posts the diff for review. After approval and inside the change window, deploy runs pre-checks, pushes in batches, runs post-checks that compare state with the pre-check snapshot, and rolls back if they fail. Every step leaves logs on the ticket.
Go deeper
Stages
- Lint:
yamllint,ansible-lint, Python linters. Catches syntax before anything runs. - Validate data: schema checks and rules (no duplicate VLAN IDs, every IP inside its prefix, names follow the standard).
- Render and test: build every device config from templates. Batfish loads them and answers questions: undefined references, ACL behavior, BGP sessions that will not come up, reachability.
- Lab: optional containerlab topology with virtual images to test the playbook end to end.
- Dry run:
--check --diffor NAPALMcompare_configagainst production, posted to the merge request. - Approve and deploy: manual gate, change window, batches.
- Post-checks and rollback: compare with the pre-check snapshot; on failure, roll back automatically.
Pre-checks and post-checks
Capture: interfaces up, BGP and OSPF neighbors and prefix counts, route table size, MAC and ARP counts,
STP root, CPU. Pre-checks also act as a gate: do not change a device that is already unhealthy.
pyATS can learn features before and after and diff them; NAPALM getters and compliance_report
work too.
GitLab CI example
stages: [lint, test, plan, deploy]
lint:
stage: lint
image: python:3.12
script:
- pip install yamllint ansible-lint
- yamllint .
- ansible-lint playbooks/
validate-data:
stage: test
image: python:3.12
script:
- pip install -r requirements.txt
- python tests/check_data.py
plan:
stage: plan
script:
- ansible-playbook playbooks/vlans.yml --check --diff
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
deploy:
stage: deploy
script:
- ansible-playbook playbooks/precheck.yml
- ansible-playbook playbooks/vlans.yml
- ansible-playbook playbooks/postcheck.yml
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: manual
environment: productionThe plan and deploy jobs run on a runner inside the management network with access to devices and Vault.
They will ask next
What does Batfish do that a lab cannot?
It analyzes the configs of the whole network mathematically, without running devices, so it can answer "can anything in VLAN 30 reach the PCI subnet" for every possible packet. A lab only tests the packets you send, but it does run the real software, so the two complement each other.
Where do you draw the line between CI and CD for network changes?
CI (lint, validate, render, test, dry run) runs on every merge request automatically. CD to production usually keeps a human approval and a change window, at least until the team trusts the checks enough for low-risk change types to deploy on merge.
Traps
- A pipeline with no post-checks. A push that "succeeded" can still have broken the network.
- Running the deploy job from a shared cloud runner that cannot, and should not, reach production devices.