Octet

Design a CI/CD pipeline for network changes. What runs before and after the change?

Show the answer

Say this first

A merge request triggers lint and data validation, then renders configs and tests them offline (Batfish) or in a virtual lab (containerlab), then runs a dry run against production and posts the diff for review. After approval and inside the change window, deploy runs pre-checks, pushes in batches, runs post-checks that compare state with the pre-check snapshot, and rolls back if they fail. Every step leaves logs on the ticket.

Go deeper

Stages

  1. Lint: yamllint, ansible-lint, Python linters. Catches syntax before anything runs.
  2. Validate data: schema checks and rules (no duplicate VLAN IDs, every IP inside its prefix, names follow the standard).
  3. Render and test: build every device config from templates. Batfish loads them and answers questions: undefined references, ACL behavior, BGP sessions that will not come up, reachability.
  4. Lab: optional containerlab topology with virtual images to test the playbook end to end.
  5. Dry run: --check --diff or NAPALM compare_config against production, posted to the merge request.
  6. Approve and deploy: manual gate, change window, batches.
  7. Post-checks and rollback: compare with the pre-check snapshot; on failure, roll back automatically.

Pre-checks and post-checks

Capture: interfaces up, BGP and OSPF neighbors and prefix counts, route table size, MAC and ARP counts, STP root, CPU. Pre-checks also act as a gate: do not change a device that is already unhealthy. pyATS can learn features before and after and diff them; NAPALM getters and compliance_report work too.

GitLab CI example

YAML
stages: [lint, test, plan, deploy]

lint:
  stage: lint
  image: python:3.12
  script:
    - pip install yamllint ansible-lint
    - yamllint .
    - ansible-lint playbooks/

validate-data:
  stage: test
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - python tests/check_data.py

plan:
  stage: plan
  script:
    - ansible-playbook playbooks/vlans.yml --check --diff
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

deploy:
  stage: deploy
  script:
    - ansible-playbook playbooks/precheck.yml
    - ansible-playbook playbooks/vlans.yml
    - ansible-playbook playbooks/postcheck.yml
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual
  environment: production

The plan and deploy jobs run on a runner inside the management network with access to devices and Vault.

Traps

  • A pipeline with no post-checks. A push that "succeeded" can still have broken the network.
  • Running the deploy job from a shared cloud runner that cannot, and should not, reach production devices.
How did you do?