Automation and SDN

What it fixes

• Human error. Typing the same 40 lines on 200 devices guarantees a typo somewhere. A template does not get tired.
• Drift. Hand-made changes slowly make every device a little different. Automation lets you compare every device against one intended state.
• Speed and scale. The team stops being the bottleneck for routine requests.
• Audit and compliance. Changes are in Git with an author, a reviewer and a ticket number.
• Faster troubleshooting. A script collects show output from 300 devices in a minute during an incident.

A sensible order

1. Read-only. Nightly backups, inventory, "which devices still run this old image", compliance reports. Nothing can break, and the team learns the tools.
2. Standard config. NTP, syslog, SNMP, AAA, banners. Low risk, high repetition.
3. Service requests. VLANs, port configuration, ACL entries, driven from a source of truth.
4. Full provisioning and closed loop. Day 0 builds from templates, then continuous checks that the network still matches intent.

What I would say about risk

Automation also multiplies mistakes. A bad template reaches every device as fast as a good one. So the process matters as much as the tool: peer review, a dry run with a diff, a canary device first, and a rollback plan.

Side by side

Why the split matters

• Protection. The CPU is small compared to the ASICs. A flood of traffic aimed at the router itself (punted traffic) can starve OSPF and BGP. Control plane policing (CoPP) rate limits what reaches the CPU.
• SDN. Classic SDN moves the control plane off the box into a central controller. Most modern controllers keep a distributed control plane on the devices and centralize policy and management instead.
• Resilience. With NSF/SSO, the data plane keeps forwarding on the old FIB while the control plane restarts.

Quick test

A ping through R1 is data plane. A ping to R1 is punted to the CPU and answered there. An OSPF hello is control plane. An SSH login is management plane.

Three flavors you will hear about

• Open SDN. A controller programs flow tables in switches with OpenFlow. Logic is fully central.
• SDN through APIs. Devices keep their own control plane. A controller or script configures them through NETCONF, RESTCONF, gNMI or vendor APIs.
• Overlay SDN. A virtual network runs on top of an IP underlay with VXLAN or similar tunnels. Hypervisor switches or leaf VTEPs build the tunnels, and the physical network just routes IP.

What a modern controller still centralizes

• One place to declare policy (who may talk to whom).
• Automation of device config from that policy.
• A network-wide view for assurance and troubleshooting.
• A northbound REST API so other systems can drive the network.

What stays distributed

In ACI the fabric still runs IS-IS in the underlay, COOP on the spines for endpoint mapping and MP-BGP for external routes. If every APIC goes down, the fabric keeps forwarding. You just cannot change policy.

Picture it as a stack

ITSM, orchestrator, your Python script
        |  northbound: REST API, JSON, webhooks
    Controller (Catalyst Center, APIC, SD-WAN Manager)
        |  southbound: NETCONF, RESTCONF, gNMI, SNMP, SSH, OpenFlow, OpFlex
    Switches, routers, firewalls, APs

Examples

East-west

Some people also name an east-west interface: controllers talking to each other in a cluster or between domains, for example to sync state or federate policy.

The constraints (Roy Fielding, 2000)

• Client-server. The client and server evolve separately.
• Stateless. No session context is kept between requests. Each call carries its token or credentials.
• Cacheable. Responses say whether they can be cached.
• Uniform interface. Resources named by URIs, changed through representations, self-descriptive messages.
• Layered system. Proxies and load balancers can sit in between without the client knowing.
• Code on demand is optional and rare in network APIs.

Anatomy of a request

GET /dna/intent/api/v1/network-device?hostname=SW1 HTTP/1.1
Host: catc.example.com
Accept: application/json
X-Auth-Token: eyJ0eXAiOiJKV1Qi...

Method, path (resource), query string (filter), headers, and for POST, PUT and PATCH a body.

Anatomy of a response

Status line (HTTP/1.1 200 OK), headers such as Content-Type: application/json, and a body.

The table

Safe means it does not change state. Idempotent means repeating it has the same effect on the server as doing it once. The response can differ: a second DELETE may return 404, but the resource is still gone, so DELETE is idempotent.

The network trap with PUT

PUT replaces the target. If you PUT an interface with only a new description, a RESTCONF server replaces that whole interface entry with what you sent, and leaves you without the IP address you left out. Use PATCH (merge) to change one field.

Why automation cares

Networks are unreliable. If a POST times out you do not know whether the object was created, and a retry may create a duplicate. With PUT you can retry until it succeeds. The same idea is why Ansible modules are written to be idempotent.

The codes to know cold

The classes

1xx informational, 2xx success, 3xx redirect (301, 302, 304), 4xx client error, 5xx server error.

RESTCONF specifics (RFC 8040)

Errors come back with a body that holds error-type, error-tag and error-message. For example data-exists and lock-denied both map to 409.

The same data three ways

{
  "vlans": [
    {"id": 10, "name": "USERS"},
    {"id": 20, "name": "VOICE"}
  ]
}

# VLANs for building A
vlans:
  - id: 10
    name: USERS
  - id: 20
    name: VOICE

<vlans xmlns="urn:example:vlans">
  <vlan><id>10</id><name>USERS</name></vlan>
  <vlan><id>20</id><name>VOICE</name></vlan>
</vlans>

Differences that come up

YAML 1.2 is a superset of JSON, so a JSON document is almost always valid YAML.

A small model

module example-vlans {
  namespace "urn:example:vlans";
  prefix exv;

  container vlans {
    list vlan {
      key "id";
      leaf id {
        type uint16 {
          range "1..4094";
        }
      }
      leaf name {
        type string;
      }
      leaf-list tagged-ports {
        type string;
      }
    }
  }
}

The building blocks

Why engineers care

The model tells you the path and type before you send anything. The device validates input against it, so id: 5000 fails cleanly instead of half applying. And the same model works over NETCONF, RESTCONF and gNMI.

The layers

get vs get-config

get-config returns config only, from the datastore you name (running, candidate, startup). get returns config plus state data from the running system. Both accept a subtree or XPath filter.

A get-config with a subtree filter

<rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get-config>
    <source><running/></source>
    <filter type="subtree">
      <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
        <interface><name>GigabitEthernet1</name></interface>
      </interfaces>
    </filter>
  </get-config>
</rpc>

The reply carries the same message-id and the data inside <data>.

Framing

With base:1.0, each message ends with ]]>]]>. When both sides announce base:1.1, they switch to chunked framing, which cannot be broken by that string appearing inside data.

Turning it on

R1(config)# netconf-yang

set system services netconf ssh

Quick test from a shell: ssh -p 830 netops@10.0.0.1 -s netconf prints the server hello.

The pieces

• Inventory: hosts and groups, in INI or YAML, static or dynamic (for example from NetBox).
• Variables: group_vars/ and host_vars/, plus secrets in Ansible Vault.
• Playbook: a YAML list of plays. A play targets hosts and runs tasks in order.
• Modules: do the work, ideally idempotently. Network modules live in collections.
• Roles: reusable bundles of tasks, templates and defaults.

Inventory

all:
  children:
    ios:
      hosts:
        SW1:
          ansible_host: 10.0.0.11
        SW2:
          ansible_host: 10.0.0.12
      vars:
        ansible_connection: ansible.netcommon.network_cli
        ansible_network_os: cisco.ios.ios
        ansible_user: netops
        ansible_password: "{{ vault_netops_password }}"
        ansible_become: true
        ansible_become_method: enable

A first playbook

- name: Collect versions
  hosts: ios
  gather_facts: false
  tasks:
    - name: Run show version
      cisco.ios.ios_command:
        commands:
          - show version
      register: ver

    - name: Print the first line
      ansible.builtin.debug:
        msg: "{{ ver.stdout_lines[0][0] }}"

gather_facts: false is common in network plays; call cisco.ios.ios_facts when you need facts.

How you see it

$ ansible-playbook ntp.yml
TASK [Set NTP server] **********************************************************
changed: [SW1]

PLAY RECAP *********************************************************************
SW1                        : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

$ ansible-playbook ntp.yml
TASK [Set NTP server] **********************************************************
ok: [SW1]

PLAY RECAP *********************************************************************
SW1                        : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

How modules achieve it

• ios_config reads the running config and compares your lines (under their parents) with it. It sends only lines that are missing.
• Resource modules (ios_vlans, ios_interfaces) gather the current state as structured data and compute the commands needed.
• ios_command runs commands. It is for reading, not for managing config.

What breaks idempotency with ios_config

• Lines the device never shows: no shutdown is the default and does not appear in the running config, so it is "missing" on every run.
• Abbreviations: int gi1/0/1 or desc never match the full words in the running config.
• Lines the device rewrites: secrets are stored hashed, so the plain text never matches.

Check mode and diff

ansible-playbook vlans.yml --check --diff --limit SW1

Run it in the change ticket and in CI. A playbook that shows changed in check mode on an unchanged device has an idempotency bug.

A minimal script

from netmiko import ConnectHandler

device = {
    "device_type": "cisco_ios",
    "host": "10.0.0.11",
    "username": "netops",
    "password": "secret",
    "secret": "enablepass",
}

with ConnectHandler(**device) as conn:
    conn.enable()
    print(conn.send_command("show ip interface brief"))
    output = conn.send_config_set([
        "interface Loopback100",
        "description mgmt-test",
    ])
    print(output)
    conn.save_config()

The methods you use most

Common device types

cisco_ios, cisco_xe, cisco_nxos, cisco_xr, arista_eos, juniper_junos, linux.

Timeouts

In Netmiko 4, send_command waits up to read_timeout (10 seconds by default) for the prompt. A big show running-config on a core switch can need read_timeout=60.

Template

hostname {{ hostname }}
{% for server in ntp_servers %}
ntp server {{ server }}
{% endfor %}
{% if snmp_location is defined %}
snmp-server location {{ snmp_location }}
{% endif %}

Data

hostname: ASW1
ntp_servers:
  - 10.0.0.5
  - 10.0.0.6
snmp_location: Building A floor 2

Render in Python

import yaml
from jinja2 import Environment, FileSystemLoader, StrictUndefined

env = Environment(
    loader=FileSystemLoader("templates"),
    trim_blocks=True,
    lstrip_blocks=True,
    undefined=StrictUndefined,
)
template = env.get_template("base.j2")
with open("data/asw1.yml") as f:
    data = yaml.safe_load(f)
print(template.render(**data))

hostname ASW1
ntp server 10.0.0.5
ntp server 10.0.0.6
snmp-server location Building A floor 2

Things that trip people

• Whitespace. Without trim_blocks and lstrip_blocks, every {% %} line leaves a blank line or indentation. You can also trim per tag with {%- and -%}.
• Undefined variables. The default prints an empty string, which can produce ntp server with no address. StrictUndefined makes it fail instead. Ansible fails on undefined variables by default.
• Logic creep. Keep templates simple and push decisions into the data.

In Ansible, the ansible.builtin.template module renders a .j2 file, and modules such as ios_config accept rendered config through src.

What NetBox models

• DCIM: regions, sites, locations, racks, device types, devices, interfaces, cables, power.
• IPAM: VRFs, prefixes, IP addresses, VLANs and VLAN groups, ASNs, route targets.
• Circuits and providers, virtualization, tenancy.
• Extensibility: custom fields, tags, config contexts (JSON data merged by site, role or platform), config templates (Jinja2), webhooks and event rules, plugins.

Reading it from Python

import os
import pynetbox

nb = pynetbox.api("https://netbox.example.com", token=os.environ["NETBOX_TOKEN"])
for dev in nb.dcim.devices.filter(site="dc1", status="active"):
    print(dev.name, dev.primary_ip4)

As Ansible inventory

# nb_inventory.yml
plugin: netbox.netbox.nb_inventory
api_endpoint: https://netbox.example.com
validate_certs: true
group_by:
  - sites
  - platforms
query_filters:
  - status: active

The token comes from the NETBOX_TOKEN environment variable.

Intended vs actual

NetBox holds intended state. The devices hold actual state. Automation turns one into the other, and drift checks compare them. NetBox is not a monitoring system and does not discover the network by itself.

The everyday flow

git clone git@git.example.com:netops/network.git
cd network
git switch -c CHG1234-add-vlan-30
# edit data/sites/dc1/vlans.yml
git add data/sites/dc1/vlans.yml
git commit -m "CHG1234: add VLAN 30 GUESTS to dc1 leaves"
git push -u origin CHG1234-add-vlan-30

Then open a merge request. CI lints, renders and dry-runs the change, a reviewer approves, and it merges.

Commands worth knowing

What goes in, what stays out

In: templates, data, playbooks, scripts, tests, pipeline files. Out: secrets (use Vault), generated files you can rebuild, and huge binary images. Config backups usually live in a separate repo.

The pieces

• Flow entry. Match on ingress port, MACs, EtherType, VLAN, IP addresses, protocol, L4 ports (OpenFlow 1.0 had 12 match fields; later versions added many more). Highest priority match wins.
• Actions. Output to a port, drop, rewrite a header field, push or pop a VLAN tag, send to the controller.
• Table-miss. A priority 0 entry that drops or sends to the controller.
• Messages. Hello, Features Request and Reply, Flow-Mod, Packet-In, Packet-Out, Port-Status, Echo.

Reactive vs proactive

• Reactive. The first packet of a new flow goes to the controller, which installs an entry. Flexible, but every new flow costs a round trip, and the controller becomes the bottleneck.
• Proactive. The controller installs entries in advance from known topology and policy. Scales far better and survives a controller outage.

Versions

OpenFlow 1.1 and later added multiple tables in a pipeline and group tables. 1.3 added meters and became the most widely implemented version. The old controller port was 6633; IANA later assigned 6653.

Where it lived on

Open vSwitch in hypervisors and OpenStack, Google's B4 inter-datacenter WAN, and research. Its ideas moved into P4, programmable pipelines and controller-based fabrics.

$ ovs-ofctl dump-flows br0
 cookie=0x0, duration=120.345s, table=0, n_packets=42, n_bytes=3528, priority=100,ip,nw_dst=10.0.0.2 actions=output:2
 cookie=0x0, duration=300.012s, table=0, n_packets=7, n_bytes=420, priority=0 actions=CONTROLLER:65535

Catalyst Center

• Design: sites, buildings, floors, IP pools, global settings (DNS, NTP, AAA).
• Policy: group-based policy with ISE and SGTs for SD-Access.
• Provision: templates, PnP zero-touch onboarding, SWIM for image upgrades.
• Assurance: health scores, client and device telemetry, issue detection.
• SD-Access fabric: LISP control plane, VXLAN data plane, TrustSec (SGT) policy plane.
• API: get a token from POST /dna/system/api/v1/auth/token with basic auth, then send it in the X-Auth-Token header, for example GET /dna/intent/api/v1/network-device.

ACI APIC

• Fabric: spine-leaf Nexus 9000 in ACI mode, APIC cluster (three or more in production).
• Object model: tenant contains VRFs, bridge domains and application profiles with EPGs. Contracts allow traffic between EPGs. With no contract, EPGs cannot talk (allow-list model).
• Internals: IS-IS underlay, VXLAN overlay, COOP on spines for endpoint lookup, MP-BGP to distribute external routes.
• API: POST /api/aaaLogin.json returns a token (also set as the APIC-cookie cookie). Query a class with GET /api/class/fvTenant.json or one object with GET /api/mo/uni/tn-Prod.json. The GUI's API Inspector shows the exact call behind every click.

Others worth naming

Cisco SD-WAN Manager and Controller, Meraki Dashboard (cloud managed), Arista CloudVision, Juniper Apstra.

The loop

1. Translation. Business or service intent becomes a network policy and then device config.
2. Activation. The controller pushes config to every device involved.
3. Assurance. Telemetry and tests confirm the network behaves as intended, all the time.
4. Remediation. When reality drifts from intent, the system alerts or corrects.

Products that use the term

• Cisco Catalyst Center with SD-Access: group-based policies between user and device groups, assurance on top.
• Juniper Apstra: intent for data center fabrics, with continuous validation of the built fabric against the blueprint.
• Cisco ACI: the policy model (EPGs and contracts) is a form of declared intent.

Honest view

The term is marketing-heavy. The useful part is real: declare the desired state, generate config from it, and verify continuously instead of trusting that a push worked.

Examples

A clean Python pattern

import os
import requests

BASE = "https://catc.example.com"
CA = "/etc/ssl/certs/corp-ca.pem"

resp = requests.post(
    f"{BASE}/dna/system/api/v1/auth/token",
    auth=(os.environ["CATC_USER"], os.environ["CATC_PASS"]),
    verify=CA,
    timeout=10,
)
resp.raise_for_status()
token = resp.json()["Token"]

devices = requests.get(
    f"{BASE}/dna/intent/api/v1/network-device",
    headers={"X-Auth-Token": token, "Accept": "application/json"},
    verify=CA,
    timeout=30,
)
devices.raise_for_status()
for dev in devices.json()["response"]:
    print(dev["hostname"], dev["managementIpAddress"])

Habits that matter

• Credentials come from environment variables, Ansible Vault or a secrets manager, never from the repo.
• verify=False is a lab shortcut. In production point verify at the company CA bundle.
• Tokens expire. A long job must refresh or re-authenticate on 401 instead of failing halfway.
• Use a dedicated service account with only the rights the job needs, so the AAA logs show what the automation did.

Tree output for ietf-interfaces (trimmed)

$ pyang -f tree ietf-interfaces.yang
module: ietf-interfaces
  +--rw interfaces
  |  +--rw interface* [name]
  |     +--rw name                        string
  |     +--rw description?                string
  |     +--rw type                        identityref
  |     +--rw enabled?                    boolean
  |     +--rw link-up-down-trap-enable?   enumeration {if-mib}?
  |     +--ro admin-status                enumeration {if-mib}?
  |     +--ro oper-status                 enumeration
  |     +--ro last-change?                yang:date-and-time
  |     +--ro if-index                    int32 {if-mib}?
  |     +--ro phys-address?               yang:phys-address
  |     +--ro speed?                      yang:gauge64
  |     +--ro statistics
  |        +--ro discontinuity-time    yang:date-and-time
  |        +--ro in-octets?            yang:counter64
  ...
  x--ro interfaces-state
  ...

How to read it

• interface* [name] is a list keyed by name.
• description? is optional config. type has no ?, so it is mandatory when you create an interface.
• {if-mib}? means the node exists only if the server supports the if-mib feature.
• x marks a deprecated node: interfaces-state was replaced by the ro nodes inside interfaces.

Turning it into paths

Where models come from

The device itself (NETCONF get-schema, RFC 6022), the vendor's published model set for that release, and the public YangModels collection that holds IETF, OpenConfig and vendor models by release.

Side by side

What the paths look like

IETF:        /interfaces/interface[name=GigabitEthernet1]/description
OpenConfig:  /interfaces/interface[name=Ethernet1]/config/description
             /interfaces/interface[name=Ethernet1]/state/counters/in-octets
IOS-XE:      /native/interface/GigabitEthernet[name=1]/description

Cisco operational models

IOS-XE also ships -oper models for state, for example Cisco-IOS-XE-interfaces-oper and Cisco-IOS-XE-process-cpu-oper. IOS XR splits native models into -cfg and -oper.

The practical rule

A multi-vendor shop gets the most from OpenConfig, especially for telemetry. But vendors implement OpenConfig at different depths and publish deviations, so test each model on each platform and release before you build on it.

Datastores

edit-config operations

default-operation (merge, replace or none) sets the behavior for nodes without an attribute. error-option is stop-on-error (default), continue-on-error or rollback-on-error (needs the :rollback-on-error capability).

Deleting one interface description

<rpc message-id="105" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <edit-config>
    <target><candidate/></target>
    <config>
      <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
        <interface>
          <name>GigabitEthernet2</name>
          <description nc:operation="remove"/>
        </interface>
      </interfaces>
    </config>
  </edit-config>
</rpc>

Confirmed commit

<commit><confirmed/><confirm-timeout>120</confirm-timeout></commit> applies the candidate and starts a timer. If your session drops or nobody sends a plain <commit/> before the timer ends, the device reverts. That is the safety net for changes that might cut your own access. Junos users know the same idea as commit confirmed.

Platform notes

IOS-XE needs netconf-yang feature candidate-datastore to offer a candidate. Junos maps NETCONF candidate to its normal candidate configuration.

Anatomy of a URI

https://10.0.0.1/restconf/data/ietf-interfaces:interfaces/interface=GigabitEthernet2/description

• The root is found at /.well-known/host-meta; on most devices it is /restconf.
• /restconf/data is the datastore, /restconf/operations lists RPCs.
• Multiple keys are separated by commas. Reserved characters are percent-encoded: / becomes %2F.

Enable on IOS-XE

username netops privilege 15 secret 0 S3cure.pass
aaa new-model
aaa authentication login default local
aaa authorization exec default local
ip http secure-server
restconf

Read one interface

curl -s -k -u netops:S3cure.pass \
  -H "Accept: application/yang-data+json" \
  https://10.0.0.1/restconf/data/ietf-interfaces:interfaces/interface=GigabitEthernet2

{
  "ietf-interfaces:interface": {
    "name": "GigabitEthernet2",
    "description": "to SW1 Gi1/0/48",
    "type": "iana-if-type:ethernetCsmacd",
    "enabled": true,
    "ietf-ip:ipv4": {
      "address": [
        {
          "ip": "10.1.12.1",
          "netmask": "255.255.255.0"
        }
      ]
    },
    "ietf-ip:ipv6": {
    }
  }
}

Note the module: prefix on the top member and wherever a node comes from another module (ietf-ip:ipv4).

Useful query parameters

depth=2 limits how deep the reply goes, content=config or content=nonconfig picks config or state, fields=name;description returns only some leafs, with-defaults controls default values.

Side by side

How I choose

• Risky config changes on many devices: NETCONF with candidate and confirmed commit, where supported.
• A quick integration, a webhook, a ServiceNow action: RESTCONF, because every language and tool can do HTTPS and JSON.
• Monitoring at scale: gNMI Subscribe, often with OpenConfig paths across vendors.
• Platform reality: use what the device does well. Some platforms are strongest on one interface.

Paths

gNMI paths are XPath-like with keys in brackets and no quotes: /interfaces/interface[name=Ethernet1]/state/counters/in-octets. An optional origin (often openconfig) says which model tree the path belongs to.

Subscribe in practice

gnmic -a 10.0.0.31:6030 -u netops -p secret --insecure capabilities

gnmic -a 10.0.0.31:6030 -u netops -p secret --insecure \
  subscribe --path "/interfaces/interface[name=Ethernet1]/state/counters" \
  --stream-mode sample --sample-interval 10s

gnmic -a 10.0.0.31:6030 -u netops -p secret --insecure \
  subscribe --path "/interfaces/interface/state/oper-status" --stream-mode on-change

--insecure is plaintext gRPC for a lab. With TLS use --tls-ca to verify the device certificate, or --skip-verify to encrypt without checking it.

Which mode for what

suppress_redundant and heartbeat_interval let SAMPLE skip unchanged values while still proving the stream is alive.

Related gRPC services

gNOI for operations (reboot, file transfer, certificates, ping), gRIBI for injecting routes, P4Runtime for programmable forwarding pipelines.

Enabling it

management api gnmi
   transport grpc default

Other platforms have their own commands and default ports; check the release notes.

What changes

IOS-XE dial-out example

telemetry ietf subscription 101
 encoding encode-kvgpb
 filter xpath /process-cpu-ios-xe-oper:cpu-usage/cpu-utilization/five-seconds
 source-address 10.0.0.1
 stream yang-push
 update-policy periodic 6000
 receiver ip address 10.0.0.50 57000 protocol grpc-tcp

update-policy periodic is in centiseconds, so 6000 is every 60 seconds. update-policy on-change works for paths that support it. Newer releases also offer named receivers, configured separately.

A common collector stack

Telegraf receives the data (the cisco_telemetry_mdt input for Cisco dial-out, the gnmi input for dial-in), writes it to InfluxDB or Prometheus, and Grafana draws it:

[[inputs.cisco_telemetry_mdt]]
  transport = "grpc"
  service_address = ":57000"

Encodings

Compact GPB needs the matching .proto files on the collector. KV-GPB carries field names with the values, so it is self-describing at the cost of size. JSON is easiest to read and largest.

Group variables per platform

# group_vars/ios.yml
ansible_connection: ansible.netcommon.network_cli
ansible_network_os: cisco.ios.ios
ansible_become: true
ansible_become_method: enable

# group_vars/eos.yml
ansible_connection: ansible.netcommon.httpapi
ansible_network_os: arista.eos.eos
ansible_httpapi_use_ssl: true
ansible_httpapi_validate_certs: true
ansible_become: true
ansible_become_method: enable

# group_vars/junos.yml
ansible_connection: ansible.netcommon.netconf
ansible_network_os: junipernetworks.junos.junos

Persistent connections

network_cli, netconf and httpapi keep one connection per host open across tasks, instead of logging in for every task. Two timers matter: the connect timeout and the command timeout, both 30 seconds by default. A slow show tech or a big config needs a larger command timeout.

The old way

Older playbooks used connection: local with a provider dictionary on each task. That pattern is deprecated; set the connection in inventory variables instead.

Picking one

The states

Example

- name: Access VLANs
  cisco.ios.ios_vlans:
    config:
      - vlan_id: 10
        name: USERS
      - vlan_id: 20
        name: VOICE
      - vlan_id: 30
        name: GUESTS
    state: merged

The result holds before, after and commands. If only VLAN 30 is new, commands is ["vlan 30", "name GUESTS"].

Why they beat ios_config lines

The module understands the feature, so it knows that shutdown is the opposite of no shutdown, removes attributes correctly in replaced, and stays idempotent without you matching running-config text.

Ansible Vault

ansible-vault create group_vars/all/vault.yml
ansible-vault encrypt_string 'S3cure.pass' --name 'vault_netops_password'
ansible-playbook site.yml --ask-vault-pass
ansible-playbook site.yml --vault-password-file ~/.vault_pass

A common pattern keeps vault_ variables in an encrypted file and refers to them from normal variables: ansible_password: "{{ vault_netops_password }}". People can read the variable names in reviews without seeing the values.

Other places secrets live

• Environment variables for scripts, set by the CI system or a wrapper, not hard-coded.
• Secrets managers (HashiCorp Vault, CyberArk, cloud key vaults) with short-lived credentials and audit logs.
• CI variables marked protected and masked so they only reach protected branches and never print in logs.

Accounts

• A separate account per automation system, not a shared admin login.
• Read-only accounts for collection jobs (backups, inventory).
• AAA accounting on, so the logs show which account changed what.

Getters

get_facts, get_interfaces, get_interfaces_ip, get_arp_table, get_mac_address_table, get_lldp_neighbors, get_bgp_neighbors, get_environment, get_config, get_route_to and more. Not every driver implements every getter; unsupported ones raise NotImplementedError.

Config workflow

from napalm import get_network_driver

driver = get_network_driver("eos")
with driver("10.0.0.21", "netops", "secret") as dev:
    print(dev.get_facts()["os_version"])
    dev.load_merge_candidate(config="ntp server 10.0.0.5\n")
    diff = dev.compare_config()
    if diff:
        print(diff)
        dev.commit_config()
    else:
        dev.discard_config()

Merge vs replace

How it works under the hood

Each driver uses the platform's best mechanism: Junos has a native candidate and commit, EOS uses configuration sessions, IOS XR has its own commit model. IOS has no candidate, so the driver copies the candidate file to flash with SCP and uses configure replace for a replace. That is why IOS needs ip scp server enable for NAPALM config operations.

Validation

compliance_report("validate.yml") compares getter results with expected values in a YAML file, for example "these BGP peers are up". It is a quick post-check.

A small example (ACI tenant)

terraform {
  required_providers {
    aci = {
      source = "CiscoDevNet/aci"
    }
  }
}

variable "apic_password" {
  type      = string
  sensitive = true
}

provider "aci" {
  username = "admin"
  password = var.apic_password
  url      = "https://apic1.example.com"
}

resource "aci_tenant" "prod" {
  name        = "Prod"
  description = "Managed by Terraform"
}

The workflow

terraform init               # download providers, set up the backend
terraform plan -out tfplan   # compare code, state and reality
terraform apply tfplan       # apply exactly that plan
terraform state list         # resources Terraform manages

State rules

• Remote backend (S3, Azure Storage, GCS, HCP Terraform, GitLab) with locking, so two people cannot apply at once.
• Treat it as sensitive: attributes, including some passwords, are stored in plain text.
• Change it only with terraform state mv, state rm and import (or import blocks).
• If someone changes a managed object by hand, the next plan shows the drift and offers to put it back.

Terraform vs Ansible

Since version 1.6 Terraform uses the Business Source License; OpenTofu is the open source fork with the same workflow.

Where the differences hide

• Deviations. A vendor module that says, for example:

  deviation "/oc-if:interfaces/oc-if:interface/oc-if:state/oc-if:counters/oc-if:in-fcs-errors" {
    deviate not-supported;
  }

  Other forms are deviate add, replace and delete to change defaults, types or constraints.
• Features. if-feature makes nodes optional. The server lists the features it implements.
• Revisions. The same module name with a different revision can rename or move nodes.
• Semantics. Two implementations can accept the same leaf but apply it differently, for example when an interface needs a sub-interface 0 before an IP address is valid.

How the device tells you

NETCONF 1.0-style servers list each module in the hello as a capability URI:

urn:ietf:params:xml:ns:yang:ietf-interfaces?module=ietf-interfaces&revision=2014-05-08&features=pre-provisioning,if-mib,arbitrary-names

A deviations= parameter, when present, names the vendor deviation modules. Newer servers point to the YANG library (ietf-yang-library, RFC 7895 and RFC 8525), which lists modules, revisions, features and deviations as data you can query. gNMI returns supported models in the Capabilities response.

How I protect the code

• Pin the platform and release in the test lab to the ones in production.
• Gather capabilities as part of pre-checks and fail fast if a required module or revision is missing.
• Keep per-platform translation in one place instead of scattering if vendor == through the code.

Step by step

1. Transport. SSH to TCP 830, subsystem netconf. Authentication is normal SSH (password or key, often via AAA).
2. Hello. The server sends its capabilities and a session-id; the client sends its own. If both list base:1.1, framing switches to chunked.
3. Lock the candidate (and running, if the platform asks for it).
4. edit-config to the candidate.
5. validate the candidate (with :validate).
6. commit with <confirmed/> and <confirm-timeout>.
7. Check reachability and state from the automation side.
8. Confirming commit (a plain <commit/>), then unlock and close-session.

The same flow with ncclient

from ncclient import manager

CONFIG = """
<config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
    <interface>
      <name>GigabitEthernet2</name>
      <description>to SW1 Gi1/0/48</description>
    </interface>
  </interfaces>
</config>
"""

with manager.connect(host="10.0.0.1", port=830, username="netops",
                     password="secret", hostkey_verify=False) as m:
    with m.locked(target="candidate"):
        m.discard_changes()
        m.edit_config(target="candidate", config=CONFIG)
        m.validate(source="candidate")
        m.commit(confirmed=True, timeout="120")
        # run post-checks here; if they fail, do nothing and the device rolls back
        m.commit()

discard_changes() first clears anything another session left in a shared candidate. hostkey_verify=False is for a lab; in production load known host keys.

What an error looks like

<rpc-reply message-id="102" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <rpc-error>
    <error-type>protocol</error-type>
    <error-tag>lock-denied</error-tag>
    <error-severity>error</error-severity>
    <error-info>
      <session-id>31</session-id>
    </error-info>
  </rpc-error>
</rpc-reply>

error-type is transport, rpc, protocol or application. For lock-denied, error-info names the session holding the lock. The error-message text varies by vendor.

Common failures

• Port 830 refused: NETCONF not enabled, or an ACL on the management plane.
• MissingCapabilityError in ncclient: you targeted the candidate on a device that does not offer it.
• unknown-element or invalid-value: wrong namespace, a typo, or a node the platform does not support.
• lock-denied: another session or a stuck session holds the lock; kill-session frees it if you are sure.

Stages

1. Lint: yamllint, ansible-lint, Python linters. Catches syntax before anything runs.
2. Validate data: schema checks and rules (no duplicate VLAN IDs, every IP inside its prefix, names follow the standard).
3. Render and test: build every device config from templates. Batfish loads them and answers questions: undefined references, ACL behavior, BGP sessions that will not come up, reachability.
4. Lab: optional containerlab topology with virtual images to test the playbook end to end.
5. Dry run: --check --diff or NAPALM compare_config against production, posted to the merge request.
6. Approve and deploy: manual gate, change window, batches.
7. Post-checks and rollback: compare with the pre-check snapshot; on failure, roll back automatically.

Pre-checks and post-checks

Capture: interfaces up, BGP and OSPF neighbors and prefix counts, route table size, MAC and ARP counts, STP root, CPU. Pre-checks also act as a gate: do not change a device that is already unhealthy. pyATS can learn features before and after and diff them; NAPALM getters and compliance_report work too.

GitLab CI example

stages: [lint, test, plan, deploy]

lint:
  stage: lint
  image: python:3.12
  script:
    - pip install yamllint ansible-lint
    - yamllint .
    - ansible-lint playbooks/

validate-data:
  stage: test
  image: python:3.12
  script:
    - pip install -r requirements.txt
    - python tests/check_data.py

plan:
  stage: plan
  script:
    - ansible-playbook playbooks/vlans.yml --check --diff
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

deploy:
  stage: deploy
  script:
    - ansible-playbook playbooks/precheck.yml
    - ansible-playbook playbooks/vlans.yml
    - ansible-playbook playbooks/postcheck.yml
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual
  environment: production

The plan and deploy jobs run on a runner inside the management network with access to devices and Vault.

Before

• Peer-reviewed merge request with a dry run diff for a sample of device types.
• Tested on a lab device with the same platform and release.
• Pre-checks: device reachable, healthy, config matches the expected baseline. Skip and report devices that fail.
• Rollback written and tested, not just "re-run the old playbook".

During

- name: Push new AAA config
  hosts: access_switches
  gather_facts: false
  serial: [1, 5, "10%", "25%"]
  max_fail_percentage: 0
  tasks:
    - name: Apply template
      cisco.ios.ios_config:
        src: aaa.j2
        save_when: modified

serial grows the batches, and max_fail_percentage: 0 stops the rollout as soon as any device in a batch fails. forks limits how many devices are touched at once, which also protects TACACS servers.

Safety nets per platform

• Junos and NETCONF candidate: commit confirmed with a few minutes timeout.
• IOS-XE with archive configured: configure terminal revert timer then configure confirm.
• Changes that touch management access (AAA, ACLs on vty, management VRF) go last and slowest.

After

Post-checks per batch, compare with pre-checks, then a final report: changed, unchanged, failed, skipped.

Where it comes from

• CLI fixes during incidents that never made it back to the data.
• Partial pushes that failed halfway.
• Software upgrades that change defaults or how config is displayed.
• Other tools writing to the same device.

Why merge-only automation cannot see it

If you only merge lines, extra config added by hand never shows up in your diff. Only a full intended config compared with the full running config (replace semantics) reveals additions as well as changes.

Practical setup

• Nightly job renders intended configs and diffs them with the running configs.
• Ignore volatile lines: ! Last configuration change at ..., certificate blobs, ntp clock-period.
• Report per device and per section, and track the drift count over time.
• Backups (Oxidized or RANCID style) in Git give a timeline of when each line appeared.
• Ansible: ios_config with diff_against: intended and intended_config, run with --check --diff.

Policy

Decide per section: some are enforced (AAA, SNMP, NTP, ACLs), where intent always wins and drift is corrected automatically; others are report-only while the team builds trust.

Phases

1. Discover: inventory, backups, NAPALM getters or gathered from resource modules, LLDP for topology.
2. Populate the source of truth: import devices, interfaces, IPs and VLANs into NetBox. Clean the data as you go; this is where you find the duplicates and mysteries.
3. Standardize: agree a standard per section, and accept that some devices must change to meet it.
4. Template per section: render, diff, adjust the template or the data until the diff is understood.
5. Enforce: push, then let drift checks protect the section.
6. Expand: interfaces, VLANs, routing, ACLs, and finally full-config replace where the platform supports it.

Why sections and not whole configs

A whole-config replace on day one means every inconsistency in the estate becomes a change at once. Per section, each step is small, reviewable and reversible, and the team learns the tools on low-risk config.

Thread pool with Netmiko

import os
from concurrent.futures import ThreadPoolExecutor, as_completed

from netmiko import ConnectHandler
from netmiko.exceptions import NetmikoAuthenticationException, NetmikoTimeoutException

USER = os.environ["NET_USER"]
PASSWORD = os.environ["NET_PASS"]

def get_uptime(host):
    device = {"device_type": "cisco_ios", "host": host,
              "username": USER, "password": PASSWORD}
    with ConnectHandler(**device) as conn:
        return conn.send_command("show version | include uptime")

with open("hosts.txt") as f:
    hosts = [line.strip() for line in f if line.strip()]

results, failures = {}, {}
with ThreadPoolExecutor(max_workers=30) as pool:
    futures = {pool.submit(get_uptime, h): h for h in hosts}
    for fut in as_completed(futures):
        host = futures[fut]
        try:
            results[host] = fut.result()
        except (NetmikoTimeoutException, NetmikoAuthenticationException) as exc:
            failures[host] = type(exc).__name__

print(f"{len(results)} ok, {len(failures)} failed")

The same with Nornir

from nornir import InitNornir
from nornir_netmiko.tasks import netmiko_send_command
from nornir_utils.plugins.functions import print_result

nr = InitNornir(config_file="config.yaml")
result = nr.filter(site="dc1").run(task=netmiko_send_command,
                                   command_string="show version")
print_result(result)
print(sorted(result.failed_hosts))

Limits that bite

• AAA: 30 parallel logins per second can overload TACACS+ servers. Ask before you raise concurrency.
• vty lines: IOS often has only line vty 0 4 (five sessions). Your job plus engineers can run out.
• Jump hosts and firewalls: connection rate limits and SSH session caps.
• Errors: catch per device, retry transient failures once or twice with backoff, never retry auth failures in a loop (account lockout).