Octet

Your Python script takes six hours to log in to 2,000 devices. How do you make it fast and safe?

Show the answer

Say this first

The job is I/O bound: most of the time is waiting on SSH and prompts, so run devices in parallel with a thread pool, asyncio or Nornir, which runs tasks across an inventory with 20 worker threads by default. Then add the safety parts: a sensible concurrency limit that the TACACS servers and vty lines can take, timeouts and limited retries per device, per-device results and logs, and a summary of what failed instead of stopping on the first error.

Go deeper

Thread pool with Netmiko

Python
import os
from concurrent.futures import ThreadPoolExecutor, as_completed

from netmiko import ConnectHandler
from netmiko.exceptions import NetmikoAuthenticationException, NetmikoTimeoutException

USER = os.environ["NET_USER"]
PASSWORD = os.environ["NET_PASS"]

def get_uptime(host):
    device = {"device_type": "cisco_ios", "host": host,
              "username": USER, "password": PASSWORD}
    with ConnectHandler(**device) as conn:
        return conn.send_command("show version | include uptime")

with open("hosts.txt") as f:
    hosts = [line.strip() for line in f if line.strip()]

results, failures = {}, {}
with ThreadPoolExecutor(max_workers=30) as pool:
    futures = {pool.submit(get_uptime, h): h for h in hosts}
    for fut in as_completed(futures):
        host = futures[fut]
        try:
            results[host] = fut.result()
        except (NetmikoTimeoutException, NetmikoAuthenticationException) as exc:
            failures[host] = type(exc).__name__

print(f"{len(results)} ok, {len(failures)} failed")

The same with Nornir

Python
from nornir import InitNornir
from nornir_netmiko.tasks import netmiko_send_command
from nornir_utils.plugins.functions import print_result

nr = InitNornir(config_file="config.yaml")
result = nr.filter(site="dc1").run(task=netmiko_send_command,
                                   command_string="show version")
print_result(result)
print(sorted(result.failed_hosts))

Limits that bite

  • AAA: 30 parallel logins per second can overload TACACS+ servers. Ask before you raise concurrency.
  • vty lines: IOS often has only line vty 0 4 (five sessions). Your job plus engineers can run out.
  • Jump hosts and firewalls: connection rate limits and SSH session caps.
  • Errors: catch per device, retry transient failures once or twice with backoff, never retry auth failures in a loop (account lockout).

Traps

  • Using multiprocessing for an I/O-bound job. It works but costs far more memory than threads for no gain.
  • Retrying failed logins in a tight loop and locking the service account in AAA.
How did you do?