Your Python script takes six hours to log in to 2,000 devices. How do you make it fast and safe?
Show the answer
Say this first
The job is I/O bound: most of the time is waiting on SSH and prompts, so run devices in parallel with a thread pool, asyncio or Nornir, which runs tasks across an inventory with 20 worker threads by default. Then add the safety parts: a sensible concurrency limit that the TACACS servers and vty lines can take, timeouts and limited retries per device, per-device results and logs, and a summary of what failed instead of stopping on the first error.
Go deeper
Thread pool with Netmiko
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from netmiko import ConnectHandler
from netmiko.exceptions import NetmikoAuthenticationException, NetmikoTimeoutException
USER = os.environ["NET_USER"]
PASSWORD = os.environ["NET_PASS"]
def get_uptime(host):
device = {"device_type": "cisco_ios", "host": host,
"username": USER, "password": PASSWORD}
with ConnectHandler(**device) as conn:
return conn.send_command("show version | include uptime")
with open("hosts.txt") as f:
hosts = [line.strip() for line in f if line.strip()]
results, failures = {}, {}
with ThreadPoolExecutor(max_workers=30) as pool:
futures = {pool.submit(get_uptime, h): h for h in hosts}
for fut in as_completed(futures):
host = futures[fut]
try:
results[host] = fut.result()
except (NetmikoTimeoutException, NetmikoAuthenticationException) as exc:
failures[host] = type(exc).__name__
print(f"{len(results)} ok, {len(failures)} failed")The same with Nornir
from nornir import InitNornir
from nornir_netmiko.tasks import netmiko_send_command
from nornir_utils.plugins.functions import print_result
nr = InitNornir(config_file="config.yaml")
result = nr.filter(site="dc1").run(task=netmiko_send_command,
command_string="show version")
print_result(result)
print(sorted(result.failed_hosts))Limits that bite
- AAA: 30 parallel logins per second can overload TACACS+ servers. Ask before you raise concurrency.
- vty lines: IOS often has only
line vty 0 4(five sessions). Your job plus engineers can run out. - Jump hosts and firewalls: connection rate limits and SSH session caps.
- Errors: catch per device, retry transient failures once or twice with backoff, never retry auth failures in a loop (account lockout).
They will ask next
Threads or asyncio?
Both work for I/O-bound SSH. Threads are simpler and work with Netmiko and NAPALM as they are. asyncio (for example with Scrapli's async drivers) scales to more connections with less memory, but every library in the chain must be async.
Traps
- Using multiprocessing for an I/O-bound job. It works but costs far more memory than threads for no gain.
- Retrying failed logins in a tight loop and locking the service account in AAA.
How did you do?