IP services

The four messages

The REQUEST is a broadcast on purpose. Every server that made an offer sees it, and the ones that were not chosen put their offered address back in the pool.

Unicast or broadcast replies

The client has no address yet, so how does the OFFER reach it? The client decides with the broadcast flag, the top bit of the 16-bit flags field:

• Flag set: the server (or relay) broadcasts the OFFER and ACK.
• Flag clear: the server sends them as unicast IP to yiaddr, addressed at layer 2 to the client MAC in chaddr.

Stacks that cannot accept unicast IP before the interface has an address set the flag.

Under the hood

DHCP messages are BOOTP packets with option 53 saying which message they are. That is why the ports are named bootps (67) and bootpc (68). Option 53 values: 1 DISCOVER, 2 OFFER, 3 REQUEST, 4 DECLINE, 5 ACK, 6 NAK, 7 RELEASE, 8 INFORM.

After the ACK, many clients probe the address with ARP (RFC 5227) before using it. If another host answers, the client sends DECLINE and starts again.

The options worth knowing by number

Option 43 for Cisco APs

A Cisco lightweight AP expects a type-length-value: type f1, length 4 bytes per controller, then each controller address in hex. Two controllers at 10.0.50.10 and 10.0.50.11: f1 + 08 + 0a00320a + 0a00320b.

ip dhcp excluded-address 10.50.0.1 10.50.0.10
ip dhcp pool AP-MGMT
 network 10.50.0.0 255.255.255.0
 default-router 10.50.0.1
 dns-server 10.0.0.53
 option 43 hex f1080a00320a0a00320b
!
ip dhcp excluded-address 10.40.0.1 10.40.0.10
ip dhcp pool VOICE
 network 10.40.0.0 255.255.255.0
 default-router 10.40.0.1
 dns-server 10.0.0.53
 domain-name corp.example.com
 option 150 ip 10.0.0.15 10.0.0.16
 lease 0 8

lease 0 8 is 0 days and 8 hours. The IOS default is 1 day. Other vendors' APs and phones use 43 with their own sub-option formats, usually matched by option 60.

In the order the packet travels

1. Access port: up, in the right VLAN, not err-disabled. show interfaces status, show vlan brief.
2. Spanning tree: a port without PortFast spends 30 seconds in listening and learning. The client can give up before the port forwards. Access ports get spanning-tree portfast.
3. DHCP snooping: if the uplink is not trusted, the switch drops the OFFER. Check the trusted ports in show ip dhcp snooping.
4. Relay: show ip interface Vlan20 | include Helper must show the server.
5. Path to the server: ping 10.0.12.1 source Vlan20 from the gateway tests the route there and the server's route back to the client subnet in one go.
6. Scope: exists for that subnet and is not full. show ip dhcp pool on IOS, scope statistics on Windows.
7. Capture: debug ip dhcp server packet on an IOS server, or udp port 67 or udp port 68 in tcpdump. In Wireshark the display filter is dhcp.

On Linux there is usually no 169.254 address at all, just no IPv4 address, unless the network manager is set to fall back to link-local.

The walk

PC1 stub  -- recursive query, RD=1 -->  resolver 10.0.0.53
resolver  -- www.example.com A? ---->  a root server        referral: com. NS plus glue
resolver  -- www.example.com A? ---->  a .com TLD server    referral: example.com. NS plus glue
resolver  -- www.example.com A? ---->  ns1.example.com      answer, AA bit set
resolver  -- answer, RA=1 ---------->  PC1

With QNAME minimisation (RFC 9156) the resolver asks the root only about com. and the TLD only about example.com., so the full name is not leaked to servers that do not need it.

Details interviewers probe

• Root addresses come from the root hints built into the resolver: 13 names, a to m.root-servers.net, each served by many anycast instances worldwide.
• Glue: ns1.example.com lives inside the zone it serves. The .com servers include its A and AAAA records in the referral, otherwise the resolver could never reach it.
• Caching: the next query for mail.example.com skips root and TLD, because the example.com NS set is already cached.
• Transport: UDP 53, with TCP 53 for answers too big for UDP.
• Client side: the hosts file is checked first on most systems, the OS has its own cache (the Windows DNS Client service, systemd-resolved on 127.0.0.53), browsers keep another, and search domains get appended to short names.

In zone file form

example.com.      3600 IN SOA   ns1.example.com. hostmaster.example.com. 2026111001 7200 3600 1209600 3600
example.com.      3600 IN NS    ns1.example.com.
example.com.      3600 IN NS    ns2.example.net.
example.com.      3600 IN MX    10 mail1.example.com.
example.com.      3600 IN MX    20 mail2.example.com.
example.com.      3600 IN TXT   "v=spf1 mx -all"
www.example.com.   300 IN A     203.0.113.80
www.example.com.   300 IN AAAA  2001:db8:80::80
shop.example.com.  300 IN CNAME www.example.com.
_sip._tcp.example.com. 3600 IN SRV 10 60 5060 sip1.example.com.

And in the reverse zone 113.0.203.in-addr.arpa:

80.113.0.203.in-addr.arpa. 3600 IN PTR www.example.com.

SRV fields

priority weight port target. Lower priority is tried first. Among equal priorities, weight sets the share of traffic. Active Directory clients find domain controllers this way, through names like _ldap._tcp.dc._msdcs.corp.example.com.

Others worth naming

• CAA: which certificate authorities may issue certificates for the domain. CAs must check it.
• DS, DNSKEY, RRSIG, NSEC, NSEC3: DNSSEC.
• HTTPS and SVCB (RFC 9460): connection hints such as ALPN, so a browser can learn that a site speaks HTTP/3 before it connects.
• NAPTR: used with SRV in SIP and ENUM.

Why backwards

DNS names get more specific from right to left. IP addresses get more specific from left to right. Reversing the octets lets reverse zones be delegated along address boundaries: the /24 owner gets 2.0.192.in-addr.arpa.

IPv6

2001:db8::25 is expanded to 32 nibbles and reversed:

5.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa

dig -x 2001:db8::25 builds that name for you.

Blocks smaller than /24

Reverse delegation follows octet boundaries. For a /28, RFC 2317 classless delegation uses CNAMEs in the ISP's zone that point into a zone you host. Otherwise you ask the ISP to create the PTRs.

Where it matters

• Mail: receivers check that a sending IP has a PTR and that the name resolves back to the same IP (forward-confirmed reverse DNS). Missing or mismatched records hurt delivery.
• Logs and traceroute output become readable.
• Some servers do reverse lookups on every login (OpenSSH UseDNS), which makes logins slow when reverse DNS times out.

The example

R1 has Gi0/0/1 inside (192.168.10.1/24) and Gi0/0/0 outside (203.0.113.2/29), with PAT on the outside interface.

The packet walk

1. PC1 sends 192.168.10.10:51004 to 198.51.100.80:443: inside local to outside local.
2. R1 routes it to Gi0/0/0 and rewrites the source: 203.0.113.2:51004 to 198.51.100.80:443, which is inside global to outside global.
3. The reply comes back to 203.0.113.2:51004. R1 rewrites the destination to 192.168.10.10:51004 and routes it inside.

R1# show ip nat translations
Pro  Inside global         Inside local          Outside local         Outside global
tcp  203.0.113.2:51004     192.168.10.10:51004   198.51.100.80:443     198.51.100.80:443
Total number of translations: 1

The column order is worth memorising: inside global, inside local, outside local, outside global.

When outside local differs

Only with outside NAT, usually for overlapping networks. After a merger the partner's server 10.1.1.50 clashes with our own 10.1.1.0/24, so we make it appear to our users as 172.31.1.50:

ip nat outside source static 10.1.1.50 172.31.1.50 add-route

Outside global is 10.1.1.50 (its real address), outside local is 172.31.1.50 (what our users use). add-route installs a route to 172.31.1.50 so the router can forward toward it.

The base: mark the interfaces

interface GigabitEthernet0/0/0
 ip address 203.0.113.2 255.255.255.248
 ip nat outside
interface GigabitEthernet0/0/1
 ip address 192.168.10.1 255.255.255.0
 ip nat inside
interface GigabitEthernet0/0/2
 ip address 192.168.20.1 255.255.255.0
 ip nat inside
!
ip access-list standard NAT-USERS
 permit 192.168.10.0 0.0.0.255

Static NAT, one to one

ip nat inside source static 192.168.20.10 203.0.113.5

Dynamic NAT from a pool

ip nat pool PUBLIC 203.0.113.3 203.0.113.4 netmask 255.255.255.248
ip nat inside source list NAT-USERS pool PUBLIC

PAT on the interface address

ip nat inside source list NAT-USERS interface GigabitEthernet0/0/0 overload

Use the PAT line instead of the dynamic pool line, not with it: one rule per access list.

Policy NAT (ip nat inside source route-map ...) chooses the translation based on destination too, for example a different address toward a partner network.

Configuration

interface GigabitEthernet0/0/0
 ip address 203.0.113.2 255.255.255.248
 ip nat outside
interface GigabitEthernet0/0/2
 ip address 192.168.20.1 255.255.255.0
 ip nat inside
!
ip nat inside source static tcp 192.168.20.20 443 interface GigabitEthernet0/0/0 443
ip nat inside source static tcp 192.168.20.30 22 interface GigabitEthernet0/0/0 2222

The second line also changes the port: outside users connect to 203.0.113.2 port 2222 and land on SSH port 22 of 192.168.20.30.

R1# show ip nat translations
Pro  Inside global         Inside local          Outside local         Outside global
tcp  203.0.113.2:443       192.168.20.20:443     ---                   ---
tcp  203.0.113.2:2222      192.168.20.30:22      ---                   ---
tcp  203.0.113.2:443       192.168.20.20:443     198.51.100.77:50112   198.51.100.77:50112
Total number of translations: 3

The lines with --- are the static entries. The third line is a live session from an internet client at 198.51.100.77.

Checklist

• The inbound ACL or firewall permits TCP 443 to 203.0.113.2, the global address.
• The server's default gateway is R1. If replies leave through another router, they are never untranslated and the client drops them.
• The ISP does not filter the port.
• Test from outside. Testing from the LAN hits the hairpin problem.

On Linux

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to-destination 192.168.20.20:443
iptables -A FORWARD -i eth0 -d 192.168.20.20 -p tcp --dport 443 -j ACCEPT

What NAT is for

Address conservation: RFC 1918 space inside and PAT at the edge. Hiding the internal addressing is a minor bonus, not a control.

What it costs

• It breaks end-to-end addressing, which is why ALGs and NAT traversal exist.
• It makes attribution harder. To know which inside host made a connection at a given time you need translation logs. Carriers running CGNAT keep them for legal requests.
• It complicates troubleshooting: the same flow has different addresses on each side of the box.

The interview answer

Security comes from a stateful firewall policy: default deny inbound, explicit permits for published services, inspection where needed. NAT runs alongside it.

How NTP works

NTP runs on UDP 123 (NTPv4 is RFC 5905). The client stamps four times per exchange:

Round-trip delay is (T4 - T1) - (T3 - T2) and the clock offset is ((T2 - T1) + (T3 - T4)) / 2. The formula assumes the path is symmetric, so asymmetric paths show up as offset error. Small errors are slewed (the clock is sped up or slowed down), large ones are stepped. IOS polls between 64 and 1024 seconds by default.

A sensible design

• Two to four internal time servers, each fed by several external or GPS sources.
• Every network device syncs to the internal servers, sourced from a loopback.
• Devices log in UTC with millisecond timestamps.

ntp server 10.0.0.123 prefer
ntp server 10.0.0.124
ntp source Loopback0
ntp update-calendar
service timestamps log datetime msec show-timezone

ntp update-calendar keeps the hardware clock in step, so the router boots with sane time. ntp master makes a router serve its own clock (stratum 8 by default): fine in a lab or an isolated network, wrong anywhere real sources exist.

Operations

Get, GetNext, GetBulk (v2c and later), Set, Response, Trap, Inform, and Report (v3 error and discovery messages).

Read-write is dangerous

A write community or user can change the device. On Cisco, a Set against the config copy MIB can make the router send its whole configuration to a TFTP server. Avoid RW; if you need it, use v3 and a tight view.

A locked-down v2c baseline

ip access-list standard SNMP-MGRS
 permit 10.0.0.50
snmp-server community Oct3t-R0-2026 RO SNMP-MGRS

Where it is going

Streaming telemetry (model-driven, gNMI) pushes counters every few seconds instead of waiting to be polled. SNMP stays everywhere for inventory, basic counters and alarms.

A memory aid: Every Awesome Cisco Engineer Will Need Ice cream Daily.

Where logs go on IOS

service timestamps log datetime msec show-timezone
service sequence-numbers
logging buffered 64000 informational
logging console warnings
logging source-interface Loopback0
logging host 10.0.0.60
logging trap informational

Syslog uses UDP 514 by default. TCP and TLS (port 6514) are available where the platform supports them. Cisco devices use facility local7 by default.

TFTP limits

The server answers from a new ephemeral port, so stateful firewalls need a TFTP helper. Block numbers are 16 bits, so with 512-byte blocks a file over about 32 MB fails unless both ends support the blocksize option (RFC 2348) or block number rollover.

SCP on IOS

ip ssh version 2
aaa new-model
aaa authentication login default local
aaa authorization exec default local
ip scp server enable

Pull an image from a server:

R1# copy scp://admin@10.0.0.5/isr4400-universalk9.17.09.04a.SPA.bin bootflash:
R1# verify /md5 bootflash:isr4400-universalk9.17.09.04a.SPA.bin

Compare the hash with the one the vendor publishes before you boot it.

OpenSSH 9.0 and later run the SFTP protocol under the scp command by default. To push a file to a device that only has an SCP server, use scp -O for the legacy protocol.

LLDP frames use EtherType 0x88CC. The mandatory TLVs are Chassis ID, Port ID, TTL and End of LLDPDU.

Security

Both give away platform, software version and addresses, which is useful reconnaissance for an attacker. Keep them on infrastructure links and turn them off on internet-facing and untrusted ports:

lldp run
interface GigabitEthernet0/0/0
 no cdp enable
 no lldp transmit
 no lldp receive

A subtle one

A switch that does not understand CDP floods it like any other multicast. Two Cisco devices behind an unmanaged switch then list each other as direct neighbours. Standard bridges never forward LLDP sent to the nearest-bridge address.

Verify with show cdp neighbors detail and show lldp neighbors detail; on Linux, lldpctl from lldpd.

Step by step

1. PC1 in VLAN 20 broadcasts a DISCOVER. The gateway SVI is 10.20.0.1/24 with ip helper-address 10.0.12.1.
2. The relay sets giaddr to 10.20.0.1, the primary address of the interface the request came in on. It adds 1 to the hops field and unicasts the packet to 10.0.12.1 on UDP 67.
3. The server finds the pool whose network contains giaddr. No matching pool means no offer.
4. The server unicasts the OFFER to giaddr on UDP 67. A relay listens on 67.
5. The relay forwards the OFFER to the client on UDP 68, as broadcast or unicast per the broadcast flag.
6. REQUEST and ACK take the same path.

Configuration

interface Vlan20
 ip address 10.20.0.1 255.255.255.0
 ip helper-address 10.0.12.1
 ip helper-address 10.0.12.2

With two helpers the relay sends a copy to each server. That is how you feed a failover pair.

feature dhcp
interface Vlan20
  ip dhcp relay address 10.0.12.1

What breaks it

• The server has no route back to the client subnet. It must reach giaddr.
• The helper is on the wrong interface. It belongs on the client-facing interface.
• No scope on the server for the giaddr subnet, or the scope is full.
• A firewall between relay and server blocks UDP 67.
• Renewals at T1 are unicast from the client straight to the server address in option 54, not through the relay. The client subnet needs a routed, permitted path to the server as well.

Why it matters

Every Windows host broadcasting NetBIOS on that VLAN gets relayed to the DHCP servers. It is noise at best. On a clean build I remove what I do not need.

no ip forward-protocol udp 37
no ip forward-protocol udp 49
no ip forward-protocol udp 137
no ip forward-protocol udp 138

Adding a port

Wake-on-LAN magic packets are often sent to UDP 9. To relay them, add the port and point a helper at the directed broadcast of the target subnet:

ip forward-protocol udp 9
!
interface Vlan20
 ip helper-address 10.30.0.255
!
interface Vlan30
 ip address 10.30.0.1 255.255.255.0
 ip directed-broadcast 110

ip directed-broadcast 110 lets the router turn the packet into a layer 2 broadcast on VLAN 30, only for traffic that access list 110 permits (for example the WoL server). Directed broadcasts are off by default since IOS 12.0.

Client states

NAK and silence

After a reboot the client asks for its old address. A server that knows the subnet is wrong answers with a NAK, and the client restarts at DISCOVER. A server with no record of the client should stay silent. ISC dhcpd only sends NAKs for a network when it is marked authoritative.

Release

ipconfig /release or dhclient -r sends DHCPRELEASE, which returns the address early.

Choosing lease times

• Guest Wi-Fi and busy shared spaces: short leases (1 to 8 hours) so the pool recycles.
• Wired desktops: long leases. The Windows Server default is 8 days, the IOS default is 1 day.
• A long lease also lets existing clients ride out a DHCP server outage longer.

On a Cisco IOS server

ip dhcp excluded-address 10.20.0.1 10.20.0.10
ip dhcp ping packets 2
ip dhcp ping timeout 500
ip dhcp database bootflash:dhcp-bindings

The ping values shown are the defaults. show ip dhcp conflict lists conflicted addresses and how they were found (ping or gratuitous ARP). They stay out of the pool until you run clear ip dhcp conflict *.

Without ip dhcp database, IOS forgets its bindings on reload and may offer an address a client still holds. Cisco's advice is to either configure the database agent or turn off conflict logging with no ip dhcp conflict logging.

On the client

RFC 5227 ARP probes use sender IP 0.0.0.0, so they do not poison anyone's ARP cache. If a reply comes back, the client sends DECLINE and the server marks the address as bad.

Usual causes of conflicts

• Printers or servers configured statically inside the pool range.
• A rogue DHCP server handing out the same subnet.
• The server lost its bindings (reload without a database).

The flags

A typical resolver reply shows flags: qr rd ra. A query sent with dig +norecurse straight to an authoritative server returns flags: qr aa.

Keep the roles apart

• Recursive resolvers serve your own clients and cache.
• Authoritative servers publish your zones to the world and should refuse recursion. In BIND that is recursion no; on the authoritative instance, or allow-recursion limited to internal clients.

Why it matters

An open resolver (recursion for anyone) gets used for DNS amplification: small spoofed queries, large answers aimed at the victim. It also gives attackers a cache to try to poison. Resolver defences against poisoning include source port randomisation and DNSSEC validation.

The rule

RFC 1034: if a CNAME exists at a name, no other data may exist there. Only DNSSEC records are allowed alongside it. That rules out a CNAME at the apex and next to MX, TXT or any other record.

Why people want it

Cloud load balancers and CDNs give you a hostname, not a fixed address, because their addresses change. www can be a CNAME to it. The bare example.com cannot.

The workarounds

• Flattening, ALIAS, ANAME: the provider looks up the target and answers with A and AAAA records. Fine for most cases. The provider resolves the target from its own location, so geo-based answers reflect the provider, not the user, unless the platform handles that.
• Route 53 alias records: point the apex at AWS resources directly.
• HTTPS record in alias mode (RFC 9460): designed partly for this, but it only helps clients that query HTTPS records.
• Redirect: serve the apex from a fixed address that redirects to www.

Positive caching

Each resolver counts down on its own. Users behind different resolvers can see different answers for up to one full TTL after a change, and you cannot flush other people's resolvers.

Negative caching

A negative answer carries the zone's SOA in the authority section. With SOA ... 2026111001 7200 3600 1209600 3600 and an SOA TTL of 3600, a "does not exist" answer is cached for an hour. The classic trap: someone tests a name before it is created, the resolver caches NXDOMAIN, and the new record stays invisible to them for up to an hour.

Resolvers also cap TTLs. BIND by default keeps positive answers at most 1 week and negative answers at most 3 hours.

Migration recipe

Keep the old server answering, or proxying to the new one, until the old TTL has passed.

Flushing caches you do control

An SOA, field by field

example.com. 3600 IN SOA ns1.example.com. hostmaster.example.com. (
        2026111001 ; serial, YYYYMMDDnn by convention
        7200       ; refresh, 2 hours
        3600       ; retry, 1 hour
        1209600    ; expire, 14 days
        3600 )     ; minimum, the negative caching TTL

hostmaster.example.com. means the mailbox hostmaster@example.com: the first dot stands for the @.

Rules

• The serial must go up on every change. Lower it by mistake and secondaries ignore the change. Serial arithmetic (RFC 1982) lets you move forward past a mistake in steps.
• NOTIFY (RFC 1996) makes updates near instant. Refresh is the safety net.
• Restrict transfers to your secondaries by address and sign them with TSIG keys. An open AXFR hands anyone your whole zone, which is free reconnaissance.

Hidden primary

A common design: the primary is not listed in the NS records and only feeds the public secondaries. It can live behind a firewall and nobody queries it directly.

The toolkit

dig www.example.com                                # what my configured resolver says
dig @10.0.0.53 www.example.com                     # a specific resolver
dig @ns1.example.com www.example.com +norecurse    # the source of truth, look for aa
dig +trace www.example.com                         # walk from the root myself
dig @ns1.example.com example.com SOA +short        # compare serials on every NS
dig +cd www.example.com                            # skip DNSSEC checks: works now means DNSSEC is broken
resolvectl status                                  # which resolver Linux is really using

Reading the status

Classic causes

Wrong resolver handed out by DHCP, VPN split DNS rules, a stale hosts file entry, a forgotten search domain, a secondary out of sync, expired DNSSEC signatures, TCP 53 blocked.

A busy translation table

R1# show ip nat translations
Pro  Inside global         Inside local          Outside local         Outside global
tcp  203.0.113.2:51004     192.168.10.10:51004   198.51.100.80:443     198.51.100.80:443
tcp  203.0.113.2:49731     192.168.10.11:49731   198.51.100.80:443     198.51.100.80:443
udp  203.0.113.2:61812     192.168.10.12:61812   198.51.100.53:53      198.51.100.53:53
icmp 203.0.113.2:1         192.168.10.10:1       198.51.100.80:1       198.51.100.80:1
Total number of translations: 4

Three hosts, one global address. ICMP echo has no ports, so PAT uses the ICMP query identifier in their place.

Port choice on classic IOS

IOS keeps the original source port when it is free on the global address. If it is taken, it picks the first free port in the same group: below 512, 512 to 1023, or 1024 and up. With a pool and overload, it moves to the next global address when one runs out of ports.

Timers (IOS defaults)

Tuned with ip nat translation tcp-timeout, udp-timeout and friends. A long TCP timer means abandoned connections hold ports. A short UDP timer breaks quiet VoIP and VPN flows.

Symptom to cause

Read the statistics

R1# show ip nat statistics
Total active translations: 3 (1 static, 2 dynamic; 2 extended)
Outside interfaces:
  GigabitEthernet0/0/0
Inside interfaces:
  GigabitEthernet0/0/1, GigabitEthernet0/0/2
Hits: 1520  Misses: 4
Expired translations: 2
Dynamic mappings:
-- Inside Source
[Id: 1] access-list NAT-USERS interface GigabitEthernet0/0/0 refcount 2

The interface lists catch swapped roles. The dynamic mapping shows which ACL and target are in use.

Watch it happen

Filter the debug with a numbered ACL that matches one test host:

R1(config)# access-list 99 permit 192.168.10.10
R1# debug ip nat 99

Lines like NAT*: s=192.168.10.10->203.0.113.2, d=198.51.100.80 prove the translation. Outbound lines with no return lines put the problem upstream.

Classic ACL mistakes

• A subnet mask where a wildcard belongs: permit 192.168.10.0 255.255.255.0 matches the wrong addresses.
• permit any: the router starts translating traffic it should not, such as VPN and its own management traffic.

ALGs on IOS

IOS NAT inspects and rewrites FTP, SIP and other protocols by default. Voice providers often ask you to turn the SIP ALG off because it mangles messages their SBC already handles:

no ip nat service sip udp port 5060
no ip nat service sip tcp port 5060

Timers

Quiet UDP flows lose their mapping when the NAT UDP timer expires (300 seconds on IOS, often less on consumer routers). Keepalives from the endpoints keep mappings alive.

Client configuration

ntp authentication-key 1 md5 N7p-Octet-Key
ntp authenticate
ntp trusted-key 1
ntp server 10.0.0.123 key 1
ntp server 10.0.0.124 key 1
ntp source Loopback0

The running config shows the key string encrypted with type 7. show ntp associations detail shows the word authenticated for each server that passes.

ntp authenticate means "only sync to authenticated sources". Any server configured without a working key keeps answering but is never selected.

Who may talk to you

access-list 10 permit 10.0.0.123
access-list 10 permit 10.0.0.124
access-list 20 permit 10.0.0.0 0.255.255.255
ntp access-group peer 10
ntp access-group serve-only 20

peer sources may sync this device, serve-only clients may only ask for time, query-only allows control queries only. Once any access group is set, everything not permitted is refused.

Junos

set system ntp authentication-key 1 type md5 value "N7p-Octet-Key"
set system ntp trusted-key 1
set system ntp server 10.0.0.123 key 1

Beyond MD5 keys

Symmetric MD5 keys are old but interoperate everywhere. Network Time Security (NTS, RFC 8915) uses TLS to agree keys and then authenticates every NTP packet. chrony and NTPsec support it; network devices mostly do not yet.

Configuration

ip access-list standard SNMP-MGRS
 permit 10.0.0.50
!
snmp-server view NMS-VIEW iso included
snmp-server group NMS-RO v3 priv read NMS-VIEW access SNMP-MGRS
snmp-server user nmsuser NMS-RO v3 auth sha Auth-Pass-2026 priv aes 128 Priv-Pass-2026

The security model is USM (user-based, RFC 3414) and access control is VACM (view-based, RFC 3415).

Things that catch people

• snmp-server user lines never appear in the running config. Check with show snmp user.
• User keys are localized to the engine ID. Change snmp-server engineID local and every user stops working until you re-create it.
• The group's level is a minimum. A request at authNoPriv against a priv group is refused.
• net-snmp rejects passphrases shorter than 8 characters.
• Informs over v3 need the manager's engine ID (snmp-server engineID remote) and the user defined against it with the remote keyword.

Test from the NMS

snmpget -v3 -l authPriv -u nmsuser -a SHA -A 'Auth-Pass-2026' -x AES -X 'Priv-Pass-2026' 10.255.0.1 SNMPv2-MIB::sysName.0

MD5 and DES are weak. SHA with AES-128 is the common baseline, and newer platforms add SHA-2 options.

The OID tree

1.3.6.1                iso.org.dod.internet
1.3.6.1.2.1            mib-2, the standard MIBs
1.3.6.1.2.1.1          system: sysDescr .1, sysUpTime .3, sysName .5
1.3.6.1.2.1.2.2.1      ifEntry: ifDescr .2, ifOperStatus .8, ifInOctets .10
1.3.6.1.2.1.31.1.1.1   ifXEntry: ifHCInOctets .6 (64-bit)
1.3.6.1.4.1            enterprises: Cisco is 1.3.6.1.4.1.9

Scalars end in .0, as in sysName.0. Table cells end in the row index, usually the ifIndex: ifInOctets.3 is interface index 3. snmp-server ifindex persist keeps ifIndex values stable across reloads.

Trap or inform

snmp-server enable traps snmp linkdown linkup coldstart warmstart
snmp-server host 10.0.0.50 version 3 priv nmsuser
snmp-server host 10.0.0.51 informs version 2c Oct3t-Trap-2026

What a flow is

A one-way stream of packets that share key fields. Traditional NetFlow uses seven keys: source and destination address, source and destination port, IP protocol, ToS byte and input interface. The device keeps flows in a cache and exports a record when a flow ends, when it goes idle (inactive timeout, 15 seconds by default) or when it has run for the active timeout (30 minutes by default on IOS). Set the active timeout to about 60 seconds, or long flows appear as spikes.

With v9 and IPFIX the collector cannot decode data until it has the template, so a collector that restarts shows a gap until the next template refresh.

Flexible NetFlow on IOS-XE

flow exporter EXP-COLLECTOR
 destination 10.0.0.70
 source Loopback0
 transport udp 4739
 export-protocol ipfix
!
flow monitor MON-V4
 exporter EXP-COLLECTOR
 record netflow ipv4 original-input
 cache timeout active 60
!
interface GigabitEthernet0/0/0
 ip flow monitor MON-V4 input

Uses: top talkers, capacity planning, DDoS detection, security investigations, billing.

Configuration

ip sla 1
 icmp-echo 198.51.100.200 source-interface GigabitEthernet0/0/0
 timeout 2000
 threshold 1000
 frequency 10
ip sla schedule 1 life forever start-time now
!
track 1 ip sla 1 reachability
 delay down 10 up 30
!
ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 1
ip route 0.0.0.0 0.0.0.0 192.0.2.9 250
ip route 198.51.100.200 255.255.255.255 203.0.113.1

• The probe target 198.51.100.200 sits beyond ISP1's first hop. ISP1's router at 203.0.113.1 can answer while its upstream is broken.
• timeout and threshold are in milliseconds, frequency in seconds. The defaults are 5000 ms, 5000 ms and 60 seconds.
• delay down 10 up 30 stops a flapping path from flapping the route.
• The host route keeps the probe on ISP1. Without it, after failover the probe would follow the backup default, succeed, and bring the primary route back: a loop of flaps.
• The backup default to ISP2 (192.0.2.9) has distance 250, so it only installs when the tracked route is gone.

Other probes

udp-jitter measures one-way jitter, loss and latency for voice and needs ip sla responder on the far Cisco device. With a codec option it also estimates a MOS score.

Verify

show ip sla statistics 1, show track 1, show ip route static. On Junos the equivalent is RPM (real-time performance monitoring) probes with ip-monitoring.

What option 82 carries

• Sub-option 1, circuit ID: usually the VLAN, module and port of the client.
• Sub-option 2, remote ID: usually the switch MAC address.

The server can use it for policy (fixed address per port, subscriber identity at service providers) and logs it. The relay strips it from the reply before the client sees it.

Why the router drops it

RFC 3046 tells a relay to discard a packet that arrives on an untrusted circuit with option 82 already present and giaddr zero, because a client could be forging it. A snooping switch creates exactly that packet. debug ip dhcp server packet on the router shows it complaining that the relay information option is present while giaddr is zero.

Fixes

On the router, trust the option on the interface facing the switch, or everywhere:

interface GigabitEthernet0/0/1.40
 ip dhcp relay information trusted
!
ip dhcp relay information trust-all

Or stop the switch inserting it:

no ip dhcp snooping information option

If the server really uses option 82 (service provider style), keep the insertion and trust it on the relay. Otherwise turn insertion off at the access layer.

Split scope

Both servers hold the same subnet with non-overlapping ranges. No sync, simple to run. Each part must cover demand on its own if the other server is down, and reservations must exist on both.

Failover with shared state

• Windows Server (2012 and later): two partners replicate leases. Load-balance mode shares the pool (50/50 by default). Hot-standby keeps a reserve on the standby (5 percent by default). The maximum client lead time (MCLT, 1 hour by default) bounds how far one partner can extend a lease beyond what the other knows.
• ISC Kea: the high availability hook supports load-balancing, hot-standby and passive-backup modes, with the partners sending lease updates to each other.
• ISC dhcpd: has a failover protocol between two peers, but ISC has ended maintenance of dhcpd and points users to Kea.

Branches

A branch that depends on central DHCP over the WAN loses new leases when the WAN fails. Existing clients keep their addresses until expiry. A long lease or a small local IOS DHCP server covers that. The IOS server has no failover of its own, so give it ip dhcp database so a reload keeps bindings.

The pieces

• EDNS(0) (RFC 6891): an OPT pseudo-record in the additional section carries the client's UDP buffer size and the DO bit that asks for DNSSEC records.
• 1232 bytes: the IPv6 minimum MTU of 1280, minus 40 bytes of IPv6 header and 8 of UDP. It was the value agreed for DNS Flag Day 2020.
• TCP: RFC 7766 makes TCP support mandatory for DNS software. It is not just for transfers.

What produces big answers

DNSSEC signatures, long TXT records (SPF, verification tokens), names with many A or AAAA records.

The symptom

Most names resolve, a few fail with SERVFAIL or time out. When an answer is too big, dig prints this line before it retries over TCP:

;; Truncated, retrying in TCP mode.

If dig +tcp to the same server then times out, TCP 53 is blocked. Older firewall DNS inspection profiles also capped messages at 512 bytes and dropped EDNS answers.

Encrypted DNS

DoH matters to network teams because it looks like ordinary web traffic and can bypass the internal resolvers that enforce policy.

The chain of trust

1. The resolver trusts the root key (the trust anchor).
2. The root zone signs a DS record for com., a hash of the .com key.
3. The .com zone signs a DS record for example.com..
4. example.com signs its records with keys that match that DS.

Zones usually have a key-signing key (KSK, the one the parent's DS points at) and a zone-signing key (ZSK, which signs the records). NSEC or NSEC3 records prove that a name does not exist.

What a validating resolver does

Valid answers get the AD (authenticated data) bit. Answers that fail validation become SERVFAIL. The client gets no answer at all rather than a forged one.

How DNSSEC outages happen

• Signatures expire because the signing job stopped.
• A key rollover without updating the DS at the registrar.
• Moving to a new DNS provider while the old DS stays at the parent.

dig +dnssec shows the RRSIGs, dig +cd turns checking off for a comparison, and delv explains validation step by step.

The sequence, trimmed to what matters

What follows from it

1. Inbound ACL on the outside interface sees pre-NAT addresses. To let the internet reach SRV1 (192.168.20.10, published as 203.0.113.5), permit host 203.0.113.5, not the private address.
2. Outbound ACL on the outside interface sees post-NAT addresses, so the sources are inside global.
3. Inbound ACL on an inside interface sees inside local addresses.
4. Outside to inside, routing uses the translated destination. R1 needs a route to 192.168.20.10.
5. Crypto runs after NAT. Traffic to a VPN peer's private network gets PATed first, then no longer matches the private-to-private crypto ACL. Exempt it in the NAT rule:

ip access-list extended NAT-USERS
 deny   ip 192.168.10.0 0.0.0.255 10.200.0.0 0.0.255.255
 permit ip 192.168.10.0 0.0.0.255 any
!
ip nat inside source list NAT-USERS interface GigabitEthernet0/0/0 overload

6. Translation happens only when a packet crosses between an ip nat inside and an ip nat outside interface. Inside to inside is never translated.

Other platforms do it differently

• Cisco ASA (8.3 and later) matches ACLs against the real, untranslated address: the opposite habit.
• Linux netfilter does DNAT in PREROUTING, before the routing decision, and SNAT in POSTROUTING, after it. Filter rules in FORWARD see the private destination of a port forward.

Why it fails

1. PC1 (192.168.10.10) resolves app.example.com to 203.0.113.5 and sends a SYN to it.
2. The SYN enters R1 on Gi0/0/1, an inside interface. Outside-to-inside translation only applies to packets arriving on an outside interface, so the destination stays 203.0.113.5 and the packet never reaches SRV1 at 192.168.20.10.

The same-subnet trap

Suppose a device does rewrite the destination (Linux DNAT applies on any interface). If the client sits on the server's own subnet, the server replies directly from 192.168.20.10. The client sent its SYN to 203.0.113.5, so it discards the reply. The device has to rewrite the source as well, so the reply comes back through it.

Fixes, best first

1. Split DNS: the internal view of app.example.com returns 192.168.20.10. No hairpin at all.
2. A DMZ on its own interface: clients and servers on different interfaces, so replies always return through the edge device.
3. Hairpin NAT on the edge device. On Linux:

iptables -t nat -A PREROUTING -d 203.0.113.5 -p tcp --dport 443 -j DNAT --to-destination 192.168.20.10
iptables -t nat -A POSTROUTING -s 192.168.20.0/24 -d 192.168.20.10 -p tcp --dport 443 -j MASQUERADE

The first rule has no -i, so it applies to inside traffic too. The second source-NATs clients on the server's own subnet so their replies return through the router.

On classic IOS the NAT Virtual Interface (ip nat enable on interfaces, ip nat source ... rules without inside or outside) supports hairpinning, but NVI is not supported on IOS-XE.

The pipeline

1. Clock filter: per server, keep the last 8 samples and use the one with the lowest delay. Low delay means the least room for error.
2. Selection: an intersection algorithm finds the range most servers agree on. Servers outside it are falsetickers.
3. Cluster: prune the survivors that sit furthest from the rest.
4. Combine: a weighted average of the survivors. One of them is the system peer.
5. Discipline: steer the local clock. ntpd slews offsets below 128 ms and steps larger ones.

Reading it on IOS

R1# show ntp associations

  address         ref clock       st   when   poll reach  delay  offset   disp
*~10.0.0.123      .GPS.            1     38     64   377  0.412   0.087  0.952
+~10.0.0.124      .GPS.            1     51     64   377  0.530   0.142  1.104
-~192.0.2.123     .PPS.            1     17     64   377 18.230   2.904  3.517
x~198.51.100.123  .GPS.            1     44     64   377 21.904 812.335  2.871
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

198.51.100.123 claims stratum 1 yet sits 812 ms away from the other three, so it is marked as a falseticker and ignored. Delay, offset and dispersion are in milliseconds.

reach is an 8-bit register shown in octal: 377 means the last eight polls were all answered.