A Linux firewall starts dropping new connections and dmesg says the conntrack table is full. How do you handle it?
Show the answer
Say this first
When nf_conntrack_count reaches nf_conntrack_max, the kernel drops packets that would create new
entries and logs nf_conntrack: table full, dropping packet. Existing flows keep working, new ones fail,
which looks like random timeouts. Short term I raise the limit and check memory. Then I find out why: a
SYN flood or scan, a leak of half-open entries, or long default timeouts on a box with huge flow counts.
Stateless high-volume traffic, such as a busy DNS server, can be exempted in the raw table.
Go deeper
Confirm
$ dmesg | grep conntrack
nf_conntrack: nf_conntrack: table full, dropping packet
$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262144
net.netfilter.nf_conntrack_max = 262144conntrack -S shows per-CPU counters; a rising drop and early_drop confirm it.
Short-term relief
sysctl -w net.netfilter.nf_conntrack_max=1048576
echo 262144 > /sys/module/nf_conntrack/parameters/hashsizeKeep the hash table at a sensible fraction of the max (a quarter is a common choice) so lookups stay fast. Each entry costs kernel memory, so check free RAM before multiplying the limit.
Find the cause
conntrack -L -o extended | awk '{print $3}' | sort | uniq -c # entries per protocol
conntrack -L -p tcp --state SYN_SENT | wc -l # SYNs never answered
conntrack -L -p tcp --state SYN_RECV | wc -l # half-open, answered once
conntrack -L -p tcp | awk '{print $5}' | sort | uniq -c | sort -rn | head # top sourcesStructural fixes
- Shorter timeouts where safe, for example
nf_conntrack_tcp_timeout_establishedwell below 5 days. - Exempt stateless traffic you do not need to track:
iptables -t raw -A PREROUTING -p udp --dport 53 -j CT --notrackand the matching OUTPUT rule. These packets are then UNTRACKED and need their own explicit ACCEPT rules in filter. - Rate limiting or SYN cookies upstream for floods.
They will ask next
Why not just disable conntrack on the box?
NAT and every
ct stateor--ctstaterule depend on it. You can only exempt specific traffic, and only traffic that needs neither NAT nor stateful matching.Why can a Kubernetes node hit this when the cluster is not under attack?
kube-proxy in iptables mode NATs every service connection, and nodes running many short-lived connections (DNS lookups to CoreDNS through the service IP are the usual one) can fill the table. Raising the limit, NodeLocal DNSCache, or an eBPF data plane are the usual answers.
Traps
- Raising nf_conntrack_max without checking memory or the hash size.
- Calling it a network outage. Established traffic still flows; only new flows fail.