Octet

A Linux firewall starts dropping new connections and dmesg says the conntrack table is full. How do you handle it?

Show the answer

Say this first

When nf_conntrack_count reaches nf_conntrack_max, the kernel drops packets that would create new entries and logs nf_conntrack: table full, dropping packet. Existing flows keep working, new ones fail, which looks like random timeouts. Short term I raise the limit and check memory. Then I find out why: a SYN flood or scan, a leak of half-open entries, or long default timeouts on a box with huge flow counts. Stateless high-volume traffic, such as a busy DNS server, can be exempted in the raw table.

Go deeper

Confirm

Output
$ dmesg | grep conntrack
nf_conntrack: nf_conntrack: table full, dropping packet
$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262144
net.netfilter.nf_conntrack_max = 262144

conntrack -S shows per-CPU counters; a rising drop and early_drop confirm it.

Short-term relief

Shell
sysctl -w net.netfilter.nf_conntrack_max=1048576
echo 262144 > /sys/module/nf_conntrack/parameters/hashsize

Keep the hash table at a sensible fraction of the max (a quarter is a common choice) so lookups stay fast. Each entry costs kernel memory, so check free RAM before multiplying the limit.

Find the cause

Shell
conntrack -L -o extended | awk '{print $3}' | sort | uniq -c   # entries per protocol
conntrack -L -p tcp --state SYN_SENT | wc -l                   # SYNs never answered
conntrack -L -p tcp --state SYN_RECV | wc -l                   # half-open, answered once
conntrack -L -p tcp | awk '{print $5}' | sort | uniq -c | sort -rn | head   # top sources

Structural fixes

  • Shorter timeouts where safe, for example nf_conntrack_tcp_timeout_established well below 5 days.
  • Exempt stateless traffic you do not need to track: iptables -t raw -A PREROUTING -p udp --dport 53 -j CT --notrack and the matching OUTPUT rule. These packets are then UNTRACKED and need their own explicit ACCEPT rules in filter.
  • Rate limiting or SYN cookies upstream for floods.

Traps

  • Raising nf_conntrack_max without checking memory or the hash size.
  • Calling it a network outage. Established traffic still flows; only new flows fail.
How did you do?