An EKS cluster stops scheduling pods because the subnets ran out of IPs. Why does that happen, and how do you fix it?
Show the answer
Say this first
The AWS VPC CNI gives every pod a real VPC address from secondary IPs on the node's ENIs, and it keeps a warm pool of spare addresses per node. Busy clusters burn through small subnets fast. Fixes, in order of effort: tune the warm pool, turn on prefix delegation so each ENI slot holds a /28, use custom networking to put pods in a large secondary CIDR such as one from 100.64.0.0/10, or run the cluster on IPv6.
Go deeper
Why it runs out
- Max pods per node is bounded by ENIs times IPs per ENI. The classic formula is
ENIs x (IPs per ENI - 1) + 2. Anm5.largehas 3 ENIs with 10 IPs each, so 3 x 9 + 2 = 29 pods. - The CNI pre-allocates: by default it keeps a whole spare ENI's worth of addresses warm on every node. Fifty nodes can reserve hundreds of idle addresses.
Fixes
| Fix | What it does | Watch out for |
|---|---|---|
WARM_IP_TARGET, MINIMUM_IP_TARGET | Keep fewer idle IPs per node | More EC2 API calls when pods churn |
| Prefix delegation | Assigns /28 prefixes (16 addresses) per ENI slot, raising pods per node | Needs free contiguous /28s; fragmented subnets fail. Nitro instances only |
| Custom networking | ENIConfig per AZ puts pods in subnets from a secondary VPC CIDR | The primary ENI no longer hosts pods, so max pods drops |
| IPv6 cluster | Pods get IPv6 from a huge prefix | Every dependency must handle IPv6 or go through egress translation |
Custom networking sketch
- Add a secondary CIDR to the VPC, for example
100.64.0.0/16. - Create pod subnets per AZ from it.
- Set
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=trueand create oneENIConfigper AZ. - Pods take addresses from the large range; nodes keep their routable addresses. Pod traffic leaving the VPC is SNATed to the node IP by default, so the 100.64 range does not need to be routable on-prem.
They will ask next
How do you see current IP usage?
aws ec2 describe-subnetsshowsAvailableIpAddressCountper subnet. On the cluster, the CNI'sipamdmetrics show assigned and warm addresses per node.Why might pods be unreachable from on-prem after custom networking?
The 100.64 range is not advertised over VPN or Direct Connect, and pods SNAT to the node address on the way out. Inbound calls should go to a load balancer in routable subnets, not to pod IPs directly.
Traps
- Sizing EKS subnets as /24s because "we only have ten nodes".
- Enabling prefix delegation on a fragmented subnet and getting allocation failures.