Amazon EKS Is Eating My IPs!
Understand how AWS EKS manages IP addresses and what you can do about it
In this article, I want to illustrate how the AWS Elastic Kubernetes Service allocates IPs for the cluster so you can better manage your platform!
We will create a new EKS cluster and perform some other actions. At every step, we will seek to understand what is happening in terms of IP allocation.
VPC
We have a VPC with a 10.0.0.0/16 IPv4 range: 65,536 IPs (2¹⁶).
Subnets
I’ve created two public subnets in the VPC that I will use for the cluster. 2¹² = 4,096, but notice how there are 4,091 available IPs for each cluster. In AWS, 5 IPs per block are not available (the first four and the last).
Note: This is not at all recommended for an EKS subnet structure. Nodes should always live in a private subnet. But it is sufficient for the demo. I also associated an IGW with the VPC and both subnets with a route table that routes any traffic outside of the VPC range to the IGW.
EKS
I created an EKS cluster with a t3.2xlarge Node Group of size 2. This will create an Autoscaling Group (ASG) and spin up the EC2 instances for the cluster workers. In EKS, AWS manages the control plane of which you have no visibility.
What Has Happened to the IP Allocation?
Interesting! An empty two-node cluster has used up 62 IP addresses. Let’s work out why!
Access config
Set up our EKS cluster kubeconfig so we can use kubectl to investigate. I already have the AWS CLI configured.
aws eks --region eu-west-2 update-kubeconfig --name testWhat is deployed?
The two nodes will take two IPs from the cluster.
What is deployed inside the cluster then?
kubectl get pods -A
There are six pods running. These are DaemonSets and Deployments that are EKS add-ons used to make the cluster function correctly.
OK, cool. So, that’s a total of eight IPs we think should be in use. What about the other 54? We don’t have any other workloads in the cluster. There are no Load Balancers eating up space. There are no out-of-cluster resources like EC2, databases, VPC endpoints, etc. What sort of AWS magic is going on here?
The answer lies in how EKS manages networking.
Pod Networking — Container Networking Interface
According to the documentation, the Amazon VPC CNI is responsible for “VPC IP addresses to Kubernetes nodes and configuring the necessary networking for pods on each node.”
“The L-IPAM daemon is responsible for attaching elastic network interfaces to instances, assigning secondary IP addresses to elastic network interfaces, and maintaining a ‘warm pool’ of IP addresses on each node for assignment to Kubernetes pods when they are scheduled.” — AWS docs
Now that we understand this, we can piece together how those IPs are being allocated. ENIs are attached to the nodes, providing a means of attaching IPs for each pod in the cluster. And some excess IPs are reserved to maintain this warm pool so that pods can obtain IPs faster and thus spin up quicker.
What is dictating the size of the warm pool?
1. ENI Capacity
Each EC2 instance type has a different ENI capacity.
We opted for t3.2xlarge, which has an ENI cap of four and each ENI can hold 15 IPs.
2. AWS VPC CNI Configuration
This component can be configured to choose how IPs are kept warm, and there are lots of settings you can look into. The default is a WARM_ENI_TARGET=1 setting. This means that EKS will attempt to keep one entire ENI spare on the node. So if a node has an ENI attached and any of those IPs are used, then it will attach another ENI so this default setting is observed.
Things Now Add Up!
- 2x nodes = 2 IPs.
- Each node has an ENI attached: 2x15 = 30 IPs.
- Each node has pods scheduled on it, so it obeys the setting above and attaches a spare ENI to each node: 2x15 = 30 IPs.
- Total: 2 + 30 + 30 = 62
BOOM!
Configuring the CNI
kubectl -n kube-system set env daemonset aws-node WARM_IP_TARGET=2
By doing this, I have changed the way that EKS will manage the pool. Instead of the default, EKS will only attempt to keep two additional IPs free per node.
We can now see the cluster has consumed 12 IPs! That’s one per node, one per pod, and two extra per node, as expected.
Takeaways
- We now understand how EKS manages IPs and how we can take control of this.
- When you move into more complex architectures, EKS and AWS in general can end up using a lot of IPs. Don’t make this a problem for yourself and make sure you have lots of room for growth in your networking!
