Fun fact about AmazonVPC CNI in Kubernetes
By Alberto on November 15, 2020
I’ve always said it. Kubernetes is all about Networking. It is a central part of Kubernetes, but it can be challenging to understand exactly how it is expected to work.
Kubernetes networking is a model that k8s employs to enable communication between its components. It is based on a flat network structure and does not require you to map ports between hosts and containers. Although Kubernetes networking can be a challenge to set up, it is an essential part of any k8s operation and one that you need to understand for successful deployment.
This post, I intend to keep it simple and will not attempt to explain or describe the Kubernetes networking operation. For that I recommend the excellent documentation of k8s here.
As we well know there are quite a few networking plugins that work with Kubernetes. To mention the most popular:
An there are many others…
The one I’m interested in writing in this post is the native kubernetes plugin for AWs, called AmazonVPC.
The versatility of this networking plugin is that the IP addresses obtained by the Pods are the same from the VPC network, which allows abstracting and not hiding in a certain way as is normally done in Kubernetes, maintaining a private network within the cluster only and exclusively for pods.
One of the benefits that we can clearly see with this plugin is the ease of integrating and connecting with EC2 instances and AWS services that run in our VPC or in some other VPC or Region, maintaining direct communication through a VPC Peering.
But also one of the big drawbacks that I saw is that depending on the type of EC2 instance that your Kubernetes nodes have, the maximum number of pods that each node will support will depend. Let’s see how it works.
It should be noted that the AmazonVPC networking plugin is the same one that the AWS EKS service uses natively.
When you create an Amazon EKS node, it has one network interface. All Amazon EC2 instance types support more than one network interface. The network interface attached to the instance when the instance is created is called the primary network interface. Any additional network interface attached to the instance is called a secondary network interface. Each network interface can be assigned multiple private IP addresses. One of the private IP addresses is the primary IP address, whereas all other addresses assigned to the network interface are secondary IP addresses.
Each pod that you deploy is assigned one secondary private IP address from one of the network interfaces attached to the instance. For instance an m5.large instance supports three network interfaces and ten private IP addresses for each network interface. Even though an m5.large instance supports 30 private IP addresses, you can’t deploy 30 pods to that node. To determine how many pods you can deploy to a node, use the following formula:
(Number of network interfaces for the instance type × (the number of IP addressess per network interface - 1)) + 2
Using this formula, an m5.large instance type can support a maximum of 29 pods. For a list of the maximum number of pods supported by each instance type click here or here.
This is very important to take into account when deploying your Kubernetes cluster with the AmazonVPC networking plugin either in EKS or using any other deployment strategy, you can always add a new node to be able to deploy more pods as well, but it has implications in increasing cost $$$ and is something that the Architect must take into account.
To finish few useful commands:
- Retrieve the capacity of pods by nodes:
kubectl get nodes -o json | jq '.items.status.capacity.pods'
- Retrieve IPs of all pods in all namespaces:
kubectl get pods -A -o json | jq '.items.status.podIP'
As usual, if you have any question, send me a message at firstname.lastname@example.org