AWS announced the support of Kubernetes version 1.27, called Chill Vibes, for Amazon EKS and Amazon EKS Distro. In this version of Kubernetes there are a lot of new features that are generally available and some of them are potentially destructive for the clusters.
In Kubernetes 1.27, seccomp is graduated as stable and activated by default: the RuntimeDefault seccomp profile will be used as default for all workloads. Passing the –kubelet-extra-args "–seccomp-default" flag in the node bootstrap script or launch template will enable the default seccomp profile for all containers running on the node. In this way, the seccomp profile is defined by the container runtime, instead of using the unconfined (seccomp disabled) mode. When the seccomp profile is enabled, some workloads may experience breakages, but it is possible to disable or create custom profiles for specific workloads. The security-profiles-operator allows for defining and managing custom profiles for the workloads.
This Kubernetes version has some features that allow better management of the pod topology and an easier way to spread balanced pods across various domains. In particular, issue #3022 unveils the minDomains parameter, giving the administrator the ability to set the minimum number of domains your pods should occupy, thereby guaranteeing a balanced spread of workloads across the cluster. Issue #3094 introduces the nodeAffinityPolicy and nodeTaintPolicy parameters, which allow for an extra level of granularity in governing pod distribution according to node affinities and taints. This particular feature is linked with the NodeInclusionPolicyInPodTopologySpread gate. Lastly, issue #3243 implements the matchLabelKeys field in the topologySpreadConstraints in the pod’s specification, which permits the selection of pods for spreading calculations following a rolling upgrade.
In the previous versions, the Amazon EKS kubelet had a 10-requests per seconds limit for kubeAPIQPS with a burst limit of 20 requests for kubeAPIburst. In the 1.27 version, the kubeAPIQPS limit is raised to 50 requests per second and the kubeAPIBurst is now 100 requests per second. These new limits are adopted by the Amazon EKS optimized AMI and improve the pod start time when there is a demand for scaling requirements. These new limits allow the Amazon EKS kubelet to manage faster the pod startups and enables smoother cluster operations.
In version 1.27 some APIs are deprecated as the other Kubernetes releases: k8s.gcr.io is frozen and now registry.k8s.io is the new repository for the Kubernetes images. It is important to update all the manifests and the configurations.
The seccomp alpha annotations (seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io) have been removed, these annotations were already deprecated in version 1.19. A possible script to check where these annotations are used in a specific cluster is the following:
kubectl get pods --all-namespaces -o json | grep -E 'seccomp.security.alpha.kubernetes.io/pod|container.seccomp.security.alpha.kubernetes.io'
Since version 1.24 the default container runtime for Amazon EKS has been containerd. In 1.27 the --container-runtime command for kubelet is removed so It is mandatory to remove the --container-runtime argument for all the node creation scripts and workflow. In Terraform it’s important to remove the bootstrap_extra_args field:
node_groups = {
eks_nodes = {
desired_capacity = 2
max_capacity = 10
min_capacity = 1
instance_type = "m5.large"
k8s_labels = {
Environment = "test"
Name = "eks-worker-node"
}
additional_userdata = "echo foo bar"
bootstrap_extra_args = "--container-runtime=your-runtime"
}
And in eksctl:
nodeGroups:
- name: your-nodegroup-name
instanceType: m5.large
desiredCapacity: 3
minSize: 1
maxSize: 4
kubeletExtraConfig:
container-runtime: "your-runtime"
It is important to upgrade the Amazon EKS cluster to a supported version. The latest Amazon EKS that is out-of-support is 1.22 and the end-of-support date was June 4, 2023, the next one is version 1.23 which will be end-of-support in October 2023.