Microsoft has released, in preview, support for pod sandboxing in the Azure Kubernetes Service (AKS). Available within all Azure regions for a subset of Azure VM sizes, pod sandboxing provides an isolation boundary between the container application and the shared kernel and compute resources of the container host.
Pod sandboxing leverages Kata Containers to provide hypervisor-based isolation per pod. Kata Containers is an open-source virtual machine that operates as a container. It is designed to provide enhanced workload isolation using hardware virtualization technologies. As Kata Containers is compliant with the Open Container Initiative (OCI), the containers work with frameworks such as Docker and Kubernetes.
In response to Howard van Rooijen asking if this is "the first true 'Hard Multitenancy' implementation on Azure", Michael Withrow, senior product manager at Microsoft, replied that they are:
Getting closer to multitenancy, but this is really about kernel isolation to prevent attacks from a shared kernel perspective. There are other things in play down the road which will make multi-tenancy more of a reality.
Kata Containers on AKS run on an Azure hypervisor using Mariner AKS Container Host (MACH). Each Kata pod receives its own kernel per nested Kata guest VM. With this approach, many Kata containers can be stacked within a single guest VM with additional containers running in the parent VM.
The Mariner AKS Container Host was released in preview in the fall of 2022. Mariner is Microsoft's internal Linux distribution that is optimized to run on Azure. The Mariner image is designed to be minimal and does not include graphical components or Windows managers.
The workflow to deploy using Pod Sandboxing is similar to the traditional containerd workflow with a few exceptions. The runtimeClassName kata-mshv-vm-isolation should be added to the pod spec. This will create a VM to serve as the pod sandbox to host the container. By default, this VM has 2 GB of default memory and one CPU core. This can be adjusted via the Container resource manifest.
The following command will deploy a cluster named myAKSCluster with one node in the myResourceGroup:
az aks create --name myAKSCluster --resource-group myResourceGroup --os-sku mariner --workload-runtime KataMshvVmIsolation --node-vm-size Standard_D4s_v3 --node-count 1
The flag --workload-runtime should be set to KataMshvVmIsolation to enable the Pod Sandboxing feature. Only Mariner is supported at the moment, so the flag --os-sku should be set to mariner. The flag --node-vm-size can be set to any Azure VM size that is a generation 2 VM and supports nested virtualization.
The preview release has a number of constraints. This includes a lack of support for Microsoft Defender and Container Insights. In addition, there is no support for Container Storage Interface drivers and the Secrets Store CSI driver in the preview release.
Pod Sandboxing is available as a preview release. Withrow notes that although this feature enables hosting mixed workloads within a single tenant AKS, best practices for multitenancy should still be reviewed and followed.