Ephemeral containers in Kubernetes 1.23 provide a way to attach temporary containers to a running pod's process namespace and run debugging utilities. The kubectl command's debug option can be used to launch such a container with a user-chosen image.
This feature attempts to solve the problem of Kubernetes native troubleshooting. This often involves opening a terminal into the running container. Without the standard set of troubleshooting utilities that we are used to in any Linux distribution, logging into the container is of no use. Many container images are based on minimal distributions like Alpine or built on distroless which lack these utilities. These restrictions exist to reduce the attack surface of vulnerabilities but they also make debugging harder.
In Kubernetes, a container is part of a bigger unit - the Pod. A Pod can have multiple containers, some of which could be init containers. Ephemeral containers are temporary containers that can be added to a running pod with the kubectl debug command.
kubectl -n myns debug my-pod -it my-ephemeral-container –image=ubuntu –target=target-container
where target-container is the container (part of my-pod) to be debugged. This command launches an interactive terminal into the process namespace of the target container. The –image parameter lets you choose the ephemeral container's image. This means that you can choose the debugger image and include whatever tools you need in it.
Kubernetes does not reserve any resources for these containers and they are not restarted on exit. Some of the usual container attributes like ports, livenessProbe, and resource requests are not supported for ephemeral containers.
Lee Verberne, site reliability engineer at Google - who has been involved in the implementation from the beginning - describes how Google's in-house container orchestrator Borg solves the debugging problem:
Borg provides a common userland for processes. Rather than including system and debugging utilities with the application binary, Borg provides a basic set of userland utilities that applications can expect in their runtime environment.
Borg is a forerunner of Kubernetes.
One solution is to copy debugging tools to the container, but as Verberne notes - "Kubernetes deploys binaries using containers, so it's natural to use containers for troubleshooting as well".
The kubectl debug command also supports other options as of 1.23. These include attaching an ephemeral container to a copy of a running pod. The copied pod can be run with the original command that the pod's container ran with or with a different command. kubectl debug also allows you to create a new pod that can run in the node's host IPC, Network, and PID namespaces and can access the node's filesystem. This is useful if you need access to the host's filesystem or processes. GKE provides a similar, custom toolbox container with utilities in its clusters which can access the host filesystem - but it does not have the other features of kubectl debug.
It is important to note that process namespace sharing has to be enabled at both the cluster level as well as in the debugged pod for ephemeral containers to work. It has been enabled by default at the cluster level since Kubernetes 1.17. If it's not enabled for the debugged pod then one option would be to launch a copy of the pod with the option enabled and then attach the ephemeral container. However, this won't work in situations where the problem does not recur in the copied pod.
Ephemeral containers are part of a multi-release plan to support the kubectl debug command. First introduced in Kubernetes release 1.16, they made it into beta in the 1.23 release. Ephemeral containers are turned on by default in Kubernetes 1.23 onwards.