Asana, a work management platform for teams, reflected on their Kubernetes adoption. Tony Liang, software engineer at Asana, elaborated on how the team built KubeApps, a framework to systematize the creation and maintenance of Kubernetes applications.
The Infrastructure Platform Team at Asana now uses Kubernetes to deploy and manage services independent of their monolith infrastructure. Before adopting Kubernetes, using a monolithic architecture in combination with Asana's legacy deployment system presented challenges with scaling. Also, adding new services entailed the risk of instability with deployments.
The platform team at Asana embraced their engineering team's interest in using Kubernetes. The goal with adopting this framework was to enable application developers to build and update services without affecting other existing services. This application engineering led approach is important to note, as in a recent InfoQ Q&A a series of Kubernetes experts emphasize the need for developer advocacy in order to drive successful adoption.
Still, the team was required to handle some problems that Kubernetes did not solve for them out of the box. AWS resource management was beyond the scope of Kubernetes. It was monotonous to do metrics collection and secrets permission across all services. Finally, for continuous delivery, application logic was expected to be packaged into images readily available in the container registry.
The Asana team created the KubeApps framework, which initially handled different aspects of deployments. Every KubeApp was defined by a set of hardware requirements, pod definitions, and service applications. KubeApp configuration exists as Python code, which facilitates programmatically generated configurations and external resources management.
KubeApp builds and pushes the Bazel Docker images or Dockerfiles images through a continuous delivery pipeline. Application code is wrapped in these images. Each KubeApp is deployed in its Kubernetes Cluster via AWS managed Elastic Kubernetes Service (EKS). This minimizes the impact of cluster failure as there is one KubeApp service per cluster.
This adoption of a managed Kubernetes service is also in line with expert opinion; a large number of organizations delegate cluster maintenance to a cloud provider, using a managed services such as EKS, Google Kubernetes Engine (GKE), or Azure Kubernetes Service (AKS).
The diagram below shows KubeApp deployments at Asana. A central management hub, kubecontrol, runs the updates via crons or as triggered by developers.
Source: https://blog.asana.com/2021/02/kubernetes-at-asana/
The creation or updates to KubeApp are triggered via kubecontrol. As per the application specifications, a set of resources are requested for KubeApp to create a new EKS cluster. The image builder service compiles Docker images as per the provided code version, and then commits the image to Elastic Container Registry (ECR). Upon building all required resources, the component specifications are provided to the Kubernetes cluster so that required Docker containers are deployed.
KubeApps implements blue/green deployments and needs a new EKS cluster to be launched and configured. After the latest KubeApp is verified, the traffic is shifted to the new cluster, and the old cluster is decommissioned.
From an observability standpoint, KubeApp Management Console (KMC) records historical information about deployments. The team plans to use KMC as a centralized web-based interface for users to interact with KubeApps.
With over 60 KubeApps running, there is still a monolith of EC2 instances at Asana. The Infrastructure Platform team continues to build new features on the KubeApps framework, extending support for more types of architectures and infrastructure providers.
Several other organizations have created similar developer-focused application management frameworks, for example, Netflix has a "Paved Road Platform as a Service" and Shopify has built a Heroku-like framework on top of Kubernetes.