Coinbase recently wrote about why Kubernetes is not part of their technology stack. Coinbase uses containers, but they run them in VMs. For deployments, they use Odin, its open-source solution for deploying their services in VMs as auto-scaling groups. Adopting Kubernetes adds unnecessary complexity to their current deployment pipeline. Moreover, they'd prefer to explore other options like Fargate or ECS before directly choosing Kubernetes. Coinbase expressed that Kubernetes is not the right tool for them at the moment.
From a technological perspective, Kubernetes doesn't solve any of Coinbase's customer problems. Conversely, for them, Kubernetes creates a new set of challenges. For instance, they'll need to dedicate a team to build the necessary infrastructure to run their services. Also, they'll have to translate their current security practices for Kubernetes. Moreover, Coinbase said that existing managed services from cloud providers, like EKS or GKE, are not mature enough yet. For instance, if they had to upgrade a cluster, they said it would require "a much more operationally-heavy focus" than they currently have.
When someone asked Kelsey Hightower what his thoughts were on Coinbase not using Kubernetes, he replied:
Coinbase built and maintains their own platform that's working for them. Coinbase provided an analysis worth studying. The major takeaway for me: asking people to manage their own Kubernetes cluster is like asking people to manage their own hypervisors when they just want VMs.
Coinbase's existing technology stack consists primarily of containers running in EC2 instances. For application service discovery, they use Route53 in conjunction with application load balancers and Envoy. They scale their services through auto-scaling groups (ASGs). They also use lambda functions to schedule deployments through step functions. Coinbase uses Odin, the orchestration platform they built to deploy their services as ASGs, and Codeflow, their internal tool to manage deployments through a UI. Odin has all the logic for doing deployments progressively using health checks and can even perform rollbacks when needed.
Coinbase defines the desired state specifications of their services like instance type or security groups through a JSON manifest, somewhat similar to a Kubernetes YAML manifest. In this regard, Drew Rothstein, director of engineering at Coinbase, said:
We enable the same key features of Kubernetes: a single deploy + rollback button in Codeflow, scaling based on some defined heuristic (we support custom AWS metrics or standard CPU metrics), and re-scheduling/moving of your containers if your VM dies or becomes unhealthy in your ASG.
At the moment, Coinbase doesn't have any plans to use Kubernetes to orchestrate their services as they don't need to fulfill an advanced use case for container orchestration. Instead, they would like to evaluate other options before directly adopting Kubernetes. For instance, as their infrastructure runs in AWS, they would evaluate managed services like Fargate or ECS. Additionally, Odin already fulfills their current needs, and there wouldn't be any significant gains to their engineers if they made the switch to Kubernetes.
Moreover, Coinbase is basing this decision primarily on customer needs, to the point where Rothstein said: "If the barrier to entry of our current platform were to significantly change and that was now a clear differentiator, then we would also explore offering a different platform." Coinbase would prefer to have a customer-focused reason to implement a change in its orchestration platform. For instance, customers' potential needs would be reducing deployment times, and having different deployment patterns beyond canaries or blue/green, or complex service mesh interactions from what they currently have.
Finally, Coinbase expressed that they don't think Kubernetes is a bad tool; it's just not the tool for them. Coinbase believes that Kubernetes is great despite its existing challenges, and projects like Knative or Fargate are increasing the level of abstraction to solve many of these challenges.