In a recent article in the Ably Engineering series, Ably's engineer Maik Zumstrull explained why real-time messaging platform Ably does not use Kubernetes.
Kubernetes is, without doubt, a very hot topic and is attracting a lot of interest from developers and companies. It aims to provide a platform to automate deployment, scaling, and operations of containers across clusters of hosts. Kubernetes has become one of the preferred solutions to manage container-based and microservice-oriented platforms.
Ably platform runs on AWS and, although all of their software is deployed in containers, it does not rely on Kubernetes or other runtime layers for their orchestration.
This might come as a surprise to many, given Ably's platform complexity and scalability requirements, and certainly one could find some rationale for it in the fact that Ably began its journey before the Kubernetes' era began. In fact, as Zumstrull explains, using Kubernetes would not provide any significant benefits, while making their system more complex.
Zumstrull goes at great length in his article to describe how Ably deals with resource management, autoscaling, traffic ingress, as well as how things could work in case Ably migrated to Kubernetes. He also explains why benefits specifically brought by Kubernetes, such as multi-cloud and hybrid-cloud readiness, infrastructure as code, and the value of a large community, would not balance Kubernetes costs.
After careful cost-benefit evaluation, it seems that introducing such an enormously expensive component would merely move some of our problems around instead of actually solving them.
InfoQ has taken the chance to speak with Zumstrull to learn more.
InfoQ: Starting with a direct question: do you think the current hype around Kubernetes is unjustified?
Maik Zumstrull: Yes, I think people are overexcited at the moment, but I'd like to stress that we don't think this is primarily a quality issue with Kubernetes. We're aware of a lot of really good work being done on the Kubernetes core services, several adjacent products like MetalLB and Calico, and by SRE teams in various places trying to build user-friendly Managed Kubernetes offerings.
Still, people seem too eager to jump on the hot new thing and introduce Kubernetes without having a good story why they need it and how the considerable investment in deploying Kubernetes will pay off later.
InfoQ: Most of the benefits you describe come from running your production environment on a platform that is highly tailored to your needs. So the question ensues, in which kind of scenarios do you think is it worthwhile to opt for developing your own deployment platform instead of using Kubernetes?
Zumstrull: Almost never! Building your own platform from first principles is a project only organizations with tens of thousands of engineers should try to tackle – and looking at existing examples, even they sometimes just shouldn't.
I wouldn't frame what Ably is doing as "building our own platform". I would describe it as using a thin layer of customization and in-house tooling on top of an existing platform. For us, that existing platform is AWS, for other teams it might be Azure or vSphere or even Kubernetes, and they're all potentially valid choices, depending on the needs of the organization.
One complication with Kubernetes as the platform is that it's not self contained: you'll need an additional platform on which to run the Kubernetes cluster or clusters. So you're either dealing with a cloud provider anyway, or you're trying to run on bare metal and now need a platform for managing physical machines. But depending on what you're trying to achieve as an organization, that might be worth doing.
InfoQ: Could you try to provide an assessment of how complex your deployment platform is, and how long it took for you to create it?
Zumstrull: The initial build of our platform happened before my time at Ably, but I've spoken to people who've been around since day one and they estimate that it took about a year from initial plans to having a running platform that already recognizably resembles what we do today, but there have obviously been many refinements since then.
It's worth noting that this happened before Kubernetes 1.0 was even released, so for Ably, the decision was never whether to build a new venture on Kubernetes or not, but whether it makes sense to retrofit a platform that's already up and running on AWS to Kubernetes after the fact.
If you want to know the full details of how Ably operates its platform without using Kubernetes, do not miss Zumstrull's article.