Transcript
Gamanji: My name is Katie Gamanji, and I am one of the cloud platform engineers for American Express. Around a month ago, I joined American Express, and I am part of the team that aims to transform the current platform by embracing the cloud-native principles and making the best use of the open-source tools. As well, quite recently, I've been nominated one of technical oversight committee member, or TOC, for the CNCF. Alongside 10 other champions within the open-source community, I'll be responsible to steer and maintain the technical vision for the CNCF landscape. Pretty much we are the committee that enables and leverages products to join the open-source community.
Today, I would like to talk about the interoperability of open-source tools, and more specifically, the emergence of interfaces.
Container Orchestrators
Six years ago, the container orchestrator framework space was very heavily diversified. We had tools such as Docker Swarm, Apache Mesos, CoreOS Fleet, Kubernetes, and all of them provided a viable solutions to run containers at scale. However, Kubernetes took the lead in defining the principles of how to maintain and distribute containerized workloads. Nowadays, Kubernetes is known for its scalability and portability, but more importantly, for its approach to declarative configuration and automation. This prompted for multiple tools to be built around Kubernetes to extend its functionalities. This created what we today know as the cloud-native landscape, which resides under the CNCF umbrella, Cloud Native Computing Foundation.
However, if you look into every single tool, they're going to provision quite similar functionality sometimes, and at the beginning, they had very different ways to converge with the Kubernetes components. It was clear that it was necessary to introduce a set of standards, and it was imperative to have the interfaces around. This is what I would like to talk to you about today.
To do so, I would like to introduce the container network and container runtime interfaces, and how this paved the path toward standardization and guidelines within the Kubernetes ecosystem. These two components were necessary to make the transition between the VMs and containers as easy as possible. In the next stage, it's something which identify as the innovation wave. We have the community concerning itself more and more with the extensibility of Kubernetes. This is confirmed by the appearance of the service mesh and storage interfaces, as well as cluster API. Lastly, I would like to conclude with the impact that the emergence of interfaces had on vendors and users in the community.
Before I move forward, I would like to introduce Kubernetes in numbers as it is today. Based on the CNCF survey in 2019, more than 58% of the companies are using Kubernetes in production. The other 42% are actually prototyping Kubernetes as a viable solution. Another milestone I would like to mention is that more than 2,000 companies are using Kubernetes in an enterprise context. This is a very important milestone because it showcases the maturity and the high adoption rate for communities. When we move towards the development community, more than 2,000 engineers are actively contributing towards the future build out and bug fixing. When we look into the end user community, more than 23,000 attendees were registered at the QCons around the world last year. This is going to be QCon in Europe, China, and North America.
The Practical Past
However, the community around Kubernetes was not always as developed and flourishing, but more importantly, it was not always as engaging. At the beginning, the picture was quite different. Nowadays, Kubernetes is known for its adaptability and flexibility to run containerized workloads with predefined technical requirements. It will be able to provision the ecosystem for application execution, while shrinking its footprint in the cluster. It's all about efficient management of the resources.
However, to reach the state of the art, complex challenges required solutionizing, such as the support for different networking and hardware systems. This prompted for the CNI and CRI to be introduced, so pretty much the container network interface and the container runtime interface.
CNI was introduced under the CNCF umbrella in early 2017, while the CRI was introduced in Kubernetes 1.5 in its alpha release. I would like to deep dive a bit more into this topic because I find them quite pivotal in terms of the standardization, but more importantly, when it comes to the transition between the VMs and containers, it's all about keeping the mindset and changing the perspective.
CNI
Exploring the networking fabric within a Kubernetes cluster is quite a challenging task. Kubernetes is known for its ability to run distributed workloads on a distributed amount of machines, while preserving the connectivity and reachability to these workloads. As such, the networking topology is highly assertive and it gravitates towards the idea that every single pod has a unique IP. This particular networking model dismisses the need for dynamic port allocation, but at the same time brings to light new challenges to be solved, such as how containers, pods, services, and users are able to access our application.
To showcase exactly where the CNI component is injected, I would like to showcase the journey of a package to be sent across two different applications on an internode setup. In this example, I have two nodes, and supposedly I'm going to have two different applications running on that. I would like to send a request from application one to application two. Once the request is issued, it's actually going to look inside the pod to see if any containers are able to serve the request. For the sake of this example, will not be able to do so, which means the request is going to go outside of the pod, and it's going to reach the root network namespace on the physical machine. At this stage, we're going to have the visibility of all the pods on that specific node. Again, will not be able to serve our request, which means the request is going to go outside of the machine for the [inaudible 00:06:33] device towards the routing table. Generally speaking, the routing table is going to have the mapping between every single node and decider for the pod's IP to be allocated on that node, which means that with minimal hops, we'll be able to reach our machine, and in a reversive matter, will go for the root network namespace to the pod/container, and our request is going to be served.
The networking world in Kubernetes dictates that every single pod should be accessible via its IP across all nodes. As such, it was necessary to have this inclusivity of different networks and networking systems to make sure that this principle is fulfilled. This prompted for the appearance of the CNI, or container network interface, which concerns itself with the conductivity of the pods and the deletion of resources when the pod is removed. Pretty much it's going to have two operations: addition and deletion. It will make sure the pod has an IP, but at the same time to make sure to clean up resources when the pod is not going to be there anymore.
When you're looking into the ecosystem at the moment, there are a plethora of tools provisioning the networking fabric for a cluster, from which we have Flannel, which has reported a lot of success from the end user community. Flannel is known for its simplicity to introduce the network overlay for a cluster. Calico as well has gained a lot of momentum lately. This is because, in addition to the network overlay, it's going to introduce the network policy enforcer, which allows the users to configure a fine-grained access control to the services within the cluster. Lately, Cilium as well has gained a lot of popularity. This is because it's aware of the networking and application layer security protocols. It will allow the Layer 3 and Layer 7 configuration, which makes it possible to have the transparency of the networking packets at the API level. Weaveworks actually came with their own tool for the networking component called Weave Net. It's known for its simplicity to be installed in the cluster with minimal configuration. It's actually a one-line installer.
CRI
When we transition to the runtime component, the story is slightly different. At the beginning, most of the containers or the runtime capabilities would be provisioned by Docker and Rocket. The runtime is the particle which intercepts API calls from the kubelet, and it will make sure to create the containers in the machine with the right specification. As I mentioned, Docker and Rocket would be the only supported runtimes at the beginning, and their logic would be very deeply ingrained within the Kubernetes source code. This presented quite a few challenges. If you'd like to introduce new features to the existing runtimes, that would be very tightly coupled with the release process for Kubernetes, which is quite lengthy. It actually allows for very low velocity rate when it comes to feature development. As well, if you'd like to introduce new runtimes, that will present a very high level entry bar because it will require a very in-depth knowledge of the Kubernetes source code. Again, this is not sustainable to move forward, and to comply with the growing space within the runtime components.
It was clear that it was necessary to have an interface which will enable the runtimes to be integrated. As such, the container runtime interface came to be about, and it provides an abstraction layer for the integration of runtime capabilities from which Docker would be just one of the supported runtimes.
When you're looking into the CNCF landscape, there are plenty of tools provisioning the runtime capabilities from which we have ContainerD and CRI-O, being the most widely used. This is because of the open-source nature. ContainerD is currently a graduated CNCF project, so it has a lot of use cases from the end user community, but as well it actually showcases its maturity. It's known for its industry standard to provision the runtime for a cluster. CRI-O is an incubating CNCF project, and it's known for its lightweight capabilities, but at the same time complying with open-source container initiative standards. The implementation of the runtime is going to be very tightly coupled with the infrastructure provider. It is only natural to allow the existing cloud providers to use their own APIs to create the containers on their machines. As such, Google is going to have their own runtime component which is going to be called gVisor, and AWS has their own component which is going to be AWS Firecracker.
The Innovation Wave
The networking and runtime components were extremely essential when it comes to the migration between the VMs and containers. This is because we actually keep the mindset in terms of reachability of our workloads would be able to do so via the IP. The same would do with the VMs. We're going to dig in the container world as well. The runtime capability actually allowed us to create the containers on the machine. These two components actually exceeded the rate for the adoption of Kubernetes moving forward. From this point, the community concerns itself more and more with the extensibility of Kubernetes rather than settling down to a very specific amount of tooling. This is confirmed by the appearance of Service Mesh Interface, or SMI, Container Storage Interface, or CSI, and ClusterAPI.
SMI
The Service Mesh Interface was introduced in QCon Barcelona 2019. It provides a solution to democratize the integration of service mesh within a Kubernetes cluster. I would like to step a bit back and just introduce the service mesh capabilities. Service mesh is a dedicated infrastructure layer that concerns itself with the traffic to be sent across services in a distributed ecosystem. What it actually means is that it focuses on the traceability and more importantly of the transparency, how the services communicate between themselves. This is important feature when we have an ecosystem where you have microservices as the basis.
The Service Mesh Interface is going to cover three areas of configuration. There's going to be the traffic policy, traffic telemetry, and traffic management. Traffic policy allows the configuration of fine-grained access control of how the services will be communicated between themselves. Traffic management, it's all about the proportional traffic to be sent across services. This is an important feature if you're thinking about canary rollouts. Traffic telemetry is concerning itself how to capture and expose metrics to make sure that we have the full transparency of communication between the services.
We have multiple tools provisioning the service mesh capabilities, but only three of them integrate with CMI, from which we have Istio provisioned by Google, we have Consul provisioned by HashiCorp, and Linkerd provisioned by Buoyant. It's worth to mention here that Linkerd is currently incubating CNCF project, and it's known for its simplicity to actually introduce the service mesh functionalities within a cluster.
CSI
When we transition to the storage space, I think this is one of the most developed and widely used areas. The story with the storage component is very similar to the runtime component. This is because it was very deeply ingrained, its logic was very deeply ingrained within the Kubernetes source code. This allowed for a very low feature development rate and a very high level bar for the new storage providers to be introduced. As such, the interface came to be about and it promotes a pluggable system for the applications to consume external storage. The CSI was introduced in Kubernetes 1.9 in its alpha release, and it moved to general availability in 1.13.
I was mentioning that this is one of the most widely contributed areas, and this is because more than 60 providers are currently integrating the CSI from which we have StorageOS, Rook, Ceph, OpenEBS, and many more. All of these providers are actually going to focus on the simplicity to configure the storage component, but at the same time, will focus on the dynamic provisioning of the storage capabilities. Again, Rook here is an incubating CNCF project. It actually provides an interface to integrate the Ceph drivers, so it's kind of an inception here.
ClusterAPI
When we transition towards the ClusterAPI, the perspective is changed completely. This is because when he talked about the networking and runtime, service mission, storage components, all of these are residing within the Kubernetes world. ClusterAPI takes the idea of interfaces of further step, it completely rethinks the way we provision our clusters across different cloud providers. I would like to deep dive a bit more into this tool, as I find it quite crucial in the modern infrastructure nowadays. Looking to the current tools, there are plenty of providers providing the bootstrap capabilities for a Kubernetes cluster, from which we have Kubeadm, Kubespray, Kops, Tectonic, and many more.
However, if we look into everything single tool and the supported cloud providers, it's going to be difficult for us to find a common denominator. What it actually means is that every single tool is going to have a very specific cloud providers it's going to support. This exposes quite a few challenges moving forward. Supposedly, what happens if you'd like to rewrite infrastructure, or to migrate infrastructure to a different cloud provider? Even if it's the same bootstrap provider, the end result is going to be that you will have to write your infrastructures code from scratch because there are very little reusable components.
Another use case is what happens if you'd like to change a bootstrap provider altogether. For example, Tectonic, which I introduced earlier, is no longer under active development, and it is to be merged with OpenShift container platform. Moving forward, it's actually quite difficult because we'll not be able to do so unless we further the project and maintain it in house. The end result is going to be rewriting the infrastructures code from scratch again.
As well, I'd like to introduce you some challenges areas to deploy platforms such as China and Russia. This is because, in this particular regions, we have specific tooling to provision our platform capabilities. Most of the time, the engineers will end up with a snowflake infrastructure, which means that we're going to lose the lift-and-shift capability altogether. However, ClusterAPI intercepts all of these challenges, and provides the solution by providing an interface for cluster creation across different cloud providers through a set of declarative APIs for cluster configuration, management, and deletion.
When we're talking about cluster API, we talk about SIG cluster lifecycle, which had its first initial release in April of 2019. Since then, they had two releases and they're actual preparing for a new release this month, which is going to result with a v1aplha3 endpoint. I was mentioning the cluster integrates with different cloud providers, and currently we have a dozen of them, from which we have GCP, AWS, DigitalOcean, Bare Metal, but more importantly, Baidu Cloud. Baidu Cloud is a Chinese provider. ClusterAPI actually enables us to provision cell clusters in China with the same ease we do so in AWS in Europe.
Let's see how ClusterAPI works. Supposedly we'd like to provision a couple of clusters in different regions, different cloud providers. The way ClusterAPI is going to work, it will require a management cluster. For testing purposes, it is recommended to use kind to provision the cluster. Kind is a Dockerized version of Kubernetes. If you'd like to use ClusterAPI in production with one of the [inaudible 00:18:48] adopters, it is recommended to use a fully-fledged Kubernetes cluster. This is because it comes with a more sophisticated failover mechanism. Once we have our management cluster up and running, we'll require our controller managers on top of it. To have a fully working version of ClusterAPI, we'll require three controllers, one for the ClusterAPI CRDs, one for the bootstrap provider, and the infrastructure provider.
ClusterAPI introduces four new resources or custom resource definitions, and will require a controller to make sure that we can create and reconcile any changes we have to these resources. The second controller is going to be the bootstrap provider. This is going to be the component which will translate the YAML configuration into cloud-config script, and it will make sure to attach the instance to the cluster as a node. This capability is currently provided by Kubeadm and [inaudible 00:19:44]. And thirdly, will require our infrastructure provider. This is going to be the component which will actually interact directly with the API and provision the infrastructure, such as the instances, IM roles, DPCs, subnet security groups, and many more.
It's deserved to mention here that you can have one or many infrastructure providers. If you'd like to create a cluster, for example, in DigitalOcean and AWS, you will require the infrastructure provider for both of them. You'll create one infrastructure controller for DigitalOcean and AWS, so it'll actually be able to interact with APIs directly.
Once we have our controller managers up and running, our dependencies are there, we'll be able to provision our target clusters. These are going to be the clusters we're going to deliver to our users and developers.
I would like to introduce the resources introduced by ClusterAPI, because I'm going to use one of them to showcase an example later on. As mentioned, ClusterAPI introduces four new resources: Cluster, Machine, MachineSet, and MachineDeployment. The Cluster resource is going to allow the higher level configuration of a cluster. We'll be able to specify the subnets for all pods and services, as well any DNS suffix, if you have any. The Machine configuration is going to be very similar to a node configuration. We'll be able to say what version of Kubernetes we would like to run, but at the same time to say, what region we'd like our instance to run, or any desired instance type. MachineSet is going to be very similar to replica set. It will make sure that we have an amount of machine resources up and running at all time. MachineDeployment is very similar to deployment. It comes with a very powerful rolling-out strategy between configurations.
It's worth to mention here that the machine resource is immutable within the ClusterAPI context. If we deploy new changes to our machines, the node without configuration is going to be taken down and a new machine with a new configuration is going to be brought up. There is no patching, there is only mutability.
To showcase the simplicity of configuring clusters across different cloud providers, I would like to introduce the current way to do so. This is just a snippet of Kubespray rolls. Kubespray actually provisions the infrastructures code for a cluster using Ansible configuration. In this particular view, we have some of the roles and developer we need to be aware while creating, troubleshooting, and maintaining the cluster. ClusterAPI completely changes the perspective and introduces all of this configuration to a couple of manifests. If you'd like to create a classroom AWS, this is going to be the only configuration you'll need to do.
In this particular case, we have a cluster resource. It's worth to mention here that the cluster resource is going to pretty much take care of major networking components for a cluster. We'll still need to add other machines to the cluster. This is step one. Step two is actually at our control plane, which means our master nodes, and then the worker nodes, but those are separate manifests. What we actually have here is cluster resource with the kind cluster. We actually write it, we're invoking the apiVersion v1alpha2. We give it a name in the metadata spec, which is going to be test cluster. In the actual specification, we choose a /16 subnet for our pods.
I would like to draw your attention towards the infrastructure reference. This is going to be the component which will actually invoke the configuration specific to a cloud provider. This makes sense because every single cloud provider is going to have their own parameters to be configured. What we actually have here, we invoke an AWS cluster resource in v1alpha2 with a name test-cluster. In the background, we're going to invoke this particular manifest. What we actually have here, we say we want our cluster to be created in eu-central-1, that's going to be the region. As well we say we want to attach an sshKey with the name default to our instance.
If you'd like to create this particular resource in GCP, these are going to be the only changes required. What we actually do here, we change our kind or version in our infrastructure reference is going to be GCP cloud, because it's going to be in GCP now. This is going to invoke the manifest with GCP particular configuration. We're going to have the region move to europe-west3. We have the concept of a project, which is very particular to GCP as well. We say we want our cluster resource to be associated with CAPI project. As well, we specify a particular network, which is going to have the name default-capi.
ClusterAPI has been so efficient and so successful in such a short amount of time because it uses the building blocks principle. It does not concern itself with bringing new API endpoints or new techniques to consumers capabilities. It's actually building on top of available primitives from Kubernetes. A very important feature for ClusterAPI is the fact that it's called agnostic. It actually provides this one common interface that we can create our clusters in different cloud providers, but more importantly, we can do so with minimal changes to our manifests. It is a very important way to move forward, because it completely rethinks the way we sell clusters. We actually can have the concept of a cluster as a resource. Another thing about ClusterAPI, it's still experimental. If you have a use case for ClusterAPI, please give it a try and feed it back to the community. Now is the perfect time to customize ClusterAPI to your use case.
Emergence of Interfaces
Within six years of existence, Kubernetes transmogrified its identity multiple times. In a recursive manner, we see more and more of the out-of-tree approach being used, which means we extrapolate components from the core binaries of Kubernetes and make them develop independently. This has been so efficient because Kubernetes is not opinionated. Of course, it's going to be assertive when it comes to the networking model and the primitives it distributes, but it's not going to be opinionated on the underlying technology around Kubernetes on top of. As well, the users have the full flexibility to use the available primitives. They can actually construct their own resources by using the custom resource definition.
This had a huge impact. If you look into the perspective, this had a huge impact on the vendors and users and the community. When you're looking to the vendors, the emergence of interfaces means innovation. As a vendor, you do not have to concern yourself of how you can integrate your component with Kubernetes. The interface is going to be already there. As a vendor you can focus on the feature development, and how can you provision value to the customer with minimal latency. It's all about innovation and healthy competition.
When you're looking into the end user community, the emergence of interfaces means extensibility. It was never as easy as it is today to benchmark different tools with the same capability. As an end user, you actually have the privilege to choose the right tool for your infrastructure with minimal compromises. It's all about leveraging further your platform and your product.
When we look into the community, the emergence of interfaces, it's all about interoperability. Kubernetes embraces different solutions or multiple solutions for the same problem, but it actually focuses as well how can it interoperate between these solutions. It's all about creating this canvas of tooling, which actually is going to leverage a platform.
This has been extremely beneficial for Kubernetes because what it created is a landscape which we nowadays know as the cloud-native landscape. However, this would not be possible if the thinking of interfaces and standardization was not built in at the early stages of Kubernetes. This was done for the container network and container runtime interfaces. As well, we actually needed that extensibility and wide adoption for this tool. This is further confirmed by the introduction of service mesh and storage interfaces. More importantly, we can say that it had a huge impact on the vendors, innovation, on the end users for extensibility, on the community for interoperability.
However, the concept of interfaces transcends the world of Kubernetes. It can be applied to different domains and areas. This is because interfaces can be the central engine for development and innovation that anchors extensibility, but at the same time embraces multiple solutions for the same problem.
If you'd like to find out a bit more details about the topics I discussed today, please visit my Medium account. I'm going to publish an article with more details or links to research further
.
See more presentations with transcripts
