Key Takeaways
- Many organisations are aiming to assemble their own cloud platform, often consisting of a combination of on-premises infrastructure and cloud vendors.
- Although Kubernetes does not provide a full platform-as-a-service (PaaS)-like experience out of the box, the combination of a well-defined API, clear abstractions, and comprehensive extension points make this a perfect foundational component on which to build upon.
- Crossplane is an open source cloud control plane that enables engineers to manage any infrastructure or cloud services directly from Kubernetes. This control plane provides a single entry point for user interaction where policy can be enforced, guardrails applied, and auditing conducted.
- Infrastructure can be configured using K8s custom resource definitions (CRDs) and YAML. It can also be managed via well established tools like kubectl or via the Kubernetes API itself, and take advantage of workflow best practices, such as GitOps.
- The Open Application Model (OAM) specification describes a model where core software delivery personas are defined and prescribed clear areas of responsibility: developers, application operators, and infrastructure operators. Crossplane is the Kubernetes implementation of the specification.
InfoQ recently sat down with Bassam Tabbara, founder and CEO of Upbound, and discussed building application platforms that span multiple cloud vendors and on-premise infrastructure.
The conversation began by exploring the premise that every organisation delivering software deploys applications onto a platform, whether they intentionally curate this platform or not. Currently, Kubernetes is being used as the foundation for many "cloud native" platforms. Although Kubernetes does not provide a full platform-as-a-service (PaaS)-like experience out of the box, the combination of a well-defined API, clear abstractions, and comprehensive extension points make this a perfect foundational component on which to build upon.
Tabbara also discussed Crossplane, an open source project that enables engineers to manage any infrastructure or cloud services directly from Kubernetes. This "cross cloud control plane" has been built upon the Kubernetes’ declarative configuration primitives, and it allows engineers defining infrastructure to leverage the existing K8s toolchain. The conversation also covered the Open Application Model (OAM) and explored how Crossplane has become the Kubernetes implementation of this team-centric standard for building cloud native applications.
Every engineering team needs a platform
Many organisations are aiming to assemble their own cloud platform, often consisting of a combination of on-premises infrastructure and cloud vendors. Leaders within these organisations recognise that minimizing deployment friction and decreasing the lead time for the delivery of applications, while simultaneously providing safety and security, can provide a competitive advantage. These teams also acknowledge that any successful business typically has existing "heritage" applications and infrastructure that needs to be included within any platform. Many also want to support multiple public cloud vendors, with the goals of avoiding lock-in, arbitraging costs, or implementing disaster recovery strategies.
Platform teams within organisations typically want to enable self-service usage for application developers and operators. But they also want appropriate security, compliance, and regulatory requirements baked into the platform. All large-scale public cloud vendors such as Amazon Web Service (AWS), Microsoft Azure, and Google Cloud Platform (GCP) offer their services through a control plane. This control plane consists of user interfaces (UIs), command line interfaces (CLIs), and application programming interfaces (APIs) that platform teams and operators interact with to configure and deploy the underlying "data plane" of infrastructure services. Although the implementation of a cloud control plane is typically globally distributed, it appears centralised to the end users. This control plane provides a single entry point for user interaction where policy can be enforced, guardrails applied, and auditing conducted.
Application developers typically want a platform-as-a-service (PaaS)-like experience for defining and deploying applications, as pioneered by the likes of Zimki, Heroku, and Cloud Foundry. Deploying new applications via a simple "git push heroku master" is a powerful and frictionless approach. Application operators and site reliability engineering (SRE) teams want to easily compose, run, and maintain applications and their respective configurations.
Tabbara cautioned that these requirements lead to an organisation buying a PaaS, which unless chosen appropriately, can be costly to maintain:
"Modern commercial PaaSs often meet the requirements of 80% of an organisation’s use cases. However, this means that the infrastructure teams still have to create additional platform resources to meet the other 20% of requirements"
Building a PaaS(-like experience)
Building a PaaS is not easy. Doing so takes time and skill, and it is challenging to define and implement technical abstractions that meet the requirements of all of the personas involved. Google famously has thousands of highly-trained engineers working on internal platforms, Netflix has a large team of specialists focused on the creation and maintenance of their internal PaaS, and even smaller organisations like Shopify have a dedicated platform team. Technical abstractions range from close to the "lowest common denominator", like that taken by Libcloud and OpenStack, all the way through to providing a common workflow but full cloud-specific configuration, like HashiCorp’s Terraform or Pulumi. Traditional PaaS abstractions are also common within the domain of cloud, but are typically vendor specific e.g. GCP App Engine, AWS Elastic Beanstalk, or Azure Service Fabric.
Many organisations are choosing to build their platform using Kubernetes as the foundation. However, as Tabbara stated on Twitter, this can require a large upfront investment, and combined with the 80% use case challenge, this can lead to the "PaaS dilemma":
"The PaaS dilemma - your PaaS does 80% of what I want, my PaaS takes 80% of my time to maintain #kubernetes"
Tabbara stated that the open source Crossplane project aims to be a universal multi-cloud control plane for building a bespoke PaaS-like experience.
Crossplane is the fusing of "cross"-cloud control "plane". We wanted to use a noun that refers to the entity responsible for connecting different cloud providers and acts as control plane across them. Cross implies "cross-cloud" and "plane" brings in "control plane".
By building on the widely accepted Kubernetes-style primitives for configuration, and providing both ready-made infrastructure components and a registry for sharing additional resources, this reduces the burden on the infrastructure and application operators. Also, by providing a well-defined API that encapsulates the key infrastructure abstractions, this allows a separation of concerns between platform operators (those working "below the API line") and application developers and operators (those working "above the API line").
"Developers can define workloads without worrying about implementation details, environment constraints, or policies. Administrators can define environment specifics and policies. This enables a higher degree of reusability and reduces complexity."
Crossplane: Controlling infrastructure via Kubernetes
Crossplane is implemented as a Kubernetes add-on and extends any cluster with the ability to provision and manage cloud infrastructure, services, and applications. Crossplane uses Kubernetes-styled declarative and API-driven configuration and management to control any piece of infrastructure, on-premises or in the cloud. Through this approach, infrastructure can be configured using custom resource definitions (CRDs) and YAML. It can also be managed via well established tools like kubectl or via the Kubernetes API itself. The use of Kubernetes also allows the definition of security controls, via RBAC, or policies, using Open Policy Agent (OPA) implemented via Gatekeeper.
As part of the Crossplane installation a Kubernetes resource controller is configured to be responsible for the entire lifecycle of a resource: provisioning, health checking, scaling, failover, and actively responding to external changes that deviate from the desired configuration. Crossplane integrates with continuous delivery (CD) pipelines so that application infrastructure configuration is stored in a single control cluster. Teams can create, track, and approve changes using cloud native CD best practices such as GitOps. Crossplane enables application and infrastructure configuration to co-exist on the same Kubernetes cluster, reducing the complexity of toolchains and deployment pipelines.
The clear abstractions, use of personas, and the "above and below the line" approach draws heavily on the work undertaken within the Open Application Model.
OAM: Team-centric standard for building cloud native apps
Initially created by Microsoft, Alibaba, and Upbound, the Open Application Model (OAM) specification describes a model where developers are responsible for defining application components, application operators are responsible for creating instances of those components and assigning them application configurations, and infrastructure operators are responsible for declaring, installing, and maintaining the underlying services that are available on the platform. Crossplane is the Kubernetes implementation of the specification.
With OAM, platform builders can provide reusable modules in the format of Components, Traits, and Scopes. This allows platforms to do things like package them in predefined application profiles. Users choose how to run their applications by selecting profiles, for example, microservice applications with high service level objective (SLO) requirements, stateful apps with persistent volumes, or event-driven functions with horizontally autoscaling.
The OAM specification introduction document presents a story that explores a typical application delivery lifecycle.
- The developer creates a web application;
- The application operator deploys instances of that application, and configures it with operational traits, such as autoscaling;
- The infrastructure operator decides which underlying technology is used to handle the deployment and operations.
To deliver an application, each individual component of a program is described as a Component YAML by an application developer. This file encapsulates a workload and the information needed to run it.
To run and operate an application, the application operator sets parameter values for the developers' components and applies operational characteristics, such as replica size, autoscaling policy, ingress points, and traffic routing rules in an ApplicationConfiguration YAML. In OAM, these operational characteristics are called Traits. Writing and deploying an ApplicationConfiguration is equivalent to deploying an application. The underlying platform will create live instances of defined workloads and attach operational traits to workloads according to the ApplicationConfiguration spec.
Infrastructure operators are responsible for declaring, installing, and maintaining the underlying services that are available on the platform. For example, an infrastructure operator might choose a specific load balancer when exposing a service, or a custom database configuration that ensures data is encrypted and replicated globally.
Exploring a typical Crossplane workflow
To make the discussion more concrete, let’s explore a typical Crossplane workflow, from installation of the project to usage.
First, install Crossplane and create a Kubernetes cluster. Next, install a provider and configure your credentials. Infrastructure primitives can be provisioned from any provider e.g. (GCP, AWS, Azure, Alibaba, and (custom-created) on-premise.
A platform operator defines, composes, and publishes your own infrastructure resources with declarative YAML, resulting in your own infrastructure CRDs being added to the Kubernetes API for applications to use.
An application developer publishes application components to communicate any fundamental, suggested, or optional properties of our services and their infrastructure requirements.
An application operator ties together the infrastructure components and application components, specificies configuration, and runs the application.
Conclusion
Kubernetes is being used as the foundation for many "cloud native" platforms, and therefore investing in both models of how the team interacts with this platform and also how the underlying components are assembled is vitally important and a potential competitive advantage for organisations. As stated by Dr Nicole Forsgren et al in Accelerate, minimising lead time (from idea to value) and increasing deployment frequency are correlated with high performing organisations. The platform plays a critical role here.
Crossplane is a constantly evolving project, and as the community expands more and more feedback is being sought. Engineering teams can visit the Crossplane website to get started with the open source projects, and feedback can be shared in the Crossplane Slack.
About the Author
Daniel Bryant works as a Product Architect at Datawire, and is the News Manager at InfoQ, and Chair for QCon London. His current technical expertise focuses on ‘DevOps’ tooling, cloud/container platforms and microservice implementations. Daniel is a leader within the London Java Community (LJC), contributes to several open source projects, writes for well-known technical websites such as InfoQ, O'Reilly, and DZone, and regularly presents at international conferences such as QCon, JavaOne, and Devoxx.
