At DockerCon EU 2015, VMware made available their Photon Controller as an open source product via the organisation’s GitHub account. Photon Controller is a component within VMware’s Photon platform, which is designed as an infrastructure stack optimised for ‘container and cloud’ workloads. The virtualisation platform offers an 'API-first' user experience, a scalable control plane, and native container support via frameworks such as Kubernetes, Docker Swarm or Mesos.
The VMware blog states that the Photon platform is a new infrastructure stack optimized for ‘cloud-native applications’, and is based upon the VMware ESXi hypervisor, but “favors scale and speed over the rich management features offered by vSphere”. The platform consists of Photon Machine, a lightweight ‘microvisor’ based on ESX with Photon OS built-in; and Photon Controller, a distributed, API-driven, multi-tenant control plane that includes the Project Lightwave container security project; and is designed for ‘extremely high scale and churn’.
The Photon platform was released at VMworld in August, with the promise of “open sourcing the Photon Controller so we could engage directly with developers, customers and partners”. The Photon Controller website states that the project delivers:
- An API-first model: A user-experience focused on the automation of infrastructure consumption and operations using RESTful APIs, SDKs and CLI tooling
- A fast and scalable control plane: A new infrastructure control plane optimised for scalability and speed, allowing the creation of 1000s of new VM-isolated workloads per minute, and supporting 100,000s of total simultaneous workloads
- Native container support: The Photon Controller is built for “large environments to run workloads designed for cloud-native (distributed) apps”, and development teams can deploy container-based applications using a variety of container orchestration frameworks, including Kubernetes, Docker Swarm, Pivotal CF / Lattice, and Mesos
The Photon Controller architecture consists of a number of loosely coupled components, or services, that are managed via ‘distributed coordination’. Endpoints are registered in Apache Zookeeper, and services may operate as ‘scaled-out’ (active/active), utilise standby servers (active/passive), or have clearly partitioned work (e.g. scheduler). The technical architecture of the Photon Controller is as shown in the figure below:
Figure 1. Technical architecture of Photon Controller
The Photon Controller external APIs are provided as a REST (JSON over HTTP) interface implemented using the Java-based Dropwizard framework. API documentation is exposed via Swagger. The external REST API is designed to be exposed by horizontally-scaled API servers that share a persistent database (the ‘CloudStore’) that acts as a single source of truth for the state of containers, clusters, VMs, disks, networks and physical hosts etc. A load balancer (like haproxy) would typically be placed in front of the API servers.
The majority of the Photon Controller management plane is written in Java, with several services implemented on top of the VMware’s Xenon framework, a ‘decentralised systems development framework’, that is currently available as a technology preview. The ESX agent is implemented in Python, and the Photon Controller CLI is implemented in Ruby, as are the integration tests for the complete product.
Each physical host managed by Photon Controller has an agent that runs on it and provides an RPC interface implemented using Apache Thrift, which all other components use to communicate with the host. There are a number of special components that oversee the health and housekeeping of other components. For example, the ‘Chairman’ is responsible for the schedulers’ health and tree topology.
The VMware website claims that Photon Controller also implements a novel distributed scheduler:
It is a hierarchy of scheduler service nodes with each node in the tree only having awareness of its direct children. Schedulers bubble essential stats on their load/utilization up to their parents, and parents route requests down through the scheduler tree to resolve placement requirements. This way the schedulers avoid heavy load on a single metrics/configurations database.
More information about the Photon Platform and Photon Controller can be found in the project’s website, and the open source code can be found in the VMware GitHub repository. Developers can download and run a small standalone deployment of Photon Controller for test purposes using the project’s “Devbox”. Questions about the product can be asked on the Photon Controller Google Group, or via Stack Overflow using the “photon-controller” tag.