Cindy Sridharan, an engineer at imgix, has written a comprehensive article on the purpose of job schedulers like Kubernetes and HashiCorp's Nomad, and discussed the application packaging, deployment and lifecycle challenges the imgix team had which led them to consider implementing a scheduler within their technology stack. At imgix the Nomad scheduler was ultimately chosen, and a well-balanced analysis and overview of the implementation (and remaining challenges) is presented in the full article, with the outline including:
- The purpose of schedulers the way they were originally envisaged and developed at Google
- How well (or not) they translate to solve the problems of the rest of us
- Why they come in handy even when not running "at scale"
- The challenges of retrofitting schedulers into existing infrastructures
- Running hybrid deployment artifacts with schedulers
- Why at imgix we chose Nomad over Kubernetes
- The problems yet to be solved
- The new problems these new tools introduce
- What the future holds for us
Sridharan states that the applications we currently develop have become vastly more complex than those we were building a decade ago. Even when the core business logic is simple, the need for features such as high reliability and availability, customer satisfaction, rapid innovation, continuous deployment, quick feedback loops and constant iteration has made the rapidly increased the importance of standard and reliable tooling. Many organisations are looking towards the "unicorn organisations" like Google, however, this has its limitations:
"Google Infrastructure For Everyone Else" only makes sense if the technology can be adapted to solve real, immediate problems faced by organizations.
Cluster schedulers were initially popularized by the white paper (PDF) on Borg from Google. Google famously has been running everything in "containers" orchestrated by Borg for over a decade now. Containerization outside large organizations was made possible by the runaway success of Docker, which in turn led to the birth of Kubernetes.
Schedulers might initially seem like a scary thing, way above the pay grade of most engineering organizations. In reality, however, schedulers can be a game changer and are an enormous step up from the traditional way of managing software life cycle. The flexibility they afford and immediate benefits they provide cannot be overstated enough.
Sridharan presents three challenges that led to the imgix team exploring scheduler technology: packaging - the need for a POSIX-like standard for packaging applications written is a variety of languages (and although Docker containers are close to this standard, they do have limitations); deployment - there is also no standard language-agnostic approach to deploying applications that may be statically linked binaries or a collection of more complicated artefacts; and lifecycle - we are all building distributed systems, and so must consider partial failure, degraded application functionality, and service level objectives (SLOs) and agreements (SLAs).
Being mindful of the opportunity cost of implementing a scheduler within the imgix stack, a discussion of why Nomad was ultimately chosen is presented in the full article. The current tight integration of Kubernetes and Docker (and the associated requirement to change the packaging format of existing imgix applications) in combination with perceived networking challenges within Kubernetes were presented as deciding factors for not choosing this technology. Nomad allows the deployment of a range of artifacts, including single statically linked binaries, and has good integration with Consul for service discovery (and the imgix tech stack was already dependent on Consul).
A good rule of thumb when it comes to adopting new tooling — especially Operations tooling — is to pick one that will integrate seamlessly into an existing infrastructure and require one to make the least number of changes to what’s already working.
Sridharan states that the immediate wins of beginning a rollout of Nomad included: minimal changes required to the deployment packaging and the existing Consul service discovery mechanism; the ability for developers to specify operational semantics of application within the job file; "democratisation of ops", in which the similarity of job files across a range of applications (e.g. services created from different language stacks, and long-running services and batch jobs) allowed all engineers to rapidly understand deployments; and the simplicity of operation - for example, Nomad is deployed to each node as a single binary. Current challenges with Nomad include the lack of access control lists (ACLs), which can be overcome by using a ingress gateway or reverse proxy like HAProxy, and the absence of the ability to specify quotas, priorities and oversubscription to cluster resources.
Cindy Sridharan's full article "Cluster Schedulers" can be found on Medium, and a brief discussion of the article can be found on Twitter.