At the microXchg 2016 conference, held in Berlin, Germany, Rick Buskens presented “Microservice Deployment Pipelines with Spinnaker”, which discussed the collaboration between Netflix and Google on the Netflix-conceived Spinnaker continuous delivery platform. Spinnaker can be used to create build pipelines for safe and predictable deployment of microservice applications across multiple cloud providers.
Buskens, an engineering manager of developer infrastructure at Google, began the talk by stating the motivations of Google’s interest in the Netflix-led development of the Spinnaker continuous delivery tool. Netflix is widely known for utilising a microservice architecture to power its streaming video platform, and the associated number and dynamism of components within this architecture means that Netflix is frequently deploying code.
Failure of any microservice can lead to disrupted user experience, and therefore the need for safe deployments is critical. Netflix has extensive experience of ensuring safe deployments at an ‘industrial-scale’. Accordingly, Google have been working alongside the Netflix team with the goal of creating effective and widely applicable tooling for supporting continuous delivery within microservice-based projects.
We know that there are lots of people out in the community that hesitate to adopt continuous delivery because it is complicated. They also hesitate to pick a cloud provider because of fear of lock-in. So we, at Google, want you to have good tools that you can use to reliably deploy across clouds.
Overall, working with [the community] on Spinnaker, we would like to set the standard for tooling in this space.
Safe deployments are difficult due to the ‘non-hermetic’, and often manual, delivery process. This is compounded further when responsibility is often spread across multiple development and operations teams. This leads to uncertainty on what is currently deployed, error-prone deployments, and failed deployments requiring intervention by those who didn’t author the deployed components.
Buskens proposed that continuous delivery (CD) can mitigate these safety risks. Core principles for CD include staging every change and verifying production-readiness, automation, avoiding outages, and maximising deployment reliability and predictability. These principles can be realised utilising techniques such as immutable images (server images, such as VMDKs or AMIs; or containers, such as Docker or rkt) to avoid configuration drift, creating environment-specific deployment steps, providing deployment strategies like blue/green deploys, and incorporating deployment signals in testing and feedback loops.
Netflix created Spinnaker to pragmatically address several of the above issues through the creation and management of build pipelines. Spinnaker is essentially a continuous delivery platform for the cloud, and a replacement for the original Netflix OSS Asgard CD platform. Inputs to the Spinnaker platform are build artifacts (e.g. packages built by a continuous integration tool like Jenkins), and outputs are images deployed to a target environment. Pipelines form atomic workflow that provide deployment stages (e.g. testing, staging) and promotion gates (e.g. ‘push to production’). These pipelines can be chained together to produce continuous delivery workflows.
Core concepts within Spinnaker include ‘server groups’, essentially a ‘secure, load-balanced, fault-tolerant microservice’ that contains basic configuration, a load balancer, security groups, machine types, and capacity/replication strategy; a ‘cluster’, which is a logical grouping of server groups; and an ‘application’, which is a logical grouping of clusters. Deployment involves the rollout of images, managing traffic cutover, and (if necessary) easy rollback. Spinnaker also provides pluggable deployment strategies that support the techniques of blue/green, ‘highlander’ (“there can only be one!”) or canary deploys.
What is important, is that Spinnaker is built on experience of deploying industrial-scale applications to the cloud [at Netflix]. We know it works, we know it scales. It embodies key best-practices.
A Spinnaker pipeline consists of configuration and stages. Configuration includes a series of triggers, parameters and notifications, and stages are ‘context-aware operations’, such as ‘push to staging’. Pipelines are designed to be extensible and reusable, and are typically integrated with upstream CI systems (e.g. Jenkins) and chained together to create a workflow.
Spinnaker currently offers multiple cloud provider support, including Amazon Web Services, Microsoft Azure, Cloud Foundry and Google Compute Engine.
We, at Google, are looking at Spinnaker as something we can build on as we work on DevOps solutions with our customers. This is a multi-cloud solution, so what we really want to make sure is that you have the right tooling and infrastructure in place, and processes in place, so that you can deliver software quickly and reliably.
Additional providers can be added by extending Spinnaker’s ‘clouddriver’ microservice.
Spinnaker was open-sourced by Netflix on November 16th, 2015, and is in production use by Netflix. The code for Spinnaker is open source and available on GitHub, and there is a Slack channel and StackOverflow tag for associated discussions. A public roadmap for the project will soon be made available. A ‘one-click’ Google Cloud Launcher Spinnaker platform is available for people interested in experimenting rapidly with the project, and tutorials are available at spinnaker.io and cloud.google.com/solutions.
The video for Rick Buskens’ microXchg talk “Microservices Deployment Pipelines with Spinnaker” can be found on the conference YouTube channel.
