"GitOps": Weaveworks Explain Their Model for Using Developer Tooling to Implement CI/CD

Over the past year, the Weaveworks team has increasingly refined the ideas around the practice of "GitOps", their name for how they use developer tooling to drive operations and to implement continuous delivery.

GitOps is implemented by using the Git distributed version control system (DVCS) as a single source of truth for declarative infrastructure and applications. Every developer within a team can issue pull requests against a Git repository, and when merged, a "diff and sync" tool detects a difference between the intended and actual state of the system. Tooling can then be triggered to update and synchronise the infrastructure to the intended state.

Alexis Richardson, founder and CEO of Weaveworks, and Ilya Dmitrichenko, community engineer at Weaveworks, have written a series of articles explaining the concept of GitOps on their company blog. Using the GitOps practices, automated build/delivery pipelines use a "pull model" to detect and roll out changes to infrastructure when changes are made to Git. This practice does not enforce specific tools or products, and instead only requires certain functionality (such as the "diff and sync") to be provided by the chosen tools.

The Weaveworks team claims that GitOps allows development teams to increase their velocity and improve system reliability. They have discussed how they have implemented GitOps internally for the delivery of their products, and also how they offer the building blocks for this approach via their Weave Cloud SaaS product. Weave Cloud offers "deployment, monitoring and management for containers and microservices", and supports Git-cluster synchronization.

The guidelines for Weaveworks' current implementation of GitOps, which uses containers and Kubernetes for deployment, includes:

1. Everything within the software system that can be described as code must be stored in Git: By using Git as the source of truth, it is possible to observe a cluster and compare it with the desired state. The goal is to describe and version control all aspects: policies, code, configuration, and even monitored events and dashboard definitions.
2. The 'kubectl' Kubernetes CLI tool should not be used directly: As a general rule, it is not a good practice to deploy directly to the cluster using kubectl. The Weaveworks team claims that many people let their CI tool drive deployment, but by doing this, they are not practicing good separation of concerns, and are "potentially giving a notoriously hackable thing access to production".
3. Use a Kubernetes controller that follows an "operator pattern": By extending the functionality offered by Kubernetes, using a custom controller that follows the operator pattern, the cluster can be configured to always stay in sync with the Git-based 'source of truth'. The Weaveworks team uses "diff" and "sync" tools such as the open source kubediff, as well as internal tools like "terradiff" and "ansiblediff" (for Terraform and Ansible, respectively), that compare the intended state with actual state.

The ability to compare and manage the current state of both infrastructure and applications so that a team can test, deploy, and rollback with a complete audit trail encapsulated within the associated Git logs is what "encompasses the GitOps philosophy and its best practices". Modern platform components make this approach much more viable in comparison with older technologies because, for example, Kubernetes is managed almost entirely through declarative config, and containers can be (relatively) easily built to be immutable.

As an example, using the Weave Cloud SaaS platform, a typical developer workflow for creating or updating a new feature within a software application would be as follows:

1. An engineer works on a new feature request in a new Git branch and creates a pull request, for example on GitHub, when they are ready to have their code reviewed.
2. The code is reviewed, and comments are made or the change is approved by a colleague. After any necessary revisions are made, the approved pull request is merged to the trunk or mainline branch.
3. The Git merge triggers the continuous integration (CI) and build pipeline, and runs a series of tests. Upon successful completion of these tests, a new container image is built and uploaded to an image registry.
4. The Weave Cloud 'Deployment Automator' watches the image registry, notices the new image, and pulls this from the registry and updates the associated YAML file in the repository that contains configuration.
5. The Weave Cloud 'Deployment Synchronizer' (which is installed to the cluster), detects that the cluster state is "out of date", and it pulls the changed manifests from the config repo and deploys the new feature to production.

The Weaveworks team states that GitOps is a "release-oriented model" for implementing and managing both operations and features. With the addition of good observability practices and tooling, which complete the feedback loop in hypothesis-driven development, how quickly a team can deliver new features to their customers depends in part on how fast they can go around the stages in a proposed OODA loop inspired software delivery life cycle:

For readers who are interested to learn more about GitOps, there are a series of blogs available on the Weavework website. The first article explains the "operations by pull request" model, and provides the motivations and high-level overview of the concept. The second article discusses the core stages of the GitOps delivery pipeline. Part three of the series examines the role of observability within this practice. Reaction from the software delivery community has generally been encouraging, with industry luminaries like Kelsey Hightower talking positively about the approach.

There is also a standalone article that explores GitOps versus "CIOps", and argues that using CI tooling may not be the optimal approach to orchestrate the deployment aspects of continuously delivering software. Not everyone is happy with the chosen terminology in this piece, for example, Matthew Skelton, head of consulting at Conflux Digital, suggested that the term "CIOps" could cause some engineers to draw the incorrect conclusion that GitOps is somehow an alternative to implementing good CI.

Additional information about GitOps can be found on the Weaveworks blog.