Pushing the boundaries of continuous delivery, you can fundamentally change the way people collaborate while building software. Christian Uhl presented at DevOpsCon Munich 2019 how deploying every feature branch using GitLab and Kubernetes helps them to get fast feedback from product owners and stakeholders.
Uhl spoke about how they don’t bundle releases anymore; every branch gets deployed into its own environment and every feature goes straight to production:
The benefit of deploying the smallest possible unit (instead of a bundle of things) is to minimize risk. I think the Book Accelerate by Nicole Forsgren explains it so much better than I could do, and we certainly experience a low level of defects as the book promises.
The deployment of branches was a decision they made after they found out that a lot of problems occurred not because they built a broken thing (think bugs), but the wrong thing (think misunderstood requirements). Uhl mentioned that when they simultaneously show the user interface of a deployed branch to the product owner or a stakeholder while they are developing, they instantly have a more meaningful discussion.
Uhl explained how their approach to continuous delivery impacts the way that they do product reviews:
Usually, a product review is scary, because it is about "will my feature pass the strict eyes of the product owner?" This leads to tension and compromises. Now, since everybody sits together as it happens, it has become a collaboration. We find many tweaks and improvements to our ideas in the dev process that we wouldn’t have found otherwise. I am very proud about many of the features of matmatch.com and how quickly we can bring ideas to life now.
At Matmatch they leverage most of the tooling GitLab and Kubernetes provide. Every branch gets built, dockerized, and then deployed with a generated name. In addition, some routing gets created. With a declarative deployment configuration, it is a matter of parameters to deploy a branch to feature1.domain.com instead of domain.com, Uhl explained. After completing both code review and stakeholder approval, they merge the branch to master and clean up the deployment.
InfoQ interviewed Christian Uhl, head of engineering at Matmatch, after his talk at DevOpsCon Munich 2019.
InfoQ: What made you decide to improve your continuous delivery pipeline? What was the need?
Christian Uhl: We are a startup, and as such we have a limited runway (or time to prove that we can be successful) - and in the beginning of such an endeavor, you don’t really know what your users want. So almost everything you build is wrong - and you need to change things really quickly to find that "pot of gold".
As an example, we had an initial feature where engineers were able to create a collection of materials as a "Project" and would be able to share them - but we quickly found out that usage was very low. Afterwards, we built a feature where people could save individual materials - now used very frequently.
Not to have any distractions from creating user value, we decided that deploying software has to be a fully automated and boring thing. We also need to find out what works and what does not super fast. We can’t wait for two additional weeks after finishing a feature to start recording user behavior.
InfoQ: How is testing done within the pipeline?
Uhl: We follow the classic testing pyramid of unit, integration and end-to-end tests. We took particular care to keep the amount of end-to-end tests to a minimum, but still testing the fundamental core of the platform, basically the stuff that makes money. These tests are executed during each commit, and nothing can be merged before every stage is green.
Since there is no more staging environment before a feature goes to production, the developers are incentivised to think carefully about the tests they write - there is no safety net or QA team to save you anymore. This corresponds to the "you build it, you run it" idea. The automated testing harness we build gives us enough confidence that nothing major is broken. With the currently available cloud offerings, we can provision quite a powerful machine to perform each test stage, and discard it afterwards, which keeps our test runtime on an acceptable level.
We invested a lot of effort into the observability of our system, so that we can find out quickly when errors occur and fix them forward. We track all errors happening in our system and have alerting for this so that we know when a new feature causes problems. In addition, we check before a release of a feature that we added some tracking to find out if a feature is accepted or ignored by users. We do have the tooling to roll back though, but honestly, we never use it. We keep it for our peace of mind.
InfoQ: What challenges did you have building your continuous delivery pipeline and how have you dealt with them?
Uhl: First of all, it’s a serious amount of work to get to the point where we are at. We’ve had to automate the deployment itself, which was already a challenge, build enough tests to be confident and finally get all the configuration right. I was able to push for it because I truly believed in the business value and agility we would be getting - but it was hard justifying the time in the early stage of a startup, when there are a million other important things to do. It was not one big challenge to build this platform, just many little things we had to figure out. We also blew up our Kubernetes cluster more than once because we didn’t clean up everything properly after merging.
But besides the technical issues, there was a lot of fear in the beginning; deploying everything straight to production scared people. I spent a lot of time explaining the benefits we would be getting. We also started small and just deployed one small, non-critical service all the time before including the core systems. Starting small helps a lot.
Since we took on that journey in 2017, the tooling has improved a lot. I figure it’s easier to start now than ever before - but we had to play a lot of catch-up.