Robin Weston, lead engineer at BCG Digital Ventures, recently presented an experience report at Continuous Lifecycle London where an external enablement team was able to introduce continuous delivery (CD) practices in an organization with high resistance to change and siloed teams (leading to a cycle time of three months). Rather than just bringing in new technology and tools, the team focused on sharing good practices and educating teams. Practices ranged from continuous integration to following the test pyramid or reducing cycle time by measuring activities and identifying waste.
Weston admitted that dedicated CD teams are an anti-pattern since they can lead to lack of (delivery) ownership by the product teams. However, his team took the challenge of accepting the engagement as a first step to establish a new culture, and not become yet another silo of knowledge. The team hoped to engage product teams with some modern development practices and enable them to take initiative from there and adopt a continuous improvement approach.
Weston's team started by running value stream mapping exercises with the engineers doing the day to day work, exposing bottlenecks that were not visible before. For instance, pull requests that could only be approved by employees in a different time zone, leading to long delays between code commit and code integration. By simply moving approval to co-located engineers, this large bottleneck was removed, and so forth, in an effort to streamline the path to production of most changes.
One of the multiple challenges, according to Weston, was to fend off the requests for this team to do the actual build, test and delivery work of the products. Sticking to the team's mission - enabling product teams to deliver features faster and with higher quality - making the backlog public, and collecting data to show the impact of the team's work on critical metrics was key to avoid getting dragged down in the everyday grunt work. In fact, clearly and consistently communicating issues and progress (via regular showcases, recorded demos, mob programming or wiki updates) and training product teams in the new practices (such as trunk based development) took most of the team's time.
Dashboard showing Continuous Delivery metrics that were the target of the enablement team
After the low hanging fruit was picked - such as the remote pull request approvals - the team defined a clear line of action focusing on establishing continuous integration practices (and principles) first, then moving from nearly only UI-based application tests to a test pyramid approach, and, finally, maturing and stabilizing the pipeline activities. Jumping to latest technology was definitely not a priority.
Basics of continuous integration were not in place, as teams did not proactively engage in fixing broken builds, for example. There was an overall lack of ownership of the entire delivery process, according to Weston. Running a CD readiness survey showed that most teams were behind on build and environments management, testing, data management, cycle time, and even, in some cases, version control was not consistently adopted. However, the results helped product teams understand the need for changing the way they developed and delivered their systems.
In terms of pipeline changes, pipeline as code was adopted, with each team responsible for maintaining their own pipeline definitions in the same repository as the application being delivered via that pipeline.
At the time Weston left the engagement, some teams were already experimenting with microservices and contract testing to decouple releases and increase delivery frequency. However, other teams were still working in release branches and coupled release schedules.