As a part of their journey towards building a reliable continuous integration (CI) system, the developer experience (devX) team at Gojek recently described how they tackled challenges around scaling the development and integration of Android and iOS mobile apps. With 140+ engineers and ~1100 pipelines triggering per day, Dinesh C, engineering manager for the devX team, emphasized the importance of segregating pipelines for providing faster feedback in his blog.
A reliable CI system enables development teams to ship the code faster, so that product managers can obtain early feedback on their features. Dinesh described that the pipelines impact the organization, as hundreds of engineers work towards a common goal. This goes in line with the current trend of adapting CI/CD for mobile apps where, according to the Appinventiv team, the benefits offered include automated builds, improved communication, automated shipping, and zero server maintenance.
To run the pipelines in parallel or in quick intervals, machines ("runners") were required. In order to overcome the challenge with the limited number of runners, the devX team used GitLab Runner to run the jobs and send statistics back to GitLab CI/CD. Using the GitLab runner configuration file, the team defined the number of jobs that could run in parallel with each runner, which reduced the completion time for the pipelines.
For creating effective pipelines, Dinesh listed waiting time and completion time as the important variables. When a developer has around 10-20 commits and average 300 minutes of pipelines running every day, waiting for pipelines to start and complete leads to a bad experience. The pipelines have different functional and infrastructure requirements as well. Listing the examples of pipelines categories, Dinesh highlighted that infrastructure planning requires categorising pipelines.
Source: https://blog.gojekengineering.com/running-ci-for-mobile-engineers-at-scale-de951b556cf4
After dividing each pipeline into different stages, each stage is divided into the smallest unit - a "job". Dinesh further elaborated that the variables - runtime environment and memory requirement determine the infrastructure needs of a job. Based on the type of job, pipelines were categorized as depicted in the diagram below:
Source: https://blog.gojekengineering.com/running-ci-for-mobile-engineers-at-scale-de951b556cf4
Coming to the aspect of feedback, an ELK stack was utilized for monitoring and measuring feedback. The metrics collected over the pipeline were passed via a Curl HTTP to Elastic data store. Dinesh mentioned that infrastructure and pipeline statistics were collected to help the DevOps team, and code metrics were collected to help the developers. Slack bots were used to notify merge request authors about the status of pipelines.
In related news, Max Lynch, co-founder/CEO of Ionic framework, wrote about a managed mobile build environment managed by Ionic. It enables teams to have their CI/CD tool like GitLab trigger builds and push code to Appflow, thereby managing mobile development infrastructure and publishing or updating on the app store. The Appflow team also keeps the mobile build tools and operating systems up-to-date and patched for security issues.
In regards to the future at Gojek, the devX team aims to package CI systems as a plug-and-play software, most likely making it open-source.