The release process for DoorDash mobile apps is based on clear-cut responsibilities shared across teams, effective communication, testing, and strict rules about handling regressions and hotfixes, explains DoorDash engineer Manolo Sañudo. While not all organizations work at DoorDash scale, many aspects of their approach can prove useful to smaller organizations, too.
DoorDash follows a rather straightforward weekly release cycle, where a release branch is created for each new release candidate to go through a week-long process of testing and fixing that eventually leads to the official release.
Each new release candidate is assigned a release manager to oversee the process and make sure everything goes smoothly. The pool of release managers is sufficiently large so no one is burdened by the workload but not so large as to hinder taking decisions consistently across releases or jeopardizing the release process evolution and improvement.
Each release candidate gets its own Slack channel to centralize status updates and conversations in one place and to prevent hotfixes for bugs in production from generating noise.
For testing, since it is impossible to do full regression testing in one week, says Sañudo, so-called component owners are responsible for testing all the components individually and tracking testing status using the mobile release management platform Runway.
Each component owner has specific tests that they must run before approving the component. And every single component must be approved before the release can be submitted for review.
From time to time, says Sañudo, regressions are discovered during this testing phase. In this case, the release manager works with the affected team to develop a fix, which is pushed to the main development branch, but the fix is only merged into the release candidate branch if the regression affects user experience. Neither bugs having no effect on the user, nor new features are allowed in this phase, and each cherry-picked fix must be justified by a team and approved by the release manager.
If a bug is detected later in the process, after the app has been submitted for review, even stricter rules apply, since implementing a hotfix could imply a delay in the release becoming available.
Although the update is not yet public, it may be waiting for review or even approved already; to implement a fix, we would have to reject the build and resubmit the app. Because this could delay the release, we evaluate whether a fix is merited and how to approach it on a case-by-case basis.
After Apple's approval, the new release is rolled out to 1% of the user base to ensure no major issues surface and after a couple of days the rollout is extended to the whole user base. In this phase, teams use key metrics to understand what may be going wrong with their components in the new release. Likewise, release managers track higher-level metrics like crash rate and any trending issues using Sentry.