The Release Process Used by Mozilla for Firefox

This article presents the release process used by Mozilla for their browser.

From 2004 Mozilla initially released few versions of Firefox, reaching version 4.0 in July 2010. But from 2011, following Google’s example, Mozilla changed the release cycle, today’s version being #30. During all this time, the Release Engineering team has improved the process of generating a new version of the browser. A group of four engineers -Chris AtLee, Lukas Blakk, John O'Duinn, Armen Zambrano Gasparian- have written an article on Dr. Dobb's Journal detailing the release process, which we are presenting here in a condensed form.

Considering possible security breaches related to their browser, Mozilla has designed a release process that would be able to quickly generate a new “chemspill” (chemical spill) version that is immediately pushed to users to fix known vulnerabilities. This process is as automated as possible to lessen “human involvement” and improve “robustness and turnaround time.” The process is used both for chemspill and regular releases. After each release, Mozilla performs a post-mortem analysis to see if there were any issues to improve upon. Problems found are fixed immediately before another release takes place.

Releases are triggered by a Release Coordinator, a person who needs to “attend triage meetings, understand the background context on all the work being landed, referee bug severity disputes fairly, approve landing of late-breaking changes, and make tough back-out decisions.” The coordinator is in touch with all teams involved – developers, release engineering, QA, PR, etc. –, having the authority to stop the build if necessary.

After having problems issuing the command to start a new build over IRC or phone, Mozilla has decided to issue the command over email sent to a list with all persons involved in the process, the subject of the email containing the text “go to build + product_name  + version”. The email contains detailed information on the source code that is to be built and released, including the time and the time zone for a time-based code repository.

The entire release process consists of the following steps:

1. Go-to-Build email sent by Release Coordinator.
2. Tagging starts – A tag such as FIREFOX_30_0_RELEASE is applied to the source code from ~85 repositories –product, localization strings, release automation code and utilities - used to create a specific version of the browser. The process lasts about 20 minutes due to the large number of repositories, time that Mozilla wants to cut down to 5 minutes by running separate tagging processes in parallel, one for each repository.
3. Builders start – A number of coordinated builds are launched, one per release platform, plus a source bundle that includes all the source code just tagged, the result being uploaded to ftp.mozilla.org. This step includes localization “repack”: the en-US locale build is unpacked and all the en-US strings are replaced with the corresponding locale then packed back.
4. Signing – the binaries are signed. For Windows, all the EXE and DLL files are signed, including the installer itself. A number of MD5 and SHA1 checksums are generated for mirrors to validate their downloads. Signing is performed on a dedicated server firewalled from outside communications. The passwords, key-phrases and key stores are transferred between release engineers using only secure channels or in person.
5. Testing – For a chemspill release, QA performs a manual test of the security bug that required a new release. If problems are found, they have to be fixed and the entire build process is restarted.
6. Generating updates – Update packages are created, ready to be downloaded by the browser updater. A large number of update packages need to be created, one for each platform, language and previous release.
7. QA testing - Community members, contractors and Mozilla employees perform manual testing as soon as signed builds are ready, but they also test the update packages. An automated functional testing process is also triggered at this time. If everything is OK, QA signs off the builds and the updates.
8. Mirroring – The updates are pushed to various mirrors around the world. When the process is done, the Release Coordinator sends a “Go Live” email announcing that the new version is now available, and the release engineers update the website and FTP servers to point to the new release.

Mozilla has learned a few lessons while developing this release process:

• It is important to consider both technical and non-technical stakeholders that might have a say in the success of a release.
• Make everybody aware of the release process and where time is spent. Initially, people inside Mozilla considered that a new version is the job of release engineers only, but it turned out that many others were actually involved in the process and were responsible for it.
• Most problems appearing during a release cycle were related to “miscommunication between teams; lack of clear leadership; and the resulting stress, fatigue and anxiety during chemspill releases.” This was solved by having “clear handoffs to eliminate these human miscommunications.”
• Team stability was another problem: too many engineers left the release team after a while, and other had to replace them just to have them leave later. This lead to “lack of accurate up-to-date documentation meant that most of the technical understanding of the release process was documented by folklore and oral histories, which we lost whenever a person left.” This was changed by taking the whole release issue seriously and making engineers feel that Mozilla had “a plan to make things better, and that people had some control over their own destiny.”
• The whole release process was improved in small steps, making each new release better than the previous one. As the automated process was improved, the team had more time available to improve it even further.

According to the authors, the release process has been improved to the point that Mozilla was able to recently generate 8 chemspill releases in 2 days in order to address a security vulnerability uncovered in a third party library used by Firefox.