At a recent Google Tech Talk in New York, Lindsay Pasricha, software test engineer at Google for the last eight years, provided a peek into the test and release processes for Google Chrome for iOS, exploring product development strategy, automated testing frameworks, and manual testing processes. Here a summary of the most important takeaways.
Development environment
Xcode is the major tool in use at Google for iOS development, together with other internal solutions. Some of those solutions are not proprietary, Pasricha states, like gyp to manage dependencies and git for version control. „And we use a lot of automated build solutions to make this go as fast as possible,” such as Pulse and Waterfall, which will be covered later on.
iOS Development challenges
One of the most important challenges to Chrome development for iOS comes from Google being such an Android-oriented company. This makes it hard to find, e.g., Google engineers owning an iPhone that will join an internal testing program; but it also makes it hard to reach a common viewpoint with non-iOS focused groups, which are not aware of specific limitations of development for iOS, such as the impossibility of having beta versions on the App Store.
Apart from that, a big challenge is related to restrictions on allowed APIs and what services can be provided in an app enforced through the App Store review process,. This is a common problem to all iOS developers, but in case of Chrome it is particularly harsh since Apple restricts any browsing experience to its own UIWebView class, which is both a “black-box”, Pasricha says, and “buggy”.
Another example is provided by the recent introduction of single sign-on across all Google apps in Chrome for iOS to simplify user experience and to improve synchronization between desktop and mobile versions of the app. “This was a huge deal to Apple, because they saw that as a potential security risk. In the end, they accepted the point that managing the keychain for the user actually improves security, so finally Chrome got approved."
Another complication originates from the fact that the App Store does not allow to distribute beta versions of apps. This counters Google’s usual strategy for Chrome, which is based on giving out canary builds that end up being used by millions of people. All processes based on development, canary, and beta builds must be entirely internal to the company when it comes to iOS, and this makes the number of users shrink to a few hundred, which has important implications on the maturity of the app once it gets published.
Release process
In Google’s planning, Chrome follows a six to nine week release process that begins with getting feature approvals for the current milestone. This requirement is common to all platforms. During those weeks, main tasks are polishing features and bug fixing on the milestone branch, which is where most of the testing gets done in preparation of the review process. During this process, both automated tests and manual tests are carried through, focusing on regression testing and new features testing.
Once testing is done, the app is submitted for review, which can be “an interesting process”, Pasricha says. Google has no privileges granted from Apple, although, according to what Pasricha explains, they have a direct line with the review team and they usually do not receive any written details about the reasons behind a rejection, rather they get an invitation to “talk,” which sometimes can get escalated to some Google’s VP. According to Pasricha, the review process is generally never shorter than two weeks, but rejections are solved quickly: they “never had a milestone completely blocked, it’s usually just a misunderstanding.”
When the app gets approved, it goes from a “piddly number of internal users at Google to millions of users in the public,” and sometimes things do not work as expected, so the Chrome teams need to do a new build. The ensuing review process for the build fixing the bug is usually shorter, since they can just put a note when submitting that states that the build fixes some bugs and there are no new features.
Automated testing
An important piece in Google’s approach to testing is using bots, which are a software infrastructure that use physical devices to run the app. This is extremely important, although fragmentation in the iOS arena has not been an issue before the iPhone 6 and 6+, says Pasricha, and could become even more important in the future.
These automated solutions are absolutely the biggest thing that helps us get that early feedback on our code quality.
As mentioned, at Google testing is carried through across three channels: development, canary, and beta. This is how each channel is defined:
- Dev channel: used as a pre-submit check for features accepted into the milestone.
- Canary channel: contains the merge set of all the changes that are going into the trunk; canary builds have automated tests, unit tests, and end-to-end tests run on it.
- Beta channel: it is the closest to the App Store release and contains additional end-to-end tests.
Here is a description of the most important test types that are carried through:
- Unit tests: traditional “lower level tests to [check] expected variable values at various states” of the execution.
- End-to-end tests: they are managed through the KIF framework, which allows test automation for iOS by leveraging iOS accessibility features. According to Pasricha, KIF appears to be “a little bit more reliable, less flaky than UIA,” Apple’s own solution for test automation.
- Performance tests: they are based on a set of URLs that are run through in Chrome. This allows to compare the performance across runs. These tests are only used to compare performance in the same milestone, not across milestones, so the idea is just to watch over the release cycle that run times do not go up. -screenshot tests: in this case also a list of URLs is piped through Chrome and a screenshot is taken. If the difference in the rendered images is below a threshold, the test passes, otherwise manual inspection is required. This idea is very powerful, says Pasricha, but unfortunately, a lot of pages changes by 80% between runs, due to content changes and ads changes, so this testing produces a lot of false positives.
Manual testing
Automated tests notwithstanding, manual testing is still required and cannot be avoided, according to Pasricha. Generally, there are a certain number of cases where a manual test is the only way forward, e.g., with specific types of users that have some rate limit, so tests cannot be executed too fast; or special enterprise account types that put restrictions on actions, etc. But manual testing is also used when no test cases have been automated yet for a given feature. This may happen with the introduction of new features, which get approved and enter the development process late. Sometimes, indeed, they get through the whole milestone without any automated tests having been written. Finally, there can also be features for which it is simply deemed not necessary to have automated tests, like the Chrome tab switcher feature.
According to Pasricha, this is the area where she would like to see some change within the Chrome for iOS team, specifically requiring “the ownership of quality to be on the software engineers plate,” so that they are able to say they have launched a feature only if they also happen to have written a test for it, otherwise the feature cannot be launched. Admittedly, this is a very hard agreement to get from all stakeholders, including executives and product managers, who do not like that features won’t be published due to the lack of automated tests.
Manual testing is not executed on the dev channel. On the canary channel, two specific forms of manual testing are carried through: comparison testing and delta testing.,Both focus on changes, since it is well known that a heat map or a risk matrix will point to all changed components. Pasricha remarks that in this context, detailed release notes are of paramount importance in this kind of testing, since they provide information about where change is and which other features may be affected by that change. Again, developers do not like providing such detailed release notes, so in this case also there is some work to do to get to an agreement.
Additional testing
Other tests that are carried through are:
- Upgrade test: once the app is published, the first kind of test that is done is an “upgrade” test, which means updating an already installed app through the App Store.. At this stage, also all new features are quickly run through.
- Accessibility and internationalization: they are not run in full sweeps for each milestone. Full sweeps are done twice a year across the product.
- Security testing: every single feature goes through a security review carried through by an entire security team, except with smaller features that just get one single security engineer assigned.
- Geolocation: this is also important since several bugs have been triggered by versions of web sites that are served only to certain geolocales.
Continuous integration
As Pasricha explains, the Chrome for iOS team has set up a continuous build system called Pulse that is in use within Google. Previously, Pasricha has been playing with Jenkins, but they finally set with Pulse. Pulse is just used to create the builds and sign them properly so that the iOS instrumentation can be run on top of them; Pulse is not used to run any tests, though. After Pulse, there is Waterfall, which is in use on all Chrome platforms, not just iOS, and is responsible to run the tests and collect all the outcomes which are represented through a matrix of green and red boxes.
Tests are run both on OS X boxes under the simulator and on actual devices. There is one issue with the simulator: it has more memory available than real devices have, so it is not useful to find any memory-related issue, Pasricha warns.
Legacy iOS version support
When it comes to deciding whether you want to support a given version of iOS, Pasricha suggests, it is important to have set a threshold for the minimum number of users that you’re willing to restart the current milestone for in case a bug is found. This could be 5%, 3% or 1%, and it will end up being a huge number of people, in the hundreds of thousands. Sometimes, you can decide to support a small group of users although it is small in relative terms, and this is what happened with the iOS jailbreak community, recounts Pasricha, which was paritcularly relevant to Chrome for iOS because they were the only users who could ever choose Chrome as their default browser. But in general, if an older platform, say iOS 5 or iOS 6, does only account for a small percentage of users, then it is important to think about how much you want to invest in supporting them. Doing so has implications such as device and configuration proliferation, which could lead the test matrix to explode. But it also exacts a cost on future development, since you get somewhat entangled with missing features. Another dimension to take into account is the willingness of developers to fix bugs for an old iOS version: if they are not going to do it, then it makes no sense to support that version in the release process.
About the Author
Sergio De Simone is a iOS Independent Developer and Consultant and has been working as a software engineer for over fifteen years across a range of different projects and companies, including such different work environments as Siemens, HP, and small startups. Currently, his focus is on development for mobile platforms and related technologies. He tries to be a successful iOS independent developer and he is always on the look for challenging and new endeavours as a consultant. You can find a few pointers about him on his contact page.