Google has open sourced Espresso, an Android automated testing framework that enables one to run tests on x86 machines in the cloud in a multi-threaded environment, solving the concurrency issues associated with UI testing.
Running tests on real Android devices is highly time consuming and expensive due to the large number and variety of such devices. A solution is to execute tests on an emulator, a controllable environment supporting multiple OS versions, screen sizes and memory constraints. This approach would catch most code bugs, the rest of them being left to real device testing and human testers.
The problem with emulators is their execution speed, the bottleneck being running Android on an emulated ARM CPU. To solve this issue, Google has created an Android version that runs directly on x86 hardware and uses VM acceleration. Another bottleneck is the Android boot-up time, and that has been addressed by taking snapshots of the OS and running them, which offers the desired OS and applications configuration in a short amount of time. Using this approach, Google has performed 82 million Android tests back in March of this year.
Comparing testing on a Nexus 4 vs. an emulator, the later needed about 65% of the time of the real device to finish the test. So, it seems that a good emulator has solved the automated testing problem. But there is yet another stumbling block. Automated testing uses the Instrumentation API on Android, and those API calls run in a different thread than the UI thread, so testing the user interface in an automated way can lead to serious concurrency issues resulting in inconsistent and unreliable test results. Google’s solution to this problem is Espresso, a testing framework that lets UI tests run safely in a multi-threaded environment and removes most of the boilerplate code associated with writing tests. Espresso works across various mobile devices, including phones, TVs, smart glasses, cars, etc., different screen and memory sizes, multiple API versions and different networks. According to the team behind Espresso, their solution catches 99% of the bugs in Android and applications, leaving a few of them to real device testing and human testers, reducing the overall testing effort.
An example of a test asserting that a view is not displayed looks like this using Espresso:
onView(withId(R.id.bottom_left)).check(matches(not(isDisplayed())));
Other test samples can be found here.
Currently, Espresso is in Developer Preview, and offered on Google Code, but the idea is to move it into the Android SDK when the framework is mature and stable enough. Espresso is used by Google to test over 30 applications of their own, including G+, Maps, Drive.
