There are a number of automated testing frameworks or tools for Android applications, including but not limited to Activity Instrumentation, MonkeyRunner, Robotium, or Robolectric. LessPainful is providing automated tested on real devices as a service.
Android has basic support for instrumentation tests, one of the possibilities being the ActivityInstrumentationTestCase2 class belonging to the android.test package and extending JUnit’s TestCase in order to provide functional testing of Android activities. When an application is tested, each instrumented activity is launched inside the Dalvik VM on an Android emulator or an actual device.
The Android SDK comes with a testing tool called MonkeyRunner, providing an API and an execution environment for running tests written in Python. The tool has APIs for connecting to a device, installing/uninstalling apps, running apps, taking screenshots, comparing images to see if the screen contains what is supposed to contain after certain commands have been performed, and to run a test package against an application. MonkeyRunner uses the InstrumentationTestRunner class to run tests by defining test cases with ActivityInstrumentationTestCase2, ProviderTestCase, ServiceTestCase, SingleLaunchActivityTestCase, and others.
Robotium is yet another framework that interacts with Android’s testing support via the InstrumentationTestRunner, providing functional, system and acceptance tests running across multiple activities. Robotium supports Activities, Dialogs, Toasts, Menus and Context Menus even for Honeycomb, and it is integrated with Maven and Ant for running continuous integration tests. Robotium is similar to Selenium, but for Android applications.
Robolectric has taken a different path. Instead on relying on Android’s testing functionality provided, this testing framework uses shadow objects and runs the tests on a regular workstation/server JVM avoiding dexing, packaging, deploying and running the application tested on an emulator or a real device, reducing the time needed to perform tests. Pivotal Labs claims to be able to run 1,047 tests in 28 seconds with Robolectric.
LessPainful has taken Android testing a step further, offering a service for automated testing of applications on multiple real devices. The user uploads the application (*.apk) and a test file written in Cucumber, a business-readable DSL, selects the configuration of devices he wants the tests to run on, and then the tests are automatically executed and a report is generated. The list of devices supported include Garmin Asus, several HTCs, LGs, Samsung Galaxy, Sony Xperia, and Motorola Motodefy.
We have talked to Jonas Maturana Larsen, CEO of LessPainful, to find out more details about the service. Following is a short interview:
InfoQ: What problems there are running apps on different versions of Android? Does a developer need to test his app on every version of Android to make sure that it runs correctly?
JML: One example: The SAXParser on Android before 2.2 has a bug in the callback to ContentHandler.startElement this causes the app to misbehave.
In general we have seen quite a few differences between OS versions. Something would crash on 2.1-update1 but work on 2.1-update3 and 2.2.
InfoQ: Is there a real Android fragmentation among different devices? Can you give an example of an application that runs on Android 2.2 (for example) on HTC and does not run on a Samsung? (Use any combination of Android version and device manufacturers.)
JML: On LG phones HorizontalScrollViews will sometimes cause background images on child views to disappear. It occurs on all the LG phones we have tested and across different OS versions.
The themes will cause your app to look different on different devices if you don't handle it yourself. Motorola will, for example, highlight an input field with a red border. I have worked on a project where we'd use exactly the same red border for show input errors.
Other problems are more hardware related than manufacturer related: Some phones with little RAM and a high resolution camera will crash if you try to process an image on the phone.
InfoQ: How are tests executed?
JML: The test is basically run as an ActivityInstrumentationTestCase2 and using Robotium. The only modification we make to the application is to strip the signature and resign it.
After the test has run the app is uninstalled and the phone settings are brought back to default.
InfoQ: What are the advantages of your service compared to MonkeyRunner, Robotium or Robolectric?
JML: LessPainful is a service not just a framework. We want to build a service that not only makes it possible to test but does so in a way that saves people massive amounts of time and helps them find more errors than they would if they were using one of the frameworks.
Another thing is that we believe that using Cucumber results in nice high level test specifications that can be shared outside the dev team.
You might say that we want to be more like github is to git than some git library is to git.
InfoQ: Do you plan on supporting more devices in the future?
JML: Yes. We plan to continually add support for more devices. If somebody asks for a device we try to get it.
We are currently working on supporting iOS devices too. We expect it to be ready for public beta during the fall.
InfoQ: What is LessPainful Enterprise?
JML: We will provide a box. Most likely it will be a Mac Mini but we are very flexible when it comes to customer needs. LessPainful Enterprise is currently not an off-the-shelf product so it really comes down to what the customer is looking for.
