Two recently launched services, Appurify Mobile Platform and Xamarin Test Cloud, promise to help developers and enterprises automate testing and performance optimization of their mobile apps by allowing them to run apps on real iOS and Android devices and supporting automated testing and continuous integration.
Mobile apps testing poses developers its own special challenges. According to Wikipedia key factors are:
-
Variety of mobile devices with different capabilities such as amount of memory, screen resolution and orientation, and network connectivity options;
-
Diversity in mobile platforms, e.g. iOS, Android, Windows Phone, with each platform supporting a number of different OS versions in their installed base;
-
Variety of mobile operator networks protocols, such as GSM, CDMA, and FOMA, and varying operating conditions affecting parameters like signal strength and available bandwidth.
Jim Cowart writing for Developer Economics also identifies immature tooling and lack of awareness as two additional challenges that developers need addressing.
Both Appurify Mobile Platform and Xamarin Test Cloud stress in their press releases the importance of testing app functionality and performance on multiple devices and claim to offer hundreds of them to choose from. Developers can write their own scripts for UI testing and integrate them in their existing development process through a command-line interface or API. Tests can also be run directly from continuous integration platforms like Team Foundation Server, Jenkins, and TeamCity. Appurify also claims they can test apps against a variety of carriers and signal strengths.
The importance of testing functionality and performance of mobile apps has been measured by Appurify through a classification of 1-star reviews of the top 200 apps in the iOS app store. What came up is that 52% of 1-star ratings were related to crashes or network performance or UI lagginess.
On the other hand, in another study commissioned by Appurify and conducted in Dec. 2013 among 400 US developers, 78% of all surveyed developers responded that they manual test their apps on real devices, while as much as 41% of them reported to count on users to find issues; just 10% of all surveyed developers declared to use automated testing.
Other companies offer mobile device testing services, such Perfecto Mobile and Keynote DeviceAnywhere. In particular, Perfecto Mobile also allows to "perform remote manual testing on real smartphones and tablets regardless of where you are". According to Appurify's CEO Jay Srinivasan, the key differentiator between its service and rivals is that it can offer a deeper level of automation and granularity, allowing customers not only to test different devices, but to test them on varying network conditions and memory levels. Half of errors seen in the wild, Srinivasan said, come not because the device itself wasn’t tested but because it wasn’t tested under the right conditions.
