Crittercism, maker of a mobile app performance management solution, has analysed one month of mobile app performance data and compiled it into a Mobile Experience Benchmark report focusing on two aspects of a mobile app's performance: app uptime and app responsiveness. Results highlight which app categories have the highest crash and error rates and provide the basis for a comparison among iOS and Android versions, as well as different devices and manufacturers.
Crittercism claims to have based its findings on the analysis of more than 3 billions events per day, gathering data from 1 billion app users who are accessing its solution monthly. The data spans over 2582 different mobile devices and 106 OS versions, relying on 691 wireless networks to access the internet. The heterogeneity of the mobile ecosystem is, according to Crittercism, one of the main factor influencing app performance.
Main Responsiveness Takeaways
- The average app depends on 6 cloud services, e.g. Facebook, Flurry, Google Analytics, etc.; the aggregated performance of such services is a key factor in determining app responsiveness;
- 9% of calls to services takes longer than 1 second; 21% of calls to services have error rates over 1%;
- Google Analytics is the best performing service, with Facebook and Twitter scoring quite low, both in error rate and response time;
- network access performance can vary a lot, going from a 0.8x response time for Canada to a 2.1x response time for India, being 1.0 the normalized US response time.
Main Uptime Takeaways
- 47% of apps crash more than 1% of the time;
- 15% have a crash rate over 2%;
- gaming apps have the highest crash rate at 4.4%;
- e-commerce and social apps crash rate is below 1%, while business, news, media, and gaming apps crash rate is over 1%;
- tablets crash more often than phones.
Android
- Gingerbread has the highest crash rate in the Android ecosystem at 1.7%, while still having a significant share among Android versions at 19%;
- KitKat, Jelly Bean, and Ice Cream Sandwich have far better performance than Gingerbread at 0.7% crash rate, with a cumulative share of 80% among Android versions;
- Samsung devices have the top spot for both Android tablets and phones stability.
iOS
- iOS 7.1 is the most stable iOS version, with a 1.6% crash rate;
- iOS 6 and iOS 7 have crash rate over 2%; iOS 6 has a share of 12% among iOS versions, while the split iOS 7/7.1 is not officially known;
- iPhone 5 is the most stable Apple mobile device.
Crittercism released a similar report in 2012 comparing mobile app crashes on iOS and Android and yielding similar results.