Teams need to match testing with the market usage patterns across geographies of their apps as consumers are expecting smooth apps functionality across all digital channels, argued Eran Kinsbruner. He developed a methodology and index for considering device/OS combinations together with other characteristics like aging, screen parameters and other testing related guidelines for data driven test coverage of the mobile market.
Eran Kinsbruner, mobile evangelist at Perfecto Mobile, spoke about building the right mobile test lab for your business at the Spring Online Testing Conference 2017. InfoQ is covering the conference with Q&As, summaries and articles.
InfoQ interviewed Eran Kinsbruner after his talk about building a mobile test coverage strategy.
InfoQ: What makes test coverage important, why should we pay attention to it?
Eran Kinsbruner: The digital space is complex more than ever. Desktop browsers like Chrome and Firefox are releasing a public beta or GA on a monthly basis, while mobile devices and OS versions are also being introduced to the market in a similar cadence. Each release not just introduces a lab challenge that requires dev and test teams to test against and identify regressions and keep up, but also introduces new features and functionalities. The reality today is that the consumers are expecting smooth apps functionality across all digital channels, whether they are mobile smartphones, tablets, desktop browser or IOT. To meet customer expectations, teams need to have an updated test lab that matches the market usage patterns across geographies. Neglecting to pay attention to test coverage introduces risks to customer brand, and plays to the hand of the competition.
InfoQ: How can we measure mobile testing coverage?
Kinsbruner: Mobile testing coverage is hard to measure, that’s why I built a methodology that is based on market usage analytics and looks at the market devices from various angles.
The way to measure mobile testing coverage is to understand through usage and customer analytics which are the top ~30 devices in your supported geography that are mostly used. Then assure that that list includes sufficient permutations from both OS version perspective as well as device ages (New and trending, Most popular, Legacy).
For iOS we are seeing 3 OS families today (iOS 10, iOS 9 – legacy, iOS 11 Beta), that ought to be covered across the top devices. The problem is that there are devices that can only run on iOS9.3.5 (iPhone 4S, iPad Mini), and once iOS11 is out, iPhone 5, 5C, iPad 4th Gen. will be also left behind with iOS 10 as the latest supported OS. For Android, the complication is even higher. Developers must be on top of the latest OS version, and to get access to that version they must use a Nexus/Pixel devices. The remaining Android OS families are spread from popularity perspective across the various OEM’s. We see vendors such as Samsung and LG that are trying to keep up with Google, but only recently provided access to Android 7.0 to their leading devices – that happened nearly six months after Android 7.0 was released. To measure and keep up with the mobile market coverage, I developed a 3 layer methodology of Essential, Enhanced and Extended segments an Index of top 32 devices per geography that takes the above considerations of Device/OS combination together with other characteristics like aging, screen parameters and other testing related guidelines and provides a data driven test coverage of the mobile market. The Index includes current devices/OS, legacy as well as trending platforms that based on market research will become very popular and their inclusion in a test lab makes a lot of sense to both developers and testers.
InfoQ: What can you do to test for end user experience as much as possible?
Kinsbruner: The only way to assure high user experience, is to mimic your user's environment, their journey as they consume your app data and their full profile. App developers should be able to define based on analytics and app user stories, a set of end user profiles such as:
- Persona (Male, Female, Age ranges, etc.)
- Location
- Network condition that is also location based (lead carriers in the supported geo’s)
- Background apps
- Non-full battery charge
Once teams can set up the above conditions, that would be the perfect lab setup to do real UX testing for their app on the right device coverage that is mentioned above.
InfoQ: Which advice do you have for deciding for which devices and OS versions to test?
Kinsbruner: I would highly recommend to use existing analytics tied to market data as the above mentioned Index report, and validate and calibrate this data on a quarterly basis as a continuous practice. In addition, I am a believer of planning ahead, and that’s why I also recommend to build or leverage a mobile market calendar that projects the upcoming releases of devices, OS versions, browsers etc. In the Index report, I do provide my own calendar for mobile and web that also include OS version usage adoption that can also guide the users to when to seriously shift the testing to the latest major OS version. The important thing about test coverage is to know your users, to know the market and assure that your test lab is able to mimic it as close as possible.
InfoQ: How can we improve mobile and web testing?
Kinsbruner: In order to mature your entire release velocity, innovation and competitive edge, you need to be able to automate the entire build process throughout your pipeline to include as many tests from both functional, UX, Performance as possible and drive faster feedback back to your developers. The test lab that you use ought to have the optimal coverage, and your automation of the build cycle need to be running unattended to assure fluent process. Each code change should trigger some level of testing on the right platforms and the results that goes back to the developer need to be reliable and with proper insights so decisions will be properly made. Skills and tool suite are obviously an important part of the overall testing process, and they need to be selected based on your app capabilities, team skills, and more. What we see in the mobile testing landscape today is that there are various open-source test frameworks that significantly varies between each other’s, therefore teams need to either leverage more than one framework in order to automate their entire test plan that will often include complex scenarios like in the UX checklist above, or do thorough analysis before deciding on the best match.