Q&A on The Digital Quality Handbook

The Digital Quality Handbook explores the challenges of testing mobile and web applications and shows how to apply agile practices to deliver quality at speed. Some of the topics covered are test automation, sizing mobile testlabs, addressing test flakiness, crowdsourced testing, performance testing, and applying DevOps practices. The book is written by Eran Kinsbruner, mobile evangelist at Perfecto, and contains guest chapters contributed by testing experts.

InfoQ readers can download chapter 12 of the Digital Quality Handbook on using smart object locators for robust test automation.

InfoQ interviewed Eran Kinsbruner about what digital quality is and why we should pay attention to it, the biggest challenges for testing mobile applications and how to deal with them, recommended agile practices for mobile testing, testing a chatbot, how to find defects earlier, and how to foster frequent and effective collaboration between testers and developers.

InfoQ: Why did you write this book?

Eran Kinsbruner: The book was developed after gathering so much personal experience that I felt was important to share with a wider audience in a form of a book. Mobile is the leading engagement channel for brands, and still organizations are struggling to assure the quality of their apps across user conditions, measuring performance, releasing faster through automation that works within CI as well as pre-production, and more. I was able to build a consortium of experts that together with myself developed the DevOps guide that can help these organizations implement ‘continuous testing’ processes and shift as many tests as possible into their build cycle.

InfoQ: For whom is this book intended?

Kinsbruner: The book is intended for developers, test automation engineers, and quality executives, as well as businesses and whoever takes an active part in the DevOps activities when it comes to releasing a digital product to the market, whether it’s a mobile app, responsive web, or both.

InfoQ: What is digital quality and why should we pay attention to it?

Kinsbruner: Digital quality relates to ensuring UX and functionality of any user engagement channel that users choose to interact with a brand. It can be a chatbot, a mobile native app, a web app, responsive app or IOT device. When building a digital quality strategy, organizations ought to consider various aspects such as the human who uses the app, his various profiles and traits such as location, background apps, network conditions, and more. In addition, considering the coverage is an essential piece of digital quality - how many digital platforms like devices/OS combinations, desktop browsers, IOT devices, voice interfaces and more, are sufficient to assure risk free app deployment.

InfoQ: What are the biggest challenges for testing mobile applications?

Kinsbruner: The biggest challenges are as follows:

• Market fragmentations across iOS/Android and the various geographies. In different geo’s users adopt different devices, and use different OS versions.
• Frequent changes of new device/OS that relates to the above, mandates continuous testing and proper automation.
• The automation frameworks, especially the open-source, are still finding it hard to close the gaps around new features introduced by new OS versions. E.g. Support for visual testing, location, FingerPrint within leading frameworks like Appium are still not supported. Voice and audio quality are an additional example.
• Managing a device lab locally is a great challenge for dev and test teams, therefore they adopt cloud services.
• Mimicking real user conditions and understanding the end-user profiles is an additional challenge. Teams need to test against the exact end-user conditions to assure that their app will work flawless.

InfoQ: How can testers deal with those?

Kinsbruner: Teams today are adopting cloud services that reduces the pain of managing a local device/desktop browser labs. In addition, teams are either developing overlays on top of the open-source frameworks like Appium/Protractor and such to close test automation coverage issues, or are using a mix of tools as part of their tool stack to get proper testing capabilities. In addition, keeping an eye on analytics and market trends as a way to understand how the market is moving, and which devices are trending up or down, can also help. With analytics in mind, having a well structured testing strategy that learns from previous executions and provides insights back to the team can help focus the testing on the most valuable tests - quality and value over quantity. As an example, identify the tests that found the most defects per platform. Or, the most problematic area in the app that accounts for many defects over and over again; such area should be focused on the most in early stages.

InfoQ: Which agile practices do you recommend for mobile testing?

Kinsbruner: I see many teams today that specifically for mobile are choosing the following practices and it allows them to be more agile:

1. Combining multiple tools (mostly open-source frameworks) for Dev-QA-Ops that addresses the various testing requirements in each step of the SDLC process, as well as matches the varying skillset. What I’ve seen in various organization is a combination of tools like Espresso and XCUItest mainly used by developers for UI unit testing and fast feedback, and Appium/Selenium used by test teams for larger regression tests and additional tests that cover areas that can’t be covered by the above mentioned tools.
2. Changing SDLC workflow toward BDD/ATDD also helps organizations to move to faster releases, and faster feedback between the teams. Moving to a BDD workflow requires an entire organizational change in mindset, tools, and processes. BDD mandates fluent communication by Business/Product Owners/Dev and Test. This communication results in implementing the exact product that matches the end-user requirements driven by the business and product owner.
3. Shifting as much valuable tests into the CI and build process is key to drive faster feedback and release cadence, as well as identify and resolve defects faster. Making sure that the build cycle is always green while eliminating flaky tests supports this objective.

InfoQ: Is it possible to test a chatbot? How?

Kinsbruner: Yes, of course.

Testing chatbots can be done in various ways.

The ability to use VTS/STV (Voice-to-speech/Speech-to-voice) techniques as part of automation in combination with data driven testing can allow teams to structure such tests. As an example, while leveraging Siri/Google Assistant voice, teams can record the audio that is being “injected” to the bot and translate it to text through automation and vise versa.

See code snippet as an example

(Click on the image to enlarge it)

InfoQ: What can be done to find defects earlier?

Kinsbruner: FInding defects earlier as a key to drive more agile processes can be achieved through robust automation. Automating the entire build process from the moment a developer committed his code changes allows teams to immediately get the impact of these changes. If per commit a triggered test suite is launched, there are obviously no surprises later in the process. The larger the suite is, the higher the confidence that the regression done by the testing team later will be smoother - This is not a bullet-proof method, but a risk based one - it relies on an analytical test selection, such that provides the highest value to the developers. Aggregating from previous runs tests from various areas like functional, UI, non-functional, UX etc. that cover the major and most important features raises in-cycle testing. The remaining tests that were not part of the cycle complement them and run on additional devices/OS versions and that’s the risk-based element - choosing the tests and target platforms that can be covered outside of the cycle.

InfoQ: How can you foster frequent and effective collaboration between testers and developers?

Kinsbruner: The collaboration is usually triggered by trust and common tools used by both Dev and Test individuals, that typically reside in one DevOps/Agile team. To build trust, the testers need to support the dev processes, build tests that are adding value and are not flaky or immature. Once the trust is built, testers can feed their tests into the build cycle as mentioned above, and be part of the dev cycle. This will ease the overall communication, the velocity of releases and of course the product quality. Having joint testing tools also increases collaboration, since when trying to solve an issue and run fast regression, both testers and developers will use the same foundation/source repository. Lastly, a test should be treated as code in order for it to be always up-to-date, robust, and reused by multiple teams.

About the Book Author

Eran Kinsbruner is the mobile evangelist at Perfecto and the author of the digital quality handbook. He is a software engineering professional with nearly 20 years of experience at companies such as Matrix, Qulicke & Soffa, Sun Microsystems, General Electric, Texas Instruments and NeuStar. He holds various industry certifications such as ISTQB, CMMI, and others. Kinsbruner is a recognized mobile testing influencer and thought leader. He is also a patent holding inventor (test exclusion automated mechanism for mobile J2ME testing), public speaker, researcher, and blogger. Kinsbruner can be found all over social media, including on Facebook, Twitter (@ek121268), LinkedIn, and his professional mobile testing blog.