Mark Hrynczak, Cloud QA Manager for Atlassian, gave a talk on this year’s company summit in which he shared his vision of how a high valuable QA team should perform.
He defines high value for a QA team as being, in the first place, totally aligned with the company strategic goals, thus contributing to solve the most important problems that an organization might face at a specific moment.
InfoQ: Can you briefly explain to the readers what “Quality Assistance” consists of and how it has evolved to the so called “QA Champion” model?
Mark Hrynczak: Quality assistance is based on the insight that a software development team needs to optimise for quality and speed, rather than trading off one for the other. Instead of improving quality by adding more testing stages, more process, and more testers, we empower and educate developers to test their own features to production quality standards. Instead of a QA team improving the quality of software directly, our team aims to improve the quality of the dev team producing the software. I’ve written more on this subject at https://www.atlassian.com/inside-atlassian/qa
InfoQ: The Net Promoter Score (NPS) is one of the recommended tools that a company may use to guarantee that strategically, they are focused on the right problem. As NPS is a tool that assesses customer satisfaction for the whole company, how do you establish a connection to the specific actuation of the QA team members?
Hrynczak: Our use of NPS is product based - the data is collected via an inline dialog that intermittently asks people to rate the product they are currently using. Our products have very large user bases, and the aggregation and analysis of the data gives us a lot of insights into the feelings of those users towards it. Those insights give us guidance for future decisions - what features users want to see, which areas of the product are hard to use, which types of bugs/behaviours frustrate them the most. We don’t aim to make a strong connection from the activity of a particular team member to the overall NPS score - it is a metric for the entire team. As we take action on the insights gleaned from the data, we should expect to see an upward trend on the subset of data relating to that insight - representing happier users and a better product.
InfoQ: One of the key ideas of the talk is about gaining the ability to substitute remediation with anticipation techniques like prevention, mitigation or eradication. Can you give us examples of their usage and how one could benefit from them?
Hrynczak: Sure, please find below examples for the techniques you’ve just mentioned:
Prevention: a pattern is noticed where bugs frequently occur with a common cause - say some particular characters trigger some unwanted effect. A tester in a traditional approach might create a set of strings they use as default test data - whenever they are delivered software to test, they use those strings, detect bugs, and are recognised as successful testers. With a prevention approach, we give those strings to the developers before they code. We agree upfront that such characters should be supported, developers write the code to handle those characters, and use them in the automated checks.
Mitigation: a production system has many thousands of users - when a bug is deployed to all those users it can have a very high impact. Traditional testing approaches would spend time and effort trying to identify and catch every single potential bug to protect the users from that impact. With a mitigation approach, we might instead change the deployment strategy - staggered rollouts, blue/green deployments, error monitoring and automated rollbacks, etc. The same bug can get through our pre-deployment checks, but the impact is limited to very few users as we stop or rollback the deployment once the bug becomes visible. With a good strategy we can limit the impact to only internal or unimportant users, and the bug is never seen and has zero impact on our actual customers.
Eradication: security issues can happen when developers are unaware, or forgetful, and do not write code defensively - and it only takes one vulnerability to make your product insecure. Instead of performing extensive security tests against every change, we can massively reduce the chance of such issues by making code safe-by-default. We can implement auto-escaping in templating libraries to avoid XSS bugs. We can require dynamically generated tokens on all forms to avoid CSRF bugs. We may not entirely eliminate the need to perform security testing, but we can drastically reduce the number of similar bugs found when it happens.
InfoQ: Do you have a concrete advice for an organization that sees QA activities as a bottleneck and wants to move towards a direction similar to yours?
Hrynczak: Shifting from a traditional model to a quality assistance mindset requires an organisational cultural change - not just QA but the developers, the leads, and other stakeholders. You might not need to convince everybody straight away - but hope that more will appreciate the model when they see the efficiency and quality goals being met. There are a few prerequisites:
Leaders that are supportive and open to the concept. They must hold the entire team accountable for results, both good and bad. They must recognise that a developer spending longer to code and test a story is preferable to completing it faster by skipping the testing. They must resist the mentality that the reason a high-impact bug was shipped was that the QA team didn’t catch it.
Developers that care about their end-users, and are intrinsically motivated to produce quality software. There should be a general recognition that a great developer is one who can capably write and ship high-quality, bug-free code, and that an employee who insists that a testing team is fundamentally required to catch their mistakes is probably not meeting that bar.
QA team members that can see their value beyond “testing”, that understand the realities of a fast development workflow, and can let go of the idea that they are the gatekeepers between unreliable devs and valuable end-users. The set of skills needed to be successful in such a role is different, and may be uncomfortable - I’ve written more on this at https://www.atlassian.com/inside-atlassian/software-QA-skills