Wellington's DevOpsDays NZ recently closed with a keynote by Katrina Clokie on the testing skills and superpowers that engineers can utilise in a DevOps setting. Clokie is the author of A Practical Guide to Testing in DevOps and is the Test Practice Manager at the Bank of New Zealand, where she leads and nurtures a cross-departmental testing chapter of 100.
Clokie spoke with InfoQ to discuss the changes she's seen in the testing landscape and how this is further impacted by the move to embrace DevOps principles. She describes DevOps as providing a "shift-right" which reframes testing in the light of live multivariate testing and platform validation through observability in production. Speaking on InfoQ's Engineering Culture podcast earlier this year, MindTree's Chris Manuel also discussed the shift to the right in testing which occurs as a result of "constantly monitoring what is happening in the live environment and using that to inform conversations about quality in the development process."
In her recent book, Clokie wrote about the cultural shift-right in testing which has been enabled through DevOps. She writes:
In a traditional project the development team may test in production after a large release to gain confidence that their changes have been applied successfully. It is a final check in a linear process, and rarely generates new work, with some exceptions for critical issues. In a DevOps world, testing in production is a way to generate constant input to the development team. It forms part of the delivery cycle, providing feedback that shapes future iterations of the software.
Clokie shared her view that while a lot of focus has been put into improving the development skills of testers, there has been very little focus on shaping developers into competent testers.
InfoQ: You delivered the keynote at the TestWest conference earlier this year, speaking on Testing in DevOps. What impact has the rise of DevOps practices had on our notions of testing?
Katrina Clokie: When DevOps is adopted successfully, the number of people involved in testing should increase but the scope of what they test might narrow. I think some teams stop asking broad questions about the quality of their product and become fixated on a set of automated checks that validate specific functionality. Testing can start to become synonymous with automated checking.
There is a lot of focus on speed to market; for many organisations this is the business driver of DevOps. The trade-off to quick delivery is often quality. As consumers in this world, it is becoming more acceptable for the software we interact with to be difficult, to behave in unexpected ways. As we become accustomed to trying again, or learn how to navigate non-intuitive interfaces, we validate this approach.
Organisations can rely on automation and machine-centred validation because users are being conditioned to accept a lower level of quality.
InfoQ: How did your personal journey into testing in DevOps affect your own impression of what it meant to be a testing specialist?
Clokie: DevOps introduces a lot of new ways to mitigate risk. Testing should be significantly impacted by this changing context. I think the biggest challenge for testers is letting go and allowing the organisation to mitigate risk in different ways.
InfoQ: Your DevOpsDays NZ keynote is titled Testing in DevOps for Engineers. How do you think the move to DevOps cultures has affected the testing responsibility of engineers?
Clokie: I think that in a healthy DevOps culture, testing is part of every engineering role.
InfoQ: What has the reaction to this change been from engineers whom you work with?
Clokie: Mixed, as with any change. Some embrace it, others not so much.
InfoQ: Do you think we'll arrive at a place where it becomes natural to see these two hats as differing strengths of a T/Π/Comb-shaped developer?
Clokie: In an ideal world, yes. In reality, I don't think so. We're becoming one-legged.
Testing has been cast as a supporting role in IT. The power dynamic in a team is usually with the developers who create the code over the testers who explore it. I don't think there's strong interest or incentive for developers to branch out. I see increasing value placed on development skills in testing, without a shift towards testing from developers.
InfoQ: How do you contrast testing in DevOps with the shift-left in testing which we've seen during the past decade?
Clokie: To me, DevOps is shift-right. We start to collaborate closely with the operations teams that support our software in production, and explore the opportunities that those new relationships provide to a delivery team e.g. monitoring as testing, A/B testing, etc.
InfoQ: What was it that inspired you to write your book on Testing in DevOps?
Clokie: We were embarking on a DevOps transformation in my organisation and it was making the testers quite nervous. I just wanted something to hand to my team and say "This. Please read this."
InfoQ: Are there any new strategies or learnings which you would have liked to have included in your book?
Clokie: I think Cindy Sridharan and Charity Majors have both produced some excellent content recently on observability. If I were writing the book now, I think their work would have been cited.
InfoQ: How would you describe the thinking which prevails and continues to define testing as an independent activity?
Clokie: Outdated. Testing is a collaborative activity, a shared responsibility.
Clokie's delivered her keynote at DevOpsDays NZ in Wellington, New Zealand, which was held from 5th to 6th November. Her book on testing in DevOps has been highly praised by Software Testing Magazine and sold nearly 6,000 copies within its first year of publication.
