Corresponding to the ideas of "test early, test often" and "test as early as possible" in the development lifecycle, shift-left testing is not new as an approach to software testing in agile environments and with microservices.
Recently, the combination of shift-left testing and CI/CD is fueling a new approach to DevOps dubbed DevTestOps. InfoQ has spoken with Dan Belcher, co-founder at DevTestOps platform maker mabl.
InfoQ.com: You could say DevOps took the world by storm. Testing, of course, has been integral to DevOps from its start, from unit tests to end-to-end tests through integration tests. Of late, the idea that companies should “shift-left” their testing activities has been gaining ground, leading to DevTestOps. Could you briefly explain why DevTestOps is a game changer?
Dan Belcher: While the idea and practice of testing in DevOps is not novel, the fundamental shift that we’ve observed is in how entire software development teams are collaborating on testing throughout of that development lifecycle. The traditional approach of relegating most testing to the later stages or using divergent and incompatible approaches to testing across the team has proven both inefficient and ineffective. This pain is magnified in a DevOps context, where the pace of change is so much higher.
DevOps-oriented teams rely on automation to continuously integrate and deploy changes as part of a CI/CD pipeline. When done well, this model is highly efficient; almost no human effort is expended from the point where a change is merged into the master codebase to the point where it is available to users. In many teams, that total timeline can be minutes. It is also effective in maintaining quality. By minimizing the size of any individual change, we can understand and verify the impact of the changes on the broader system, and react quickly to any issues.
Many teams fall short of achieving this level of performance because they run into defects and broken tests after changes have already been merged to master and when they are already “in the pipeline” for delivery. When we find issues on master, we have to stop the pipeline so that people can fix the defects and/or update the tests. In a CI/CD world, this is incredibly costly because changes are accumulating as we wait for the pipeline to flow again, and that much larger batch of changes is much harder to test and reason about — basically the opposite of what we were aspiring to with CI/CD.
Shifting left is the only way to avoid this. High-performing teams shift left to ensure they do not introduce disruptive changes into the pipeline. That means investing deeply in quality in the stages before the code is merged to master. This includes not only unit testing and code reviews but also functional and end-to-end testing. In some teams, that means that developers assume more of the burden for end-to-end testing, while in others we see more collaboration and pairing between developers and QA engineers in the early stages. The goal is the same in either model: achieving the quality and speed of DevOps by keeping that pipeline flowing.
InfoQ.com: For an organization that has already made the shift to DevOps, what does it take to go DevTestOps? What benefits can be expected?
Belcher: DevTestOps requires a cultural shift starting with empowering test practitioners with the process and tools to collaborate with team members in other roles to test early and often. By recognizing collaborative concern for leading test practices among all members of the development team and incorporating best practices into the automation process, teams will be able to embrace DevTestOps to improve product quality.
As discussed in the last section, teams who are shifting left are investing deeply in quality in the stages before the code is merged. Building a culture of quality across the development team from beginning to end means promoting testing collaboration, but also requires the team to have the right tools and practice in place so development doesn’t slow down and quality isn’t negatively impacted. For a team that wants to adopt DevTestOps, mabl is the ideal solution. In short, it’s a SaaS solution that doesn’t require heavy lifting from the IT organization for infrastructure management. Mabl’s highly-intuitive UI makes it easy for anyone on the team to create and manage tests; built-in auto-healing reduces test maintenance, and native integrations makes it easy to connect with the more important tools in the development workflow (GitHub, Bitbucket, Jira, etc.). Mabl also recently rolled out a Selenium migration tool that enables users to easily migrate existing tests into the platform to avoid the work of recreating tests, while also immediately benefitting from test insights and auto-healing.
InfoQ.com: mabl describes itself as "the intelligent test automation company”. Could you please elaborate on this and explain what “intelligence” mabl provides?
Belcher: Mabl’s “intelligence” is most easily recognized by the decisions that it makes that are typically tedious work for software developers and QA engineers.
Mabl uses intelligence to determine if your application’s behavior is abnormal. While running tests, it observes many aspects of quality, including functional behavior, speed, visual presentation, errors and more. In addition to allowing the user to monitor these attributes directly, mabl models the attributes over time, including “normal” variability, and notifies your team when the behavior is anomalous. To illustrate: you may be interested in identifying unexpected visual changes within your application. While it may be possible, albeit tedious, for one of your team members to review all visual changes, many mabl users opt to leave the initial review to mabl itself. Because mabl can filter out any changes that the model predicted (such as those in dynamic elements that change regularly), it will only notify you when there are visual changes in elements that don’t typically change. The service takes a similar approach with speed; rather than alerting you every time a page takes more than a specified amount of time to load, mabl notifies you when latency is significantly outside of the normal range. This model-based approach leads to dramatically less noise and allows software teams to focus on actual issues.
The most important, and most frequent, decision that mabl makes relates to functional quality. With most UI test automation frameworks, the tests are tightly coupled to the structure of the front-end code. When that code changes, the tests fail. Teams are left to spend a great deal of time determining and implementing remediation for those failures. In some cases, they need to fix the application — resolve bugs in the new code — but in more cases, they need to update the tests to accurately exercise the modified code. Because mabl’s tests are based on a continually updated model of the application, it can automatically identify most cases where the tests simply update those tests without your team’s involvement, allowing you to again focus on regressions and issues that actually matter to the user.
Mabl’s intelligence helps you create more resilient tests that evolve over time, resulting in higher quality tests and less test maintenance. Native integrations with solutions like Bitbucket and GitHub to create intelligent pipelines further enable users to automate end-to-end test runs at every step of the development process. Mabl tests can be triggered in a local environment automatically as part of a preview environment when code is merged into master and integrated into a full regression suite.
InfoQ: Could you describe how mabl came into existence and what it strives to bring to current software development practices?
Belcher: Izzy Azeri and I originally founded Stackdriver together, which was acquired by Google in 2014. We started mabl in 2017 with the mission to help Enterprise software teams innovate faster by realizing the full potential of DevOps. We chose to focus on QA automation in particular because it is the most acute pain point for so many teams in their DevOps journey. We see intelligent testing as a way of not only enabling QA to keep pace with development, but also of improving alignment across the entire team and delivering better user experiences.
With a framework that is accessible to developers, testers, and non-technical users alike, mabl allows anyone to contribute to quality. With an intuitive SaaS platform and rich integrations across the DevOps ecosystem, mabl fits seamlessly into the DevOps workflow. By taking advantage of the advances in machine learning, data analytics, and cloud computing, mabl makes it much easier for enterprises to manage quality at scale. The company has raised $28M from Charles River Ventures (CRV), Google Ventures (GV), and Amplify Partners.
mabl has recently introduced new shift-left testing oriented features to its platform as well as new CI/CD pipeline integrations, including automated test integrations with GitHub and GitHub Actions, Bamboo, Bitbucket, Jira, and Selenium. Developers can now create headless local runners when testing and debugging production issues without interrupting their daily workflow. Additionally, they can now branch their test definitions for more effective test change management.