Regression Testing Strategies: an Overview

Costly and time-consuming, regression testing is a nagging pain point for the whole delivery team. Luckily, it is possible to make regression testing less painful and more efficient. To do so, you should devise an efficient regression testing strategy, fully covering the needs of your product and assuring product quality at an optimal cost. This requires an understanding of the nature of regression testing, the reasons for it and the ways to perform it.

Reasons for regression testing

A software regression is a defect that appears after a certain event (for example, a software patch or upgrade). Regression testing ensures that recent changes to the code leave the rest of the code intact, thereby preventing software regression.

Change is the key concept of regression testing. The reasons for these changes usually fall into four broad categories:

• New functionality. This is the most common reason to run regression testing. The old and new code must be fully compatible. When developers introduce new code, they don’t fully concentrate on its compatibility with the existing code. It is up to regression testing to find possible issues.
• Functionality revision. In some cases, developers revise the existing functionality and discard or edit some features. In such situations, regression testing checks whether the feature in question was removed/edited with no damage to the rest of the functionality.
• Integration. In this case, regression testing assures that the software product performs flawlessly after integration with another product.
• Bug fixes. Surprisingly, developers’ efforts to patch the found bugs may generate even more bugs. Bug fixing requires changing the source code, which in turn calls for re-testing and regression testing.

It’s important to understand the difference between these two types of testing. Re-testing features test cases that report bugs. Testing engineers perform them again to check bug statuses (fixed, not fixed, etc.). When bugs are still present, the development team receives a bug report for further debugging. After debugging, the testing team runs regression testing to assure that the application still works as intended.

Types of regression testing

The reasons for regression testing provide grounds for distinguishing three types of this testing technique. The types of regression testing are:

1. New bug fix: verifying that a recently found bug is fixed successfully.
2. Old bug fix: making sure that a bug, once found and fixed, didn’t emerge again.
3. Side-effect: verifying that recent bug fixes haven’t damaged old functionality.

Regression testing strategy: basic factors

Having analyzed the reasons for regression testing and its types, we can proceed with working out an effective regression testing strategy. When designing a regression testing strategy, a team relies on two factors:

1. Product nature. This is the key factor for choosing a relevant regression testing strategy and careful regression test planning. For example, approaches to testing a landing page and a comprehensive professional portal will differ significantly. While for a landing page regression testing mostly features UI and usability test cases, for a portal it may employ multiple test cases for security, performance, compatibility testing and more.
2. Product scale. Large-, medium- and small-scale products require a different approach to regression testing. For a small product, a single round of manual regression testing may be enough, while for medium and large products with rich functionality, both manual and automated regression testing may often be run.

These factors allow testing teams to select adequate regression testing approaches and methods.

Regression testing methods

Regression testing follows two implementation methods: manual and automated.

Manual regression

Manual regression testing is the basic method for regression testing for every product, regardless of the methodology (waterfall, Agile and others). A regression test suite dwells on test cases describing areas of the application that have seen recent changes and its adjacent areas. This type of testing always precedes automation, in some cases being even more efficient than the latter. For example, it’s impossible to write test scripts for testing application areas adjacent to the part of the code that was changed.

Manual regression testing proves to be efficient at the early stages of the product delivery process. For a product we worked on, an iOS image processing app, manual regression testing helped detect several bugs hampering the app UX. The app failed to render images correctly and crashed when the user changed screen orientation.

Still, testing teams seek to automate regression testing. The main problem with manual regression testing is that it is time- and effort consuming. For complex products, running a regression test suite over and again inevitably hinders a testing engineer’s concentration and his or her performance. Thus, in such cases teams prefer to recur to automation.

Automated regression

Automated regression testing is typical for medium and large projects (six-months long or more) at the stage when the project is stable (no critical changes in business logic and UI are expected).  With thorough regression test planning, automation reduces the efforts a testing team spends on tedious and repeatable tasks, and spares them time for testing that requires human opinion, such as exploratory testing and UX testing.

However, today teams often start regression test automation at the early stages. It works well for Agile development where teams should deploy a product at least weekly and have no time for warming-up manual regression testing. Communicating with the whole team (stakeholders, managers, BAs) and studying the use case document, testers can understand the stakeholders’ needs, the product business logic and expected results for testing.  The main task in early automation is choosing the testing framework. It should provide for easy scripting and low-cost test maintenance. The created infrastructure can be reused for future products, accelerating test automation.

In certain cases, automation may help detect bugs that weren’t found by manual regression testing. Bugs that appear randomly are the most illustrative example. When we tested the image-processing app described above, automation helped detect several random bugs thanks to automated testing timeouts. If a script didn’t fit in the timeout, it was automatically marked as failed. As a result, the same script was run again and again until it passed, and random bugs did appear at a certain point.

Regression testing approaches

Regression testing provides for two possible implementation approaches:

1. Full regression

This regression testing approach comprises of all regression test cases covering the product in full. QA teams usually perform it at the final stages of the product delivery process or before major releases. Full regression is also used when the product required significant functional and/or non-functional changes or when these changes affect the root code. Luckily, testing teams don’t have to write a whole regression test suite whenever it is needed. They usually revise functional, non-functional, unit and integration test suites and choose test cases that repeatedly find bugs throughout the product delivery process. These test cases make a regression test suite.

Though lengthy and tedious, this retest-all approach is effective as it helps discover possible issues throughout the whole application. However, this type of testing makes no sense when done often. Teams usually perform it before changing the development environment. Let’s consider the image processing app we’ve already mentioned. The app was originally developed for iOS 8, so the team used XCode6 IDE. But later the customer asked to enable users to run the product on newer devices powered by iOS 9. This required a transition to another IDE - XCode 7. After the transition, testing engineers had to run full regression testing to make sure all the features developed in XCode 6 stayed functional.

Full regression testing can also be specifically required by the customer when he or she wishes to be completely sure about the product stability and its ability to satisfy their needs.

2. Partial regression

Partial regression testing features the modified part of a product and the adjacent areas that might have been affected. Testing teams use specific strategies to ensure that partial regression testing yields good results. The main strategy here is a risk-based approach. Testing engineers single out the application areas that might be affected by recent code changes and select relevant test cases from the test suite.

A QA team can go further and apply the risk-based approach to a regression test suite when the product acquires new features of any kind. This selection technique significantly reduces testing time and effort, and is another good choice for iterative regression testing for Agile product delivery process when teams are pressed for time. Partial regression also helps reconsider the full regression test suite for the final development stage and discard obsolete test cases.

The choice of an approach depends on the scope of changes, the methodology and the stage of product delivery process. In our testing practice, we stick to risk-based partial regression testing whenever possible.

Regression testing in popular methodologies

Product development methodology is one of the key factors to consider when choosing a suitable regression testing strategy. In the waterfall methodology, testing is introduced when the product is fully developed. Testing in waterfall usually falls into three stages:

• Testing as such. The testing team gets down to work and runs all test suites/scripts they developed to cover the product. When testing is done, the team hands bug reports to the development team, who then fixes the bugs.
• Stabilization. When bugs are fixed (supposedly), the testing team runs regression testing to check bug fixes and their adjacent areas. Stabilization may take up much time, as bug fixes often generate new bugs. The testing team needs to evaluate them in terms of severity (critical, major, medium or trivial). When the product has critical and major bugs, the development team proceeds with another round of bug-fixing.
• Regression testing. When critical and major bugs are fixed, and stabilization doesn’t find bugs of the kind any longer, the testing team runs full regression testing to make final amendments to the product.

Thus, stabilization and full regression are a critical and time-consuming part of testing in waterfall methodologies.

In Agile development, regression testing usually falls into the two above-mentioned subtypes: partial (iteration) regression covering the features developed during the iteration and the adjacent areas, and full (major) regression performed before major releases and deployment. However, to be efficient, regression testing suite requires maintenance: the testing team should add new relevant test cases and delete obsolete ones timely. This approach significantly reduces the time and effort spent in this tedious type of testing and with no damage to product quality.

Regardless of the methodology they follow (waterfall or Agile), product teams can choose certain optimization approaches.

Optimizing regression testing: Kanban and DevOps

The Kanban approach consists of using a product dashboard that helps to clearly visualize work and track progress and improvements. This way, every team member can estimate their workload, relate it to the team work, set deadlines and ensure efficiency. In waterfall, Kanban helps estimate the time needed for stabilization and plan testing efforts more carefully. In Agile, the Kanban dashboard helps select test cases for iteration regression and critical points for full regression.

DevOps comprises of continuous integration (CI), continuous delivery (CD) and continuous deployment. This approach heavily relies on automation: a small part of code is sent to the repository, where CI ensures automatic integration of the parts of code into a build. Then the build is shipped to the staging environment (CD), and testers run automated testing. This approach significantly minimizes regression-related problems, which speeds up testing and deployment in Agile projects.

Regression testing and other types of testing

A software product has functional and non-functional features. Code changes can touch both types and so does regression testing.

Regression functional testing

Functional testing features customer requirements and business logic, as well as product specification and verifies whether the application works as expected. The purpose of regression testing in this case is to verify that recent changes haven’t destroyed or hindered the functional features already in place.

Regression non-functional testing

Regression testing may also be a part of non-functional testing efforts. The most common types of non-functional testing efforts requiring regression testing are:

• UI testing

UI is what users see when running an application. This type of testing involves verifying that the application menus, icons, bars, dialogue windows, etc. are displayed correctly. UI testing assures the application look and feel, which often predetermines the product popularity.

Regression UI testing may be required when the product functionality has grown, and multiple new UI elements may cause confusion among users. Striving to improve UX, product owners and BAs analyze the application and decide which old functions can be merged with the new ones, and which can be replaced altogether. Any changes involve code changes that call for regression testing.

This happened with the iOS mobile app we worked on. The app saw several releases, and at some point the app side bars no longer covered critical functionality added recently. So, the product owner decided to upgrade the bar functions. Then, regression testing uncovered that the UI got mixed up. Instead of performing key app functions, the side bar featured functions of low importance, while key functionality was unavailable at hand. 

• Compatibility testing

This type of testing checks how easy it is to use the product across various hardware and/or operation systems. Consequently, basic types of compatibility testing include cross-browser and cross-platform testing. Depending on the product, testing teams may perform compatibility regression testing for different configurations.

• Security testing

Security testing detects an app’s vulnerabilities for outside hackers and examines code quality from the security perspective. This is the most common approach to security testing. However, for web applications requiring high security level (medical applications, banking apps, etc.), security testing also features user authorization. These applications offer role-based access. A typical case is a hospital web application, where doctors have access to the data related to their work activities and nothing more. If the rights were changed for some role or some employee resigned, testing engineers run regression security testing to make sure the access rights for the rest of the workers remains intact.

• Localization and internationalization testing

These types of testing are typically performed when an application goes global. Localization testing verifies that the application behaves according to cultural norms accepted in the target region and correctly displays information in supported languages. Internationalization testing verifies that the application works correctly regardless of the region, language or cultural specifics.

Surprisingly, these types of testing may also involve regression testing. For instance, one of our products aimed to cover a large audience. Therefore, it supported multiple languages. A translation service provided relevant translations. When running functional regression testing, the testing team also paid attention to the correct displaying of the information in various languages. Sometimes, translations were erased as a result of development efforts, i.e. due to code changes. Thus, the team started to perform regression localization testing to verify that development efforts didn’t affect the existing non-functional features.

Summing up

The best way to cope with regression testing is to design an appropriate testing strategy. This strategy comprises of three key elements:

• Testing methods (proper ratio of manual and automated regression).
• Regression testing approach (distribution of full and partial regression in product delivery process).
• Quality regression testing suite. The suite may involve both functional and non-functional test cases, provided they cover the features that were changed during a given stage of product delivery process.

The specifics of the product governed by stakeholders’ needs make a cornerstone for choosing the right regression testing strategy. Another important factor is the product scale (small, medium and large). A good strategy for regression testing reduces testing time and effort and enhances the product quality, making it fully correspond to the customer’s needs.

About the Author

Tatiana Bessonova is Lead Functional Test Engineer with 14 years of experience in waterfall and agile projects. Experienced in 11 domains, Tatiana widely applies her excellent analytical skills and practical business approach to challenging projects in Healthcare, Manufacturing, Entertainment, Retail and more.