The Resurrection of Product Risk Analysis

Product risk analysis (PRA) is not only useful in testing but is also applicable during the various phases of sequential or agile system development.

Although it seems that interest in the PRA is decreasing, I see an increasing number of testers attending my presentations and workshops that deal with PRA and its usefulness in testing IT systems. That, to me, indicates that PRA is still alive!

One reason PRA is dying is said to be that the outcome of PRA will not be used during the project so it is of no use to perform one. To refute that sentiment, I will introduce a different application of PRA that elevates it from project level to domain level.

Risk management

Many management activities are concerned with identifying and controlling risks. Risks have both external and internal causes. A few examples of external risk-bearing factors are fraud, infrastructure, legislation, competition, terrorism, and political and social developments. There is also finance-focused risk management, specifically practised by financial institutions.

The risk management described here aims at identifying risks connected with the development and implementation of an information system. The risk is the probability that development and implementation will cause measurable economic damage to the company, and perhaps also will cause other, less measurable damage. Damage could be that the results of a project are less favourable than expected or that the organisation will suffer direct or indirect loss. Risks extend beyond IT and concern the business, too. Examples of damage that may occur are:

• Loss of market share
• Compensation claims
• Damage to image and reputation, for example through corrective measures
• Increase in aftercare and higher maintainability costs
• Higher than planned costs of internal provisions
• Extra staffing needs

Product risk vs. project risk

All of the risks involved in a project are divided into two categories: project risks and product risks.

This distinction is important, as it concerns two entirely different types of risk, each requiring a different approach to control them.

Project risks

Project risks relate to the management of the process and can be broken down into categories of IT project risks and business project risks.

IT project risks are linked with the conditions required for developing the information system: the availability of enough people with the right expertise, a stable environment, etc.

The business project risks relate to the time and budget allocated to the project, and the project manager establishes measures to limit them. The project risks and the limitation measures are included in the project plan. In other words, the project manager is able to define measures to mitigate project risks.

Product risks

Testers usually focus on this category. They look at the functional and non-functional requirements or absence of them, at the stability and complexity of the information system, at the quality of the documentation, and at the quality of the coding. The test project can cover all of these risks with appropriate tests. However, the testers cannot mitigate these risks. They can only demonstrate that these risks are properly addressed in the software.

PRA as element of risk and requirement-based testing

Product risks arise from the use of a system by a company or organization. Stakeholders are all those people and/or departments that have a special interest in the correct functioning of an information system (Fig. 1). They are the people for whom the test project is conducted. To reach the most accurate risk analysis, the input of stakeholders is essential.

Fig.1: Stakeholders and their interests.

The stakeholders and test manager perform a PRA together. This PRA may take place in different ways. In many cases, the stakeholders will be interviewed individually. While this is an accepted method that may yield substantial results, it’s time-consuming. The statements of one stakeholder may influence other stakeholders and may require additional interviews. Another approach that’s considered more efficient is the product risk-analysis workshop (PRAW). During a PRAW, the stakeholders jointly identify the product risks. A moderator supervises to make things go smoothly. This workshop can use various techniques like a brown-paper session, a failure-mode effect analysis (FMEA), or a mind map.

During a PRA, the stakeholders in an information system define the product risks and assign priorities. A PRA consists of steps:

• Risk elicitation: the exercise to define all possible risks related to a system
• Risk balancing: determination of the relative weight of the risks

Risk prioritisation

After the determination of the risks, the next equally important step is to assign a priority to each of the risks. These ranks help determine the order in which the tests will be designed and executed. The most important risks go first. The priority of product risks is indicated using the acronym MoSCoW:

Must test: The aspect must be tested, otherwise no acceptance.

Should test: It’s important to test this functionality but the test can be skipped after consulting the stakeholders.

Could test: If we have time, we will test this.

Won’t test: We will not test this.

The resulting “risk priority” of a risk amongst others consists of the importance of the feature to the organization, the potential damage it can cause if poorly handled, and rules and regulations from outside the organization.

Matching of product risks and requirements

Product risks and requirements are bound inextricably. In short, defining one or more requirements must mitigate each risk. We can use the outcome of the PRA as a first test in which we check whether the collection of defined requirements is complete, although we do not yet test whether or not the requirements are correctly defined.

For the first test, we compare defined, identified product risks with the requirements. In perfect situations, a product risk exists for one or more of the defined requirements but in practice, the situation is not always so ideal. Often, we find differences in the two lists. We may identify a product risk may with no related requirements. That’s when we must consider if the risk really exists and if it does, we must add a requirement. If the risk does not exist, we must remove the risk from the risk list.

In other situations, we may find a defined requirement for which we identify no related risk. In that situation, we must determine whether or not a risk is present and correspondingly either remove the requirement or add the risk to the list and assign it a priority. After that, we must check that requirements related to a risk indeed address the risk completely. A requirement that states input validation must take place on one of the data entry screens may not be sufficient to cover the risk of a crashing system as result of invalid input.

For this process to work correctly, the risks and requirements should have been identified and defined in different sessions and preferably not by the same people. Prior knowledge of the requirements will bias the participants during the risk-analysis process.

Requirements and priorities

After we have checked all requirements against product risks and we have listed all necessary requirements and risks, the requirements will carry two types of priority: the risk priority as it was identified during the PRA and functional priority. The functional priority indicates the importance of the functionality to the stakeholders. The indicators for functional priority follow a similar MoSCoW pattern:

Must have: Cannot be missed.

Should have: Almost essential; nice to have this feature if at all possible.

Could have: Not essential but we could have this if it does not affect anything else.

Would have: Not essential now but we would like to have it in the future.

The risk priority and the functional priority are not directly related. They can be the same but do not have to be. A feature of a system that seems relatively unimportant may become a high risk after it is implemented!

Fig. 2: Functional priority is not always equal to risk priority.

Risks and their uses during the test process

The PRA is a fundamental element of risk and requirement-based testing (RRBT). In this test-management approach, we assign priorities to requirements based on the risks related to each requirement. During the test process, the software that implements the requirements with the highest risks is tested first with more thorough test-design techniques.

During the test process, the testers consider the risk priority in many situations – for example to estimate the test effort, to plan the test activities, to report on findings, and as a basis for their test reports.

Test strategy

The test manager will define in the test strategy, how the system will be tested. In this, the risk priority will play an important role: high-risk requirements ask for a heavier test-design technique than lower-risk requirements do. The test strategy also describes the acceptance criteria.

Estimation and planning

If the budget is too small to test everything (which is usually the case), the test manager, in consultation with stakeholders, will identify which parts must be tested and which may eventually be skipped. During the test project, the high-risk requirements are tested first.

Progress management

During progress management, the test manager will monitor whether the desired quality is achieved and if that happens within the available test budget. To determine the quality, the test manager examines to what extent the product risks are covered by the quality of the system under test.

Issue management

With each finding, the tester records which product risk is affected. The test manager and steering group determine, based on these data, whether or not a finding will be solved within the current release, the next release, or not at all.

Reporting and advice

In test-level and test-end reports, the test manager will relay the final status of the test project to the stakeholders and determine whether the system under test meets the acceptance criteria as laid down in the test strategy.

From project to domain: from PRA to domain product risk analysis

If we look at the product risks in more detail, we see that multiple PRAs within a single domain will lead to similar outcomes, with small differences based on the specific project at hand. It seems that there is a lot of duplication of effort.

However, if we turn to analysing the product risks at domain level and hand the PRA results to any project starting within that domain, we can use the predefined risk list as a guideline to determine what specific risks we might need to add and what weights to attribute to the different risks. In doing so, we are less liable to omit important risks or starve our resources by doing too much redundant work. Furthermore, removing the repetitive and more or less boring work will speed up the PRA process.

How to do domain product risk analysis

To do domain product risk analysis (DPRA), we look at the bigger picture and try to produce a result that is reusable. Investing additional energy to get this analysis right will therefore pay us back as the result is reused and kept up to date by subsequent projects. We might therefore split the domain product risk into a number of sessions and try to get the best representatives of each stakeholder to participate, preferably at the program level instead of the project level.

In many ways, a DPRA session resembles a PRA session but since we are no longer dealing with a single project with a specific impact, we need to add steps to the beginning of the process:

• Identify the critical process chains within or impacted by the domain.
• Identify the critical activities/tasks of your domain.
• Identify the critical business processes support by your domain.

After these steps, we perform a PRA session relating the risks to the activities, processes, and chains that you have identified. Although this sounds easy, it requires the contributors to reflect in a more abstract way on their work.

Since we perform the DPRA to come to a kind of super PRA, we can reuse some best-in-class PRAs of projects in this domain as input, if they are available.

Applying the DPRA to your project

If the domain product risks have been identified, they need to be applied as soon as a project is started. To do this we identify the following steps:

• Identify what domain(s) your project is part of.
• Identify the critical chains/activities/business processes your project impacts.
• Copy the risks associated with these from the domain PRAs.
• Evaluate this risk list and take it as a basis for your risk analysis.

Once you have this list of initial risks, you have to tailor it to the specifics of your project. To do this, take these three steps:

• Identify whether the project introduces new critical activities to the domain.
• Add new product risks induced by the critical success factor of your project.
• Investigate whether the technical complexity of the (addition to the) system justifies a reclassification of the risk impacts.

Depending on the size of the project or the impact of the change, you can deal with these questions in a desk session or require a PRA session. Since most of the work has already been done in the domain PRAs, tailoring the risk list based on the considerations above becomes easier and less time-consuming.

Maintain the DPRA risk list

As soon as the DPRA is finished and the first domain product risk list has been released, there is a risk of obsolescence. Product risks and priorities may change over time and the risk list must reflect those changes.

The input for this maintenance will come from project feedback: the results of each PRA at the start of a project are sent to the maintenance department. There, the risk list is compared to the domain product risk list. Each deviation is discussed with the stakeholders and the domain product risk list is adjusted as required.

PRA and agile system development

Now that we have brought PRA to the domain level, we can apply the outcome of the analysis not only to testing but to development, too. If we know the risks to our organisation in a given domain, we can determine which requirements are important within that domain. And if we know which requirements are important (the ones that are related to the highest risks) and which are less important, we can use this information when we start a new development project or maintenance program.

When requirements have a functional priority (from “must have” to “could have”) and a risk priority (from “must test” to “could test”), we can combine these to determine the overall priority. Requirements (known in agile development as user stories) with both a high functional priority and a high risk priority will be ranked most important and will be developed first. The ones with a high functional priority and a medium risk priority will come next, and so on. If we apply the matrix depicted in Fig. 2, we can assign a combined priority for each requirement.

Fig. 3: The requirement/risk matrix.

This rank of requirements or user stories, which consists of a combination of risk and business value, is applied to the product backlog. At the start of each sprint, the product owner will use this ranking to select the user stories that will be elaborated in the upcoming sprint.

The risk priority assigned to a requirement or user story also impacts the velocity and thus the planning of this sprint. High-risk requirements will require more effort from both development and test focus. After all, compared to lower risk areas, the tester will apply heavier test-design techniques and will spend more time to test high-risk requirements and user stories.

Fig. 4: The impact of risk on the sprint backlog.

From risk and requirement-based testing to risk and requirement-based development

The PRA and the product risk list that it produces are of vital importance to a test project: they allow the stakeholders to define acceptance criteria and they support the test manager in defining the scope and planning the project.

From experience, we know that several PRAs performed within one domain often lead to the same product risk list. By defining the product risks for that domain, the time spent on performing a PRA for any new project will decline, which will improve the efficiency of the test process.

However, we can bring down the overall development cost even more when we perform a PRA at the level of the business domain and use its results as a guide within the projects. By defining the product risks for a business domain, less time must be spent on the execution of a separate PRAs for each project within that domain and the efficiency of the test process will improve.

This domain risk analysis (DRA) has even more advantages. Because the product risks are known before a project starts, the product owner can decide whether the efforts to implement and test the requirement outweigh the usefulness of that requirement given the risk that is related to that requirement.

This way, risk-management techniques translate from the test process to the entire development process, from risk and requirement-based testing to risk and requirement-based development.

About the Author

Chris C. Schotanus is Principal IT Consultant at CGI, a global IT and business process services provider with 68,000 professionals delivering high-quality business consulting, systems integration and managed services. Chris has more than thirty years of experience in IT, twenty of which were spent in the field of testing, test management, setting up test organisations and test automation. He advises organisations regarding test policy and test organisation, defines test process implementations and performs test process assessments, both in the Netherlands and internationally. Main areas of attention are agile and DevOps development, risk management, (Acceptance) Test Driven Development and SOA governance. He is one of the founders of TestFrame, CGI’s method of structured testing and author of several books and publications on test policy, test methods and (test) process improvement. He can be contacted via chris.schotanus@cgi.com