Data-Driven Decision Making – Product Management with Hypotheses

The Data-Driven Decision Making Series provides an overview of how the three main activities in the software delivery - Product Management, Development and Operations - can be supported by data-driven decision making.

Introduction

Software product delivery organizations deliver complex software systems on an evermore frequent basis. The main activities involved in the software delivery are Product Management, Development and Operations (by this we really mean activities as opposed to separate siloed departments that we do not recommend). In each of the activities many decisions have to be made fast to advance the delivery. In Product Management, the decisions are about feature prioritization. In Development, it is about the efficiency of the development process. And in Operations, it is about reliability.

The decisions can be made based on the experience of the team members. Additionally, the decisions can be made based on data. This should lead to a more objective and transparent decision making process. Especially with the increasing speed of the delivery and the growing number of delivery teams, an organization’s ability to be transparent is an important means for everyone’s continuous alignment without time-consuming synchronization meetings.

In this article, we explore how the activities of Product Management can be supported by data from Hypotheses and how the data can be used for rapid data-driven decision making. This, in turn, leads to increased transparency and decreased politicization of the product delivery organization, ultimately supporting better business results such as user engagement with the software and accrued revenue.

We report on the application of Hypotheses in Product Management at Siemens Healthineers in a large-scale distributed software delivery organization consisting of 16 software delivery teams located in thee countries.

Process Indicators, Not People KPIs

In order to steer Product Management in a data-driven way, we need to have a way of expressing the main activities in Product Management using data. That data needs to be treated as Process Indicators of what is going on, rather than as People Key Performance Indicators (KPIs) used for people evaluation. This is important because if used for people evaluation, the people may be inclined to tweak the data to be evaluated in favorable terms.

It is important that this approach to the data being treated as Process Indicators instead of people evaluation KPIs be set by the leadership of the product delivery organization in order to achieve unskewed data quality and data evaluation.

Hypotheses

One of the central questions in Product Management is "what to build?" This is very important to get right as building software features that, once ready, do not get used by the customers is a total waste.

In order to approach the question of "what to build?" product delivery teams run small experiments to explore the customer needs, ideally in Production. Each experiment needs associated measurements that are used to either confirm or disprove initial assumptions.

This process is the subject of the Hypothesis Driven Development (HDD). It is well-described in How to Implement Hypothesis-Driven Development. In essence, an experiment is called Hypothesis in HDD and is described using a <Capability> / <Outcome> / <Measurable Signal> notation:

Hypothesis:
We believe that this <Capability>
Will result in this customer <Outcome>
We will know we have succeeded when we see this <Measurable Signal> in production

The definition of Hypothesis for a feature is done before the feature implementation begins. A product delivery team declares which <Capability> they want to put into the product to achieve a specific customer <Outcome>. The customer <Outcome> becomes evident when a defined <Measurable Signal> becomes visible in production.

With that the product delivery teams define upfront how they envision the features being used by customers in production. During the feature development, the teams additionally implement <Measurable Signals> to create instrumentation necessary to see how the features are being used in production. Finally, once the features have been deployed to production, the teams evaluate the actual <Measurable Signals> to understand whether the Hypothesis turned out to be true of false (Build → Measure → Learn).

After that, the process is repeated; based on the results of the first Hypothesis, the second one is formulated, implemented and measured, and so forth.

In our experience, the entire process also moves a product delivery team from being "just a feature factory" towards becoming responsible for delivering software that is actually being used by customers in production.

That is, the focus of the product delivery teams is set on the value they provide to the customers, as opposed to just counting features delivered to production.

A team that consistently works with Hypotheses navigates the customer problem domain effectively by optimizing for customer usage of the software based on data from production (automated feedback loop; every release is a scientific experiment). A team that does not work with Hypotheses just produces features and does not measure the feature usage by customers in production.

From Projects to Products

Today, many organizations transition from projects to products. Introducing Hypotheses can support the transition. This is because in projects, the requirement engineering is scoped for Development, where the project often ends, thus excluding Release and Operations aspects.

Stating a Hypothesis with a <Measurable Signal> as part of the requirement engineering process necessarily includes Release and Operations aspects (Dev + Ops = DevOps, so to speak). This way, the requirement engineering includes the evidence of user behavior, which is a step towards products and away from projects.

Additional Hypotheses-related aspects that help transit from projects to products are:

• As the Hypotheses cover business success criteria to be achieved, the development team is becoming more business-driven without the need of a project.
• The development team can run with less management involvement under the guidance of <Outcome> being measured using <Measurable Signals>. The team can do many releases on their own, which used to require separate projects.
• The current state of the team is transparent with the values of <Measurable Signals>, which also guide the team decision making on the way to the <Outcome>.

Enablement

With the Indicators Framework defined, it was clear to us that its introduction to the organization of 16 dev teams could only be effective if sufficient support could be provided to the teams.

For the definition of Hypotheses we expanded our Business Feature template to include the <Capability>, <Outcome> and <Measurable Signals> fields.

We ran small workshops of 30 - 60 minutes with the teams where we took one requirement the team was about to take up and turned it into a Hypothesis. Later, once the requirement was deployed to production, we met the team and evaluated the <Measurable Signals>.

Adoption

We introduced the suggested Indicators Framework to an organization of 16 dev teams working on "teamplay" - a global digital service from the healthcare domain (more about "teamplay" can be learned at Adopting Continuous Delivery at teamplay, Siemens Healthineers). The teams got quite interested in Hypotheses right from the start.

It was easy for the teams to grasp the Hypotheses definition process. The discussions during the feature Hypotheses definition proved to be very valuable for further requirement engineering (using BDD). It was possible to hammer out the scope of the features very early on. A Hypothesis definition at this point created a nicely defined boundary, in which detailed user stories can be created later on.

An example Hypothesis definition from one of teamplay User Administration is:  

When implemented, the first Measurable Signal values showed that it will take time until the usage of the feature increases to the point that was specified in the Hypothesis. However, there was qualitative feedback from Sales that customer onboarding using user invitations became significantly easier (unexpected qualitative Measurable Signal). Based on that feedback, the team can see over time whether the Hypothesis definition should be changed to reflect the usefulness of the feature to the customers as reported by the customers.

Another example Hypothesis from teamplay Data Access Management is:

The implementation of measurable signals varied by team. A few teams implemented the measurable signals in production. With the planned increase of the frequency of team releases, we think that the teams will increasingly adopt the implementation of the measurable signals.

One of the teams had an interesting experience with Hypotheses. After a Hypothesis definition for a feature, the team started implementation. During the implementation, they encountered significant limitations in an external framework used. The limitations were so severe that it became clear that the <Capability> could not be implemented as intended and, therefore, the <Outcome> could not be achieved. The team started looking for another external framework.

Soon, we will be introducing a "teamplay Service Standard" - a short list of topics important to us in a service. Two topics there, will concern Hypotheses: "Establish data-driven prioritization for a service" and "Define what success looks like and iterate towards it". With that, we are going to give the Hypotheses an additional push in the organization.

Prioritization

Our teams need more experience with Hypotheses in order to consistently use the data at hand as an input for prioritization. The data comes in different forms:

• Positively / negatively tested Hypotheses
• Unexpected insights from Measurable Signals

Now that the data is available, it needs to be taken into account by the dev teams, and especially product owners, to make the best prioritization decisions. The prioritization trade-offs are:

• Invest in features to increase product effectiveness and / or
• Invest in development efficiency and / or
• Invest in service reliability

Summary

In summary, if a team optimizes their Product Management Process using Hypotheses, then the team is able to gradually optimize their ways of working in a data-driven way so that over time the team can achieve a state where they build features evidently being used by the users.

Hypotheses help depoliticize and enable transparency in the decision making process of the software delivery organization. Finally, it supports the organization in driving better business results, such as user engagement with the software and revenue.

This article is part of the Data-Driven Decision Making for Software Product Delivery Organizations Series. The Series provides an overview of how the three main activities in the software delivery - Product Management, Development and Operations - can be supported by data-driven decision making. Future articles will shed light on data-driven decision making in Development, Operations and combinations of data-driven decision making in Product Management, Development and Operations.

Acknowledgements

Thanks go to the entire team at "teamplay" for introducing and adopting the methods from this article.

About the Author

Vladyslav Ukis graduated in Computer Science from the University of Erlangen-Nuremberg, Germany and, later, from the University of Manchester, UK. He joined Siemens Healthineers after each graduation and has been working on Software Architecture, Enterprise Architecture, Innovation Management, Private and Public Cloud Computing, Team Management and Engineering Management. In recent years, he has been driving the Continuous Delivery and DevOps Transformation in the Siemens Healthineers Digital Ecosystem Platform and Applications - "teamplay".

The Data-Driven Decision Making Series provides an overview of how the three main activities in the software delivery - Product Management, Development and Operations - can be supported by data-driven decision making.