Feedback is often touted as the golden bullet, which will solve every software development woe. Hey, we certainly love it! But there are a couple of important aspects to feedback that need to be kept in mind, rather than simply declaring that ‘more feedback, more frequently’ is always the answer.
- not all feedback is equal
- feedback costs
That means we have to choose wisely where we gather feedback and what use we plan to make of it.
1) Not all feedback is equal
Feedback is not the same as prediction. Data does not tell us what to do – it simply indicates a problem. Your order system tells you that sales are down this month compared to the same period last year. You will certainly want to ask why. Perhaps you ring a customer and ask them why they haven’t bought your new upgrade this year. They will tell you their reasons.
These are two different pieces of feedback – one quantitative, one qualitative. Both tell you something, but neither tells you what to do. It’s not always obvious even that action needs to be taken. Maybe last year you ran an advertising campaign and saw a resulting spike in sales. You wouldn’t expect to see sales as high this year. Indeed, by fiddling with your product in response to an imagined problem, you might actually cause a worse one.
Much feedback, then, is about ensuring that you are clearing all the ‘noise’ that obscures the signal. And it is not always easy. The hard feedback of figures (sales, usage and registration, for example), do not lie, but they sometimes require interpretation. Falling sales may be catastrophic, but although the figures compel attention, they contain few clues on how to improve matters.
But – and it’s a big but - this kind of feedback is crucial. So crucial, that we recommend always making the ‘concept to cash’ feedback loop as tight and swift as possible.
That’s because subjective data and feedback can be misleading. People don’t always tell you the truth about what they like or dislike. Numerous studies, for example, show that people claim to want to eat healthily, but then go on to make unhealthy food choices. If you were a café-owner who used the first set of data to choose a variety of salads to put on your menu, you might find profits fell – while the burger bar next door sold hundreds more unhealthy meals every day.
It’s not just that people intentionally or unintentionally mislead you, but they may not know what they want. That’s why many of the feedback techniques now in use, including prototyping and test environments, help us to look at actual usage and behaviour (rather than claims).
When Monash University wanted to find out what prospective students thought of their course-finder system, they didn’t ask for opinions. Instead, they created paper prototypes of two different design routes. Students then interacted with these ‘paper computers’ to carry out a task. Although the facilitator later asked students which option they preferred, the most important information was gleaned from observing actual behaviour – the difficulty or ease with which students negotiated the system. The results led the team to choose the option which they themselves disliked from a design point of view, but which students had clearly found easier to use. More importantly, the exercise also threw up other problems – including that nearly a third of students were unable to complete the task because they could not understand where to find further information. No subjective set of questions would have elicited such high quality feedback.
Finally, if not all types of feedback are equal, neither are all sources. To return to our imaginary company with falling sales… What happens when you start calling other customers and they all say different things, perhaps opposing things? What about when you add in the Marketing Director’s opinion, the CFO’s and the CEO’s nephew’s… To whom do you listen?
Many organisations try to dodge this question – perhaps because they know what the answer ought to be, but are all too aware that this differs from what actually happens. All of us have encountered the HiPPO (highest-paid person’s opinion), but we have to work hard to make sure that pet likes and dislikes from senior management do not override the voice of the customer. And to ensure this, we need to make sure that we are working with real customers as closely as possible.
There are times when you don’t listen to customers. There are times when customers want your company to do something which you know will be unprofitable, or damaging in some way. On other occasions, lots of feedback risks diluting the vision (perhaps an idea so radical that no-one really gets it right now). Customers should not control the development process, but they should certainly inform it. Only customers can tell us what they are prepared to pay for. We ignore them at our peril.
Eric Ries tells a story in his book The Lean Start Up. A teenager was testing the product but she didn’t want to invite any friends to chat until she knew if the product was cool or not. Ries gave up on her and recruited another teenager … and the same thing happened. As Ries comments ruefully, ‘You start to see a pattern… no matter how stubborn you are’.
The problem is that for many organisations, contact with actual customers is rare. Instead you may have an ‘internal customer’ or perhaps a ‘proxy customer’. It doesn’t really matter whether you call this person the Product Owner, the Customer Representative or ‘Business Customer’. While it looks like a useful shortcut – the proxy customer is going to make all the prioritisation decisions and accept or reject the work – the fact is they are still not a real customer and therefore their feedback is of limited value. The team still has to get the product into the hands of real customers and users. There is no substitute for this.
This is not an excuse to do masses of up front exploration, however. As we’ve just discussed, no simulated or subjective feedback is as good as real data. So that means continuing to shorten the concept to cash loop as much as possible in order to get your product into the hands of real, paying customers.
Just to make the whole subject slightly more difficult, there’s one added complication. Types and sources of feedback are unequal – and so is the attention we pay to them. All of us – no matter how sophisticated and educated we think we are – suffer from cognitive bias. This is the effect that means we unconsciously give greater weight to evidence with chimes with our own views, and discount that which opposes them.
Rather than being delighted by a new input that will enable us to improve the project, we see feedback as leading to rework. How many times have you heard someone on your team groan, ‘I’ve got to do it all over again!’? If you’re honest, how many times have you felt that way yourself? Searching out feedback, especially unwelcome feedback, requires a significant culture shift for most organisations and individuals that far outweighs the difficulty of finding a real customer to join the team, instituting faster development cycles or setting up tests to observe user and customer behaviour.
2) Feedback Costs
Feedback is great. Brian Harry, a Microsoft Technical Fellow, enthuses, ‘Feedback helps you clarify your understanding. Feedback helps you see things in new ways. Feedback helps you correct your course. Feedback helps you learn. Feedback makes you and your work better. Whether you follow specific Agile practices or not, feedback early and often is a critical component of being more successful.’
Feedback underpins much of the Agile Manifesto. Acting on feedback is how we deliver iteratively, and opportunities for feedback are formalised in many Agile practices, from pair programming in XP, to demonstrations and retrospectives in Scrum.
Among all the discussion regarding how and when to embed feedback loops in software development there is sometimes a strange silence around an important aspect of feedback. It costs.
Before we are drowned by howls of protest, we should be quite clear that we don’t think this means you shouldn’t get feedback. It’s a bit like buying a house – a surveyor’s report costs you money, but it doesn’t cost as much as buying the house without one and then finding you’ve got subsidence, dry rot, rising damp and live woodworm. Nor, of course, is feedback just about avoiding disaster – it’s also about discovering what customers love, improving quality and increasing value.
But the fact that feedback costs does mean that you need to pay attention to the type of feedback you seek and how frequently you seek it. As we stressed above, feedback is often about knowing when you have enough – failing early and cheaply is often a better strategy than investing a great deal in ensuring that you have a winning formula.
It is always worth considering whether the feedback loop we have embedded provides a justified economic benefit.
Are you going to use the feedback you have gathered? Are you actually able to respond to it? Does gathering feedback in advance offer a saving over just launching the experiment and seeing if it sinks or floats?
Our point is that while the general rule – gain feedback early and often – remains valid, you have to use your individual judgement to answer what type of feedback you want and how often.
As Jurgen Appelo sensibly writes in Management 3.0, ‘there is a point at which it makes no sense anymore to further reduce the length of the feedback cycle. It may be great to reduce the Scrum sprint length from four weeks to one. But reducing it to one day might not be worth the trouble. At some point performance improvement levels off and does not outweigh the added costs of increased overhead and measurements’.
In the exploratory phase you want to find out a key piece of information, and you need to find it out as cheaply and fast as possible. Setting up a simulated environment may mean investing more than the feedback is worth. Instead we test using wireframes or paper. In the same way, the set up of some unit tests or acceptance criteria can prove more costly to maintain than beneficial. System-level tests can slow down the whole process. In such cases, it can actually be cheaper to fix a problem later in the process. There is no substitute for the individual or team’s intelligent judgement in such matters.
It is worth adding that although frequent feedback costs more, it does not necessarily cost more by the same order of magnitude. Don Reinertsen writes in The Principles of Product Development Flow: ‘A feedback loop that has a long time-constant can only smooth variation that has equally long time-constant. This means that we often need to embed fast time-constant feedback loops inside the slow ones. These fast-time constant feedback loops do not need a large dynamic range because cumulative variation is roughly proportional to the square root of time. Thus, it takes less resource margin to maintain control for fast feedback loops.”
Conclusion
Neither its cost nor it being obscured by noise is intended to stop you gathering feedback. But points are both intended to remind you that feedback, like any other tool, should be used appropriately. Here are the top ten points to remember when gathering feedback:
- Invest the minimum time, effort and money to answer your questions
- Feedback in and of itself does not tell you what to do, how to prioritise or provide a plan.
- Seek feedback from the marketplace rather than internal opinions or artificial results created from a simulated environment
- Feedback is often dependent on context – you don’t always need a strict double-blind clinical trial, but you should recognise your prior assumptions.
- Remember that behaviour matters more than opinion
- Speed is often more important than precision: a series of experiments sometimes find the answer faster than calculation
- As you gather feedback more often, costs per transaction reduce
- The more risky ‘getting it wrong is’, the more you need to seek feedback. It sounds obvious – but most people hide from risk.
- No tests or internal feedback should delay you seeking direct customer feedback
- You need the right attitude to feedback – embracing change, not fearing rework. All of us have confirmation biases causing us to seek opinions which agree with ours and reject those which disagree.
About the Author
Alex Adamopoulos is an executive with more than 25 years’ experience in global services organizations. He has extensive international experience with a deep understanding of culture, work, and life ethics especially in relation to establishing alignment and crossing cultural barriers. Over the years, Alex has brought know-how and practical business experience to companies that want to excel and compete globally. With a focus on performance measurement, business value and bottom-line profitability, Alex has successfully applied working models and practices to accelerate the solutions and strategies of companies to drive results.
