Lean Start-Up, and How It Almost Killed Our Company

Since Eric Ries published his 2011 book – The Lean Start-Up – applying his experience with IMVU to business innovation, the idea has grown in popularity as a methodology. In fact, it has ceased to focus on the world of start-ups and has become the development methodology of choice for larger companies looking to improve their innovation success rate.

The MVP is almost universally accepted within software development as an aim, while ‘customer validation’ appears in every plan, presentation and project review.

As with many methodologies, as Lean Start-up has grown in acceptance and adoption, many of its subtleties have been lost. Rather than analysing the specific context in which it works best, ideas have hardened into orthodoxy as part of a solution intended to support a consultancy.

That consultancy exists because despite espousing the lean start-up method, very few companies practise it effectively. This article is not, however, a critique of companies doing lean innovation badly. Rather than calling for greater adherence to some original purity of method, I would say that the adoption of Lean Start-Up as a method (as distinct from the adoption of common-sense principles from within it), fails to take account of the limited conditions in which it is most appropriate. This holds true for the related approach of ‘Little Bets’, as described by Peter Sims in his 2013 book or Michael Schrage’s The Innovator’s Hypothesis.

The fault does not lie in the principles of Lean Start-Up, but in their application as a universal recipe to innovation success. Simple solutions are tempting – but they are rarely effective. I say this with more humility since it is a trap into which I, and my fellow founders, fell with our start-up, Gamevy. Let me tell you our story.

From Blackjack Attack to BornLucky Gameshows

The three founders of Gamevy had met researching and writing a series of books on Agile and Lean practices. Naturally, when it came to our own start-up, our mantra was to get the product out in front of customers. Find real data. Then adapt. Fail fast, fail cheap. From planning using the Lean Canvas to setting up a series of tests before we wrote a line of code, we tried to keep to a tight Build, Measure, Learn loop.

Gamevy had a clear vision. We wanted to add more fun into real money games, (real money, being the somewhat disingenuous way the gambling market likes to portray itself). In particular we wanted to build a game that felt like a TV gameshow, where combining a particular skill (answering trivia questions) with a luck/ chance mechanic gave you a shot at winning a big, jackpot prize.

The main business models were pleasingly simple – either a multiplayer version in which contestants wagered their stake against each other and we raked the winner, or a single-player version in which we acted as the house. Unfortunately, both were a form of gambling, a highly regulated industry in the UK. In either case we would need a gambling license. Within the strict rules of that regulation, we would not be able even to test a game with real money until we had a license in place.

We had created a series of playable paper prototypes. We looked at their possible value and profitability, how fun and playable they were, and how easy to build or market they might be. We took several of these to ‘play-testing’, including a memorable weekend at a fantasy convention, playtesting for 24 hours with hundreds of convention-goers all wearing corsets…

One game was the clear winner – a game we called Blackjack Attack.

At the time, our main options seemed rather like this:

What would you choose?

Be honest now, which sounds like the route the Lean Start-Up method would encourage you choose?

We decided to go fremium first.

The Increment Trap

We had two key questions – will people play the game and will people pay for it? Of course, we had some ideas about what we wanted the repeat play and retention metrics to look like and what we would deem a ‘success’ and what a ‘failure’.

Getting into the Facebook store took longer and was more costly than we had anticipated. In order to test the game with a statistically significant volume of customers and to charge real money, we were driving a bunch of requirements that we knew were not where we really wanted to be in the long term.

Once we were fully launched and we started getting feedback, it wasn’t perfectly clear.

Were people playing the game? Yes! Our repeat play figure was a relatively healthy 10%, although we suffered huge drop off during the first game.

Were people paying? Well, some people were. We were above the limit we’d set ourselves as ‘FAIL’, but it wasn’t enough people and not as much as we’d like.

That was all OK! We were getting feedback and we knew what we needed to do – improve!

So we began to see if we could up those numbers, refine the product, improve the drop-off rates, convert more people, see what referral and sharing metrics might look like, if the business model of gaming could be made to work…

Our changes worked. We managed to make improvements to all of our key metrics – but none of them were going fast enough. Looking forward we could see that it would be perfectly possible to spend the next year and all our resources improving the game. Would it ever do well enough?

Four months after we’d begun work on it and 1 month after launch, we bit the bullet. We needed to go back and take the other choice – the much bigger bet of regulation – and we needed to do it with a different product, one which accepted a different series of trade-offs.

20/20 Vision

With hindsight – that wonderful vision – we realised that the learning we had bought with such hard work was not that valuable to us after all. Rather than focusing on the product, we should have focused on the particular market we wanted to be in. The differing business models of the fremium and real money gaming markets turned out to be crucial. Interestingly, a traditional business plan might have helped us focus on that far better than Lean Start-Up.

We were extremely lucky that our costly experiment did not cost us too dear, that we stopped it before it was our only experiment.

In the last year, Gamevy has gone on to gain our gambling license and build two games (both very imperfect, still). In the last month, we have shifted our focus away from our eventual goal (be an operator direct to consumers) and towards being a supplier to a couple of specific partners. Doing this has meant delaying our launch, making the prospect of consumer feedback more distant as we redo work to integrate with a partner platform. We have, painfully, decided on a trade-off that lets us stay alive longer in the hope that it makes our eventual success more likely. This is not failing cheap and failing fast. Only time will tell if it’s the right move.

It has made us think hard about Lean Start-Up and some of the considerations that might have got us here faster if we’d thought about it differently.

When the MVP is not what you want

This is a popular diagram used by Spotify to explain the concept of an MVP. The problem is that like many simplifications, it offers as much of a block to real understanding as it does clarity. The bottom ‘correct’ How-to progression shows 5 ‘solutions’ to the problem of transport. Yet, of course, they are not really 5 different versions of the same thing – they are 5 separate products, each with specific benefits and requirements, each needing different learning validation.

If you want to build a car, then the learning you gain from dedicated skate-boarders is not going to help you very much. In fact, as you listen to your customers and implement features about carbon composite materials and epoxy fibre reinforcement to increase pop, you are going to get further away from your goal of creating a car, not closer.

It’s just an illustration! No-one’s meant to take it literally!

Yes, we know, but actually, we think the diagram rather neatly illustrates why Lean Start-Up doesn’t help you much where barriers to entry really exist. Where there is something that makes building your end-goal difficult (like the complexity of a car), then ‘steps’ along the way can often turn out to be not the right steps at all but rather choices that lead you in a different direction.

Let’s take Gamevy’s example. We knew that real-money gaming had formidable barriers to entry. Rather than facing those, we decided to try to validate the idea in an area with lower barriers to entry – fremium gaming. But in doing so we discovered several things:

1. the learning was not that useful – these were not our eventual customers. They were essentially telling us about skateboard features when we wanted car enthusiasts. Even if some social gamers are also real money gamers, we were talking to them within the context of a social gaming market. It turned out that this really mattered when it came to learning.
2. we discovered new barriers to entry existed that we had not foreseen. The costs of running the game live – technical, marketing and operations – were much higher than anticipated. Since this wasn’t the ‘real’ market we wanted, these were wasteful costs.
3. The MVP was expanding. In order to try and get the validated learning – that is, would people PAY, for the game – we needed to build an entire fremium economy around it – referral and sharing mechanisms, sales, alerts and notifications…

The Barriers to Entry

Context matters – the higher the barrier to entry, in general, the less minimal an MVP will be. Trying to take smaller steps can often lead to errors like those Gamevy made.

As my co-founder Paul Dolman-Darrall once commented, ‘little bets’ or MVP steps work best in places that have lower barriers to entry:

• Disruptive markets where the barrier to entry will be attacked. We see this in examples such as Uber – attacking the traditional barrier to entry held by cab-drivers, or Air BnB, attacking the traditional hotelier market. They took a calculated risk that regulation either could or would not be applied to them. In many industries – gambling, healthcare etc – this risk would simply be too high.
• Industries with naturally low barriers to market. These are typically commodity or service billing industries, which offer a respectable living but are unlikely to lead to the asymmetric payoff associated with successful start-ups.
• Incremental improvements where the barrier to entry has already been paid. For large companies trying to innovate on an existing product or service – this is the most likely scenario and explains why small bets and the concept of the internal lean start-up works so well for larger companies.
• New industries where no barrier to market exists at all – arguably this is exactly where Eric Ries’s company IMVU was placed – 3D avatars were entirely new within an online social market that was itself in its infancy.

Markets with high regulation or incumbent competitors often require the opposite strategy – big bets. As Steve Jobs called it, moments in the life of a business when you ‘bet the farm’.

As for Gamevy and our decision to bite the bullet and gain our Gambling License, there is often no way around these big bets. If we had failed to get the license the company would have closed down. There remain dozens of failure points in our near future – but in our case the big bet was the only one that mattered.

Step-by-step versus ‘little bets’

Pixar is a frequently quoted example of little bets (including within Peter Sims’ book). There is an important differentiation between enforced step-by-step development and a deliberate ‘little bet’ or ‘MVP’ strategy. It is a difference that is frequently elided, but I would go so far as to claim that prior to their first success, Pixar’s actual strategy was to take the biggest bets that they could feasibly manage.

Pre Toy Story, for example, the team made several computer-animated sequences for adverts. Although such projects may well have provided valuable technical learning, they were a product of necessity – providing a small revenue stream to keep Pixar afloat. At no point did the team decide to pivot and chase such work; they were eager to be able to reject it. Similarly, the shorts that the team created (including Tin Toy, the genesis for Toy Story), were made to show off the company’s hardware capabilities and to gain a business customer. Pixar was not trying to build a consumer base, it was offering a shop window to gain a partner. As it successfully did when commissioned to make Toy Story for Disney.

Today, the available technology of YouTube, Vimeo and crowd-funding might encourage a young animation company to go direct to consumers. Who knows if Pixar would have made the same choices today? Who knows if they would have been more or less successful in doing so? Pixar accepted some painful trade-offs in order to survive.

I suggest that if in the 80s and 90s someone even richer than Steve Jobs had offered Pixar the opportunity to work ONLY on their own full-length animated feature film with no need to worry about revenue or a distributor, they would have jumped at the opportunity to turn down the adverts, the shorts and Disney’s heavy-handed oversight. It is only with hindsight that we perceive decisions that at the time appeared to be compromises as small steps along the way to animation dominance.

Characterising decisions made as ‘little bets’ or a series of MVPs, when in fact they were enforced by necessity (need for revenue, inability to acquire distribution or customers without a partner etc) offers a misleading picture both for start-ups and larger companies.

MVP vs MDP, The Great Start-Up Experiment

Start-ups work rather differently to how large companies run innovation projects. A big company with a portfolio of innovation products is the perfect place to implement the ‘little bets’ strategy – investing more in this seemingly-successful idea, killing off this poor one. For start-ups a poor innovation product is its only product. When it fails or delivers only a small revenue stream, there are a limited number of times that the start-up can pivot, or kill an idea and start again.

Each independent start-up is its own ‘little bet’ – the market gains the benefit of the few that succeed, but that’s not much consolation for the 80% of start-ups that close within the first 3 years. Those that succeed will have a mixture of good ideas, good management, good funding and luck. The Lean Start-Up method fails to say much about the equally important, latter two.

Since working in a start-up often means significant sacrifices, individuals would certainly rather fail fast and then move on to something more successful. But what happens when an MVP actually makes failure more likely?

In the IMVU story, there is little real cost to launching a buggy, poor product. Eric Ries humorously mentions the personal cost to his reputation as a technologist, but not to the product itself. This pre-supposes that there is a large pool of potential customers, that the costs of acquiring them will not outweigh the value of their feedback or revenue if acquired later, and that negative feedback will not have an impact on the product or company’s future.

In fact such circumstances are rare. In many industries, an MVP either delivers little learning or offers significant risk – launching a buggy drug or financial instrument might shut down your company. Instead, the company needs to focus on delivering the Minimum Desirable Product or MDP.

Teasing out the difference between viable and desirable can prove extremely difficult. Generally customers are not able to tell you what innovation they find desirable until they see it. And seeing it means that you have to do the work up front before validation – testing concepts and ideas will not always provide reliable feedback. As any marketer will tell you ‘would you pay for this?’ is a meaningless question compared to seeing what customers actually put their hands in their pockets for.

Take a non-technical example… Publishers will often say longingly that they are looking for ‘the next J. K. Rowling’. And if you ask children ‘what kind of book do you want to read next?’ they may tell you that they want something ‘just like Harry Potter’. Does that mean that publishers should be commissioning dozens of series about a boy wizard set in a boarding school? No. Although there are plenty of copy-cat books which seem to suggest publishers have not really figured this out. The disappointing sales of such series mean publishers are paradoxically LESS likely to take a chance on a new author with a kooky idea.

Novels are not MVPs. Attempts to make them so – outlines, single chapters and serialisation – are rarely successful. Few readers want to try a chapter knowing it might be months before they get the rest of the novel and any fiction author will tell you the first chapter will have many rewrites by the time the last chapter appears. For authors, it is hard work to get agents or publishers to read a manuscript and not much easier to get readers to your blog. There is also a cost to offering up first drafts: your limited group of readers will be unlikely to read your next draft and worse, they may leave a review telling everyone else how rubbish you are.

These risks hold equally true for most start-ups: a limited pool of customers who are potentially expensive or difficult to acquire and whose negative feedback is likely to be overheard and have a major impact. Eric Ries frequently stresses that early launches, ones expected to fail, should not have any press or marketing – but it’s a strangely out-dated idea to believe that only advertising or PR impacts on a brand’s reputation.

The Minimum Desirable Product – one which at least a sub-set of customers will love – is often not very minimal at all. Apple is the classic example – a company that focuses on ensuring the product is loved – even if that means completely redesigning something in order to achieve an aesthetic improvement (as occurred with the i-phone). Most start-ups have only one shot at such an idea.

At Gamevy today, we build game prototypes for internal testing – they help us refine the mechanics of the game, including how easy or hard it feels and what the win rate / sense of agency might be. But we do not put them live – partly because without official and expensive testing via an external party we would be violating our gambling license to do so and partly because we can’t risk lowering our repeat play and retention metrics by offering our (or our partner’s) expensively acquired customers an inferior game. Perhaps one day – in our successful future – we might implement a labs or beta environment where a few customers can play these prototypes for real and where we undertake the external testing early as part of our initial development cost.

At the moment, the cost of getting to the MDP is less than the risk of the MVP.

In such cases if all the Lean Start-Up method contributes is to say ‘make sure you don’t add in anything those customers won’t care about’, then it is not especially useful. You will still have to make a series of judgement calls on what is necessary, what is desirable, and what is a ‘nice-to-have’. The only validation you are likely to get are from informal focus groups or existing ‘friendly’ customers – neither of which offers a guaranteed guide to market performance.

Conclusion

In 2004, when Eric Ries co-founded IMVU, it was still relatively cheap to acquire customers online. Those costs have since increased significantly as the online environment matches other media in marketing budget requirements. This makes parts of the model around customer acquisition rather like turning for advice to the goldrush pioneers when nowadays the mining companies have moved in.

The insights of Lean Start-up are still valuable – but their best application is in larger companies looking for a more effective way to manage their innovation portfolio. Where customer acquisition is the ‘problem’ of a different department or asks for transition and conversion of existing customers, a single-minded focus on product development and validating learning may be exactly what is required. Similarly when the company already works in the space and has dealt with the barriers to entry, a single-minded pursuit of keeping the product as small as possible, is also excellent discipline. After all drugs companies and car manufacturers are both as determined to avoid ‘waste’ as any software company.

For start-ups, high barriers to entry can make the size of the MVP so large that there is little point in calling it a ‘little bet’. Instead, we should apply the common sense principles of avoiding waste and attempting to set up experiments to validate underlying business assumptions as soon as possible – although we should accept that the results are rarely binary and therefore there is little clarity on how to proceed. This is certainly incremental development with a focus on customer learning, but calling it a guaranteed process is overclaim by any standards.

In the end, I would sum up our caveats as these:

• Smaller is not necessarily better and viable is not always the right measure.
• Validated learning sounds great – but barriers to entry may force you to develop a product blind and without experiments. Blind progress may be better than open-eyed stasis.
• Pivot points do not only come from customer feedback – there are many other types of serendipity that may intervene and offer a choice. That choice is never easy because almost every trade-off hurts.
• All the metrics and hypotheses in the world will probably not help you when reality bites. Since luck plays such a major role in what occurs, try to keep this as light-weight as all your other planning.
• No process or discipline can guarantee success. There are always new, exciting and unforeseeable ways to fail. We’ll let you know what original ones Gamevy comes up with in the next year.

About the Author

Helen Walton is co-founder and Marketing Director of Gamevy, a tech start-up which recently won the PitchICE award and will launch its games soon via a partner. Her writing has appeared in diverse places, from the Daily Mail to the Tate Britain, and on topics from lipstick to organisational change. The three Gamevy founders met while writing the VFQ books which now form the BCS Agile Practitioner qualification. Three years of researching and debating methodologies, and interviewing hundreds of organisations led to the founders also setting up a community and conference - Spark the Change - for people who want to improve the whole organisation, not only IT or product development. Running in London 1-2 July 2015, Spark attracts people from around the business and from all industry sectors. Leading thinkers on the future of work and management will be speaking, along with case studies from innovative companies including WL Gore and Spotify, and practical workshops on skills and tools to implement real, lasting change.