In the 1990s we were at an all time low in the software development industry. Programming was becoming a mainstream profession and we were mostly terrible at it. A gathering of some of the best minds in the industry occurred in 2001 and, while I suspect they couldn’t agree on a methodology, what they did agree on has had the biggest impact on the industry, ever. What it did was answer the problems of the ‘90s. And they created a community for like-minded people to discuss how best to deliver software. Now that Agile is becoming mainstream and we are getting a lot better at building software successfully, a new problem is starting to become apparent.
A great quote I (Gordon) once heard was, “never confuse the composing of a symphony with the writing down of the score”. There are only a few genius minds that can come up with a symphony that really works and when those symphonies are played they invariably touch the audience’s minds to excite inspiration and exultation.
I am probably taking this analogy too far, but I think it also really works for software. The problem as I see it is that we have solved how nice the code is and how well it is written, even whether it is written in the right language using great designs or patterns. Let me explain, all of these things are extremely important. Unfortunately it is only these things most software engineers focus on.
The thing that has plagued software projects and product delivery for years is deciding what to build. The answer is often, “what the user asks for”, but this is unsatisfactory as if you ask a user they typically do not know what they need either.
At university a lecturer told Gordon a story of a car manufacturer in the USA who conducted a survey asking people what they wanted in a car and duly built what they thought would be the ideal American car. The outcome? It was a flop. Why? The answer can often be complex to these sorts of questions, but in this case the root cause I believe can be traced to the fact that car users are not car designers.
The same can be observed for users of computers; they most definitely can tell you what they don’t like about something but most of them can’t help you design a system from a blank sheet of paper. They, like most people, work best with something that appeals on multiple, visceral channels, be they visual sensory or auditory. We’re also fundamentally better at working back from an end-point, rather than try to reach an unknown one. Trying to define requirements without being able to appeal to feedback on each of these channels will be nearly impossible for them.
The 2002 Standish Group survey of successful waterfall delivery found 80% of delivered features are never or rarely used. Even if this is not completely accurate, as I am not sure how this is measured, I am confident there are features in the software I’m using to draft this that I will never use or even know about and therefore never need. My friend and comrade in software delivery innovation, Russ Miles, has gone on record to point out that he has “a version of Microsoft Word v1.04b on a Mac circa 1987 that has all the features I need to get most of my work done”. This is not just trivial ‘bloatware’, this is waste on a massive scale. Waste of money, waste of processing and, most importantly, a waste of that most precious of resources: human life. Think what else could have been built if all those hours spent on Clippy (to pick an easy target) could have been spent on building something actually useful and needed!
To give another example, there are features on my mobile phone that are equally as pointless. The point is not to criticize the devices or software itself, it is more to point out that someone defined it, wrote it and tested it. Of the trillions of dollars worth of software that has been developed even in the last 10 years, how much is pointless and not used or even not fit for purpose? Just imagine if all of that had been useful? Maybe 2010: A Space Odyssey might have actually come true.
I am not saying there aren’t truly great things being produced all the time and advances being made at a mind boggling pace. The point is that the majority of software features being built today will probably not be used. I’d say that that was pretty depressing, if not downright irritating, and should especially be for the person who is composing the symphony.
Agile is becoming mainstream too, which is fantastic, and hopefully that means that we can start to put individuals and interactions over processes and tools. The Agile manifesto is a concentration of some of the best minds and most successful delivery systems in the software development industry. What they were doing was trying to distill what made software delivery successful, ie.: loads of collaboration, small batches, regular reviews of working software. All of these things ensure that delivery has as much chance as possible to be a success, as opposed to the huge risks of large batches and large projects.
The thing I’m worried about is what happens next? Let’s say that agile software development has finally attached the engine to the drive chain… and we’re off! We can now deliver at an astounding rate, that’s great! For decades we’ve been attempting to do just that, so we have achieved a lot. But, as Russ also pointed out in his recent talk at QCon London 2013, now that we’ve achieved the goal of “delivering valuable software frequently”, shouldn’t the next goal be to focus on “delivering valuable change frequently”. What if the software seems valuable and in fact turns out to not be? What if software is only part of the picture? We could have just spent 10 years turning on the faucet and attaching our mouth to the tap only to discover that we’re now drowning…
There’s a real danger that we will start to ‘successfully’ produce more of the wrong stuff, even faster and more often. Sure, the chances of getting the right thing done are significantly increased with techniques such as high degrees of collaboration but it is naïve to assume that by adopting agile techniques you start to build the “right thing, right”. After all, it’s still the same set of people.
We were involved in a fabulous gathering in February organized by our friend and guru Gojko Adzic. He managed to get a group of some of the most successful people in the software industry together for a one-day, intense workshop. We grappled with just this problem at length and Gojko has expressed the outcomes on his blog.
During the session we were introduced to techniques we had looked at before and new techniques that we’d never seen in action, all of them exceptionally well thought through and all of them trying to help us understand the fundamental question of why In 2001 the Agile manifesto members were unlikely to agree the “right” method to deliver software, we were also unlikely to agree. Each technique in its own right is very powerful and each ideal for a specific context, and each firmly targeting the question of ‘why’ are we building this software.
One thought that hit Gordon was that what we, technologists, need to help business leaders to articulate why they need a solution using visual methods because doing it is extremely hard.
These are real practices that work in the real world in the same way that XP and Scrum enable teams to try and become Agile, with all that term means. These frameworks are to enable us to try and achieve this.
One of the outcomes from this day workshop were the following principles for good software development:
- Organisations focus on delivering outcomes and impacts rather than features
- Teams decide what to do next based on immediate and direct feedback from the use of their work
- People know why they are doing their work.
- Everyone cares.
We have ordered them differently from the original we arrived at during the session because it feels more to me us like a natural hierarchy.
Those principles chime perfectly with a new emphasis, even a call to arms, that we are attempting to put to the industry. To coin a phrase that is perhaps overused enough to be close to a cliché, while standing on the shoulders of the original agile manifesto’s authors, we as an industry are starting to achieve part of that message. We’re delivering more and more, which is a good thing.
Now it’s time to start delivering the right thing, and it all starts with asking ourselves why we are delivering software in the first place. As Russ puts it, “We are not (only) software developers; we deliver valuable change”. That’s the story for our next 10 years as an industry. Getting that right is going to be no less difficult than getting Agile into the mainstream has been. But it is absolutely critical to our future.
Welcome to the future of successful software delivery, it’s going to be all about Why?
About the Authors
Russ Miles is Principal Consultant at Simplicity Itself and works with their clients to continuously and sustainably delivering valuable software. Russ' experience covers almost every facet of software delivery having worked across many different domains including Financial Services, Publishing, Defence, Insurance and Search. With over 16 years experience and through consultancy, coaching and training, Russ uses a holistic view of the software delivery process in order to implement multi-faceted continuous improvement programmes touching on everything from developer skills and practices, creating and evolving the best architectures and designs for a given domain, through to advising the management of various companies on how to apply lean and agile thinking and practices to better tune their return on investment from their software development effort. Russ is also an international speaker on techniques for achieving the delivery of valuable software as well as a published author, most recently of "Head First Software Development" from O'Reilly Media. He is currently working on two new books; "Programming Spring" for O'Reilly Media that launches the Simplicity Itself technique of "Test Driven Learning" for the first time publicly, and another book, working title being "Field Guide to Continuous Improvement for Software Delivery Team Members" that captures the different thinking tools and techniques that a professional software developer can apply concretely to their own continuous improvement goals.
Gordon Weir is an expert in large scale agile and lean organizational change. He has travelled a long way, from the New Zealand Young Engineer of the year in 1998 while working for the Royal New Zealand Navy, to a Technology department head at Bank of America Merrill Lynch in New York, a move he made after living in the UK for 14 years as a consultant initially and latterly in Investment Banking. As a result of this journey, he has developed a deep knowledge and understanding of diverse cultures, and what is necessary to help them change.
