Introduction
"They don't get it. They aren't doing it right. They aren't really agile until they follow all the principles and practices." Have you ever heard these comments before? With the increasing hybridization of the agile world there is growing disdain for non-pure agile thinking. Ironically, and paradoxically, agile's chief accomplishment was to get us away from a one-size fits all development philosophy in waterfall. Now, it seems the zealous lieutenants of agile are dooming themselves to repeat the same mistake ......holding agile up as the only way. This article will attempt to show that not only is hybridization necessary but even preferred if we as software professionals seek continued innovation and better methodologies.
Waterfall just doesn't work
Agilists are famous for saying this and indeed they have a great deal of evidence to support their views. But their rhetoric would probably be more accurate if it was reworded to say: "Agile tends to work better on certain types of projects and in certain types of organizations with certain types of technologies." It's not the case that waterfall never worked. It just didn't work consistently on all projects. You can find plenty of veterans in the industry to tell you they've done just fine with their waterfall process. Are they lying? Are they resisting change?
The truth is both techniques tend to work depending on the mix of variables. I’ll discuss some of these variables below, but there are others. My point in highlighting these is to show how context plays an important role in how a software development effort is approached and perceived:
1. Project Type
Is this a fixed bid effort or an ongoing strategic software product that will form the core of the business? Fixed bid efforts are usually better suited to traditional waterfall process, while Scottrade would find it more advantageous to use agile. Why? In the first case the project is seen as an expense to managed, tracked and controlled. At Scottrade their online trading platform "IS" their business. A more involved explanation of this distinction can be found in one of my previous articles: Why Agile Adoption Fails in Some Organizations
2. Technologies Used
Web based technologies usually require less deployment time then more traditional heavy client applications and because browsers follow (for the most part) industry standards it’s easier to implement. They are the optimal technology to employ with agile methodologies. If you're deploying a heavy client product that requires installation on 16,000 laptops, desktops, with a mix of operating systems (or virtual machines) across an organization spanning 15 divisions....you're development process will be much more waterfallish. You may still develop in time-boxed iterations and deploy your code to a TEST environment every 2 weeks, but delivering that same product to production on an iterative 2 week basis is complex and fraught with risk. It requires much more planning. Installing something new on everyone's desktop every 30 days is not a good idea. By the time all the associated issues and problems with the first installation were resolved, one would be ready for another 30 day installation. The product would be in a constant state of flux. Delivery of functionality is important, but so is stability, consistency and performance, one has to weigh the technologies used against the intent of the product. If more change and rapid feature delivery are important does the current technology mix support it? If what’s prized is a stable, and reliable system that hardly ever goes down then reworking your entire software delivery process may not be necessary.
3. Project Complexity
When the entire project is a new social media startup agile is a natural fit. The complexity of the project is probably contained within the developers/founders minds and their needs for information and requirements clarification exist nearby (maybe even in the same room). Their development effort will probably not require any integrated hardware. They will not interface with another division of the company with its own development/release cycle. They will probably not be integrating their software with another vended software team, and lastly there will probably not be any business process improvement effort that they need feedback and information from.
Contrast this with a Six Sigma Black Belt business process reorganization project spanning 6 divisions of the company. The software development piece of this project may be small in respect to the entire effort. Their needs for information and requirements clarification will hinge on the "steering committee" which may not be able to meet more than once a month. There may be other vended software that they are required to interface with and pull information from (or send information to), so negotiation and timing of those integrations is crucial.
Sitting around these examples are differing levels of complexity. While polar opposites are portrayed here, the point should be clear that the higher the complexity of the project the more likely that deeper planning is required......even necessary. The context of complexity gives us pause to consider our approach.
4. Culture
Is the company open to new ideas and ways of thinking? Do they want to explore agile software development? How do they adjust to change in general? If the company is conservative then your introduction of agile may not find a strong following and may create enemies. Caution may be required. A more liberal organizational would probably see change as liberating and empowering. The flip side of this coin is that an organization could be too fad driven. In this culture new ways of thinking become the flavour of the month and rapidly find fertile ground before discernment and reflection have a chance to decide what’s best.
The Third Way
Sitting between agile and waterfall is pragmatism These are the shops that have taken pieces and parts of agile and waterfall and cobbled together their own process/movement. They focus on techniques and results and skip the dogma and rhetoric.
The pragmatists are often scorned by their agile peers as "unwilling to go all the way". I would contend, instead, that they're doing exactly what the original Agile founders did: experiment with new ideas on how to develop and build systems. The pragmatists see that a dogmatic approach to software development that doesn't foster flexibility, taking account of the contexts mentioned above, is doomed for failure.
Truly, new ideas come from the pragmatists.
Far from "not getting it", they are seeking out those pieces and parts that work consistently for them and maybe even for all projects. This willingness to experiment and tinker and not follow the well worn path is invention at its purest form. It should be encouraged and watched. If history is any guide, something revolutionary could spawn.
Here are some pragmatic approaches I'm seeing and hearing about in the market place that seem to make sense:
1) Use pair programming on tough issues/defects or critical architectural systems. Rather than enshrine this XP technique as the norm, pragmatists are using it as needed. Further, I've seen this evolve into group programming where a team gets together in a room and hashes out the code and key integration pieces needed up front.
2) Layer more traditional PM practices over a Scrum approach. Vehement agilists will tell you this is bad all the way through. But in my experience this does have its place. For example a project that has a heavy business process component that finds the software development project is only part of the overall effort. One may be able to get away with using scrum to develop the software, but that scrum process will need to meld tightly and neatly with the overall business project and its practices, procedures, timelines, risks, and budget. In most enterprises, scrum isn’t the methodology chosen to run a business process project.
Another example is a project with an offshore vendor. These projects can be difficult to coordinate owing to time zone differences and the limited resource capacity of the vendor. Additionally, the vendor may ask that the 15 minutes daily stand-ups, iteration review time and iteration planning time be billed in addition to the product costs. One has to weigh whether the additional communication is worth the extra premium in project costs. You can often achieve the same quality with a weekly touch base meeting and strongly defined specification. Additionally, the vendor is usually contractually obligated to deliver what the customer asked for. Doing otherwise, means no payment. The incentive to ‘get it right’ is very high in this situation and this may make a waterfall approach more palatable, even acceptable.
3) Change management in agile - agile embraces change, but embracing any and all change can set a team up for failure. We’ve all been part of projects where a customer will change their minds frequently throughout the development process to the point of revising a set of stories many times over even though the original requirement was developed and tested several iterations ago. This can occur for good reasons, but bad reasons for change come about too: product owner that can’t make up their mind, is not aware or accountable for a budget and feels the project is endless, or needs to "see it built" before he/she can say it’s right The scope creep in these situations can be harmful to the overall project and may find leadership doubting agile. Stronger change management techniques orchestrated by a scrum master or more traditional PM, are becoming necessary to document the progression of the system and justify budget and schedule variances.
4) User Story Manager - The agile approach is that the team and product owner put these together during a sprint planning session. However, this often leads to poorly written stories that don't account for all the details. Agile requirements misunderstanding is a common problem. Designating a full-time user story writer, while expensive for a project, is worth the money invested. The clarity provided leads to easier coding, better testing, and ultimately stronger quality.
These are just some of the things I've seen. But, I'm interested in hearing from others. What innovations in agile and scrum are you seeing?
Summary
The pragmatists don't see either approach, agile or waterfall, as a 'take or leave it' package. I'd liken them to independent thinkers who see the world as a supermarket of ideas.
"Let's use daily stand-ups, but I still need that Gantt chart to integrate with other departments."
"I like showing our work to our clients every 30 days, but we need some kind of strong change management because this client has some truly whacky ideas."
Agile hybridization is good. It's fostering innovation in the field. Without this, we'll surely repeat the mistakes of the pre-agile world. In a sense extreme agilists are right: pragmatists aren't following the methodology correctly. But, that's not a bad thing.
About the Author
Christopher R. Goldsbury is a software development professional who has played the roles of developer, architect, scrum master, development manager, project manager and quality assurance manager throughout his career. Chris writes on his experiences and ideas at his blog.