Now is the Time for Lean ALM

Software is becoming the ultimate “special sauce” for business innovation. The business processes supporting software delivery are transitioning from ancillary to criticalⁱ . Software delivery for many organizations is now a critical business process, and one that has to be integrated. Unfortunately many organizations are ill-equipped to treat their application lifecycles as business critical, often addressing the software delivery process as a black art practiced by strange peopleⁱⁱ . In this article we examine why organizations need to transition to Application Lifecycle Management (ALM) and what we can learn from Lean thinking in transforming ALM from an inflexible, expensive, dogmatic approach to one more able to reduce waste and deliver measurable value.

Traditional approaches just don’t cut it

Today organizations’ software delivery and application maintenance is managed in a traditional project-oriented way. A project is identified, resources allocated, and work happens. Work moves through the development production line from requirements, design, development, testing, and deployment. Each specialist area has a clear set of goals and is managed to those goals. But often projects that look successful when moving through manufacturing end up getting into trouble just before going into production. Depending on whom you believe, failed projects account for as much as 15 percent, and challenged projects another 50 percent, of all software activitiesⁱⁱⁱ . The reality is the larger the project, the more likely it is to fail. As the unknowns and complexity increases, it becomes difficult to optimize a process that works. As software moves up the value chain in terms of business innovation, the complexity and levels of unknown increase. Innovation, by its very nature, is about dealing with the unknown. That means traditional approaches break in the following ways:

• Upfront planning requires upfront knowledge – When approaching problems the team has never encountered before, it is hard to create detailed plans. More time spent on planning does not create better plans; the only way to create better plans is to actually do the work and fold in that experience.
• Business does not need perfection, it wants action – Well-defined processes tend to err on the side of completeness, finding it hard to offset requirements during the process to deliver value faster. Modern businesses are focused on time -to-market and delivering some ‘thing’ rather than delivering perfection. Business is short term, frequently measuring value in quarters, opposed to software delivery which may focus on projects in terms of years rather than months.  

Cloud and Mobile approaches mean more complex systems

Traditional software development approaches tend to focus on custom development. Software completely owned by the organization can be built using (third) generation languages from the ground up. But modern software development is very different, with software assembled from a set of services. Mobile combines web with local application services running on a variety of different device infrastructures, thus increasing the complexity and risk. You lose control because your software comes from many locations and is not managed by you or anyone you know. Just like ecosystems in nature, you cannot manage it, you can only observe and react to it.  If your software development is starting to resemble an ecosystem you should:

• Focus on visibility - Gaining visibility of the assets, their stability and how they are used will provide valuable insights on what assets to test, which order software should be delivered and the inherent risk of certain key services. If a service has a large number of known defects, it is important to either write code defending your application from those defects or explore using another service.
• Manage the trade-offs in an explicit way - Instead of allowing software engineers to make decisions about which assets to use or which services to connect to in isolation, make the architecture a visible and managed set of assets. Decisions between buy, build, and use need to be front and center to any planning activity.
• Be connected to the ecosystem -  It is all too easy to use a service while never returning to its source to keep abreast of the latest work, known bugs, or observations made by the community. It is important to be not only a consumer of the product, but also an active member of the larger community.  That requires your application development process to be connected to the community or vendor process.

And Agile is not enough

For many people, using an Agile method will solve these problems by creating a flexible, team oriented approach that delivers value. There is some truth to that^iv ; Agile methods do provide organizations with a framework enabling teams to better respond to change and deliver value to their customers. In reality, Agile adoption is neither deep nor broad enough. Many teams still wrestle with the concepts of Agile, finding their traditional practices so very different than those described in the Agile manifesto. And Agile at the team level is fundamentally different than adopting Agile at the enterprise level where size, complexity, and governance models get in the way of adoption. Organizations may want to do Agile, but still require detailed plans, organization around departmental models, and are resistant to the idea of releasing software frequently. For many organizations, ‘water-scrum-fall’ is the reality^v. Agile adoption is constrained by:

• Software development is organized in specialist departments – Breaking down departmental structures and discarding existing organizational models requires significant change, which for many centralized software delivery organizations is beyond their appetite or capacity. Also aligning Agile teams to business areas requires either the creation of new teams, or structuring software delivery in a very different way.
• Budget holders need details before projects can start – Faced with the classic dilemma of needing knowledge to plan, but needing a plan to get knowledge, many software delivery groups rely on a detailed requirements analysis to determine the size and scope of the project. Requirements often change once some code has been created and cause a large amount of friction between the Agile team and the original plan.
• Work is often a collection of projects – Agile relies on the idea that self-managed, motivated, skilled teams can solve any problem as long as they are empowered to. The reality is numerous people in many different locations and teams work on all sorts of releases and projects. Software is increasingly an assembly, thus services that are being bundled are often owned and developed by other parties. Those parties cannot be part of the Agile team because they are from an external vendor, open source project, or different part of your own organization.

Merging ALM and Lean seems to be the answer…

So it can’t be all Agile, and traditional approaches just don’t work. What’s needed is an approach to software delivery that supports improvement, while not assuming Agile is the only way of delivering value. Such an approach should draw on the best practices of management, while transforming those practices in response to the very fluid environment that software is being delivered into today. It is time for Lean Application Lifecycle Management, the application of management discipline to the practice of software delivery to adopt Lean thinking, an approach that focuses on increasing value and reducing waste. Lean ALM will:

• Approach the discipline of software delivery as a key business process - For many organizations, the realization they are a software company, or software is increasingly becoming a key part of their business is a fundamental one. It requires them to approach software delivery in the same way as other key business processes. This is not a request for more money, since software delivery often has large amounts of money being spent on it, but a change in philosophy.
• Provide tools to help improve software delivery - Lean comes with a collection of tools to help people improve their processes. Techniques; such as the Five Whys, Kanban, and Gemba; encourage practitioners to explore their processes in a different way^vi . With ‘process improvement’ a KPI for many organizations today, this is not entirely new. However, for many organizations, save for a tick box on the project sign off, quantifiable improvement is barely focused on. By introducing techniques and a supporting organization focused on helping people use them, a Lean organization should always be continuously improving. Lean process improvement focuses not on the process, but instead on the people by equipping them with the techniques necessary to deliver value and reduce waste.
• View software delivery as a series of flows - Instead of focusing on the disciplines, artifacts, or roles, think of software delivery as a flow, or series of flows. By visualizing the flows it is possible to understand how work moves through the system, identify batch size issues, queues, and waste.  This means the traditional departmental, or discipline, focused approach to organizing the delivery of software has to be broken down. Specialty disciplines still exist as part of a connected flow, not in isolation. 
• Focus on outcomes - Having a clear focus on outcomes that make sense to the business and delivery organizations makes for clear direction. For many organizations, having a clear set of measures describes not only the objectives for the team but the outcomes desired. However, most software delivery value chains are fragmented, with each group having its own measures. Business, business analyst, development, testing, and release/operations all have clear, but often contradictory, measures. Perhaps the most famous disconnect is between development and operations. Operations are measured on stability, SLA’s, development changes, time and cost. The tension is meant to create compromise by both parties, delivering high-quality software, but delivers most functionality at an appropriate cost. Unfortunately, because the trade-offs in development are hard to make for most organizations, quality is the only active measure, meaning high-quality software is released but is often feature light, costs too much, and is late.  The DevOps movement, not discussed here, has emerged in direct response to this tension driving organizations to integrate processes, measures and tools^vii .

ⁱ Software is eating the world reference
ⁱⁱ A great article on what software delivery so hard
ⁱⁱⁱ The often-quoted report on project failure is the Standish report
^iv There are many sources that describe how Agile is improving software delivery; the chaos report provides great data showing the success rates of Agile
^v I coined the term Water-scrum-fall when I was working at Forrester Research. A discussion can be found here
^vi There are many great Lean books, websites, and organizations. A great starter is this one
^vii More on DevOps can be found here

About the Author

Dave West is the Chief Product Officer at Tasktop. In this capacity, he engages with customers and partners to drive Tasktop’s product roadmap and market positioning. As a member of the company’s executive management team, he also is instrumental in building Tasktop into a transformative business that is driving major improvements in the software industry. As one of the foremost industry experts on software development and deployment, West has helped advance many modern software development processes, including the Unified process and Agile methods. He is a frequent keynote at major industry conferences and is a widely published author of articles and research reports, along with his acclaimed book: Head First Object-Oriented Analysis and Design, that helped define new software modeling and application development processes. He led the development of the Rational Unified Process (RUP) for IBM/Rational. After IBM/Rational, West returned to consulting and managed Ivar Jacobson Consulting for North America. During the past four years he served as vice president, research director at Forrester Research, where he worked with leading IT organizations and solutions providers to define, drive and advance Agile-based methodology and tool breakthroughs in the enterprise. You can also connect with Dave via Twitter @DavidJWest.