Implementing Agile Delivery for Non-Software IT Projects

"How are you going to run a strategy initiative using an Agile delivery model?" asked Elle, the exasperation clear in her voice, "where is the working software here, and how do you show immediate business value?"

The background of this conversation was a major transformation initiative adopted by MotoClaim & Co. to embrace the service-oriented enterprise philosophy. Elle was the delivery manager for the overall initiative and was no stranger to Agile. I was the consulting process architect for the program. We had both just come out of a senior management meeting where it was decided that the first phase of the program will include building a strategy to execute the entire transformation, and developing the required foundational architecture. In this meeting, in spite of a lot of skepticism from the senior management including Elle, I had pushed for using Agile to execute the first phase.

Elle's question is definitely not specific to just MotoClaim & Co. In my last ten years of Agile consulting, I have been through this debate multiple times with my client stakeholders, even the ones having considerable maturity in Agile delivery. The most that anyone is willing to go is an iterative approach when it comes to non-software initiatives. By non-software initiatives, I am referring to those IT projects that are not meant to deliver working software to production such as strategy definition projects, architecture projects, internal assessment initiatives and consulting initiatives. Although these projects may hold a lot of intrinsic values for the organization, they are typically invisible to the end users and do not affect their business-as-usual.

So, what is the problem in using Agile delivery for a non-software project? For starters, if we look at the Agile literature available today, most of it, including the Agile principles, centers on delivering 'working software' early and frequently–and working software is indubitably linked to projects that deliver software to production. Since non-software projects do not deliver working software, it is difficult to perceive how they will align to the core Agile principles of satisfying customers through early and continuous delivery of valuable 'software', deliver 'working software' frequently, and making 'working software' as the primary measure of progress. Another key reason is multiple ownership, especially for strategy projects, which typically have many high-profile stakeholders with conflicting interests. Unavailability of industry-established metrics to measure non-software projects is a third aspect. I have seen many architecture development projects shut down mid-way because the business stakeholders did not feel that the money spent in the initiative justifies the outcome. It is therefore not surprising that organizations often find it difficult to justify Agile delivery for non-software projects.

"So, what's the point," you may ask, "why is it so necessary to use Agile for these non-software projects? Typically, these are short-term projects running for three months to a year. Why can we not simply follow a waterfall or iterative approach?" To answer this, we will have to go back to the basics of Agile adoption.

We all understand that not all projects need Agile delivery. Then, what are the criteria of adopting Agile for a project? Using the illustration provided by Ogunnaike and Ray¹, we say that any project that falls outside the simple zone must use Agile. The ones inside the simple zone will derive little or no value using Agile. To identify if a project is simple, we typically conduct Agile Readiness Assessments (ARA) for projects that reviews various parameters to ascertain four key factors:

1. How well do the stakeholders know their requirements?
2. How well do the stakeholders know their technology?
3. Are there conflicting interests from different stakeholders?
4. Is there a high risk of failure?

Needless to say, a majority of strategy, architecture, and consulting projects will come up with strong risks on all these four points, and therefore will almost always require high visibility, early risk mitigation, adaptability to constantly change, and quick demonstration of business value. Using Agile here is both undeniable and indisputable. In fact I would argue that Agile methods suit no other projects better than non-software projects.

The next obvious question is, "How do we make this happen”? In my opinion, as soon as organizations stop practicing 'prescriptive Agile.' Even though the very foundation of Agile is to be adaptive and not prescriptive, prescriptive Agile is one of the major oxymorons across the Agile delivery landscape today. In his insightful article, Abraham Kiggundu writes,

I have come to believe that the main reason many efforts to go agile fail is because practitioners tend to forget that the manifesto is a statement of utopian ideals to aspire to. Any prescriptive practices that come out of it like Scrum and Kanban are simply practical attempts to stride towards this ideal however, due to the nature of the principle, none of these practices can ever actually get to the ideal.

Arijit Sarbagna's article² also highlights this concept: "If you practice Agile, you must have come across the ever-new demon called 'Prescriptive Scrum'."

A famous actor in a popular Indian film once stated that, "Religion is omnipresent. It cannot be dispelled. If you try to dispel a religion, you will 'be' the religion." True enough. For several years, the software industry has been religiously practicing the now outdated waterfall approach, until Agile methods showed them the power of quicker deliveries and self-organized teams. While Agile methods successfully dispelled the slow moving waterfall, many organizations now picked up Agile as their religion. The process whose manifesto declared "People over Processes" has now became a standardized prescriptive process in itself.

The only prescriptive practice that you need to deliver a project successfully in Agile is 'to be Agile' and non-software projects are no exceptions.

To successfully execute non-software projects using Agile, I have defined the following five-point agenda for my own work. While this may not be 'the' solution, this agenda has helped me set clear expectations with stakeholders, bring sanity to project execution, and mine the benefits of Agile. While this agenda may look similar to the principles followed in an Agile software delivery project, their applications are quite different when it comes to non-software projects.

1. Define 'Working software': Although this term is generally perceived as an executable, it can be used to refer to any deliverable that is produced by the project and brings value to the customer. For example, it can be an improvement roadmap, an architecture prototype, a strategy document, or a cost-benefit analysis.
2. Define 'Customer': In a development world, it is easy to see who the end customer is. However, in a non-software world, identifying customers may be tricky. Typically, these are the stakeholders that have direct or indirect interest in the initiative. However, unlike the end users of a system, this set of customers may have conflicting priorities, internal biases or preconceived notions. That many of these stakeholders are senior management and even C-level executives makes this task even more difficult. Choosing the 'right' Product Owner, who can rein in these complexities, is most crucial.
3. Define 'Done': Work with the sponsors and product owner to identify 'done' for each story/deliverable. Since a formal quality assurance may not be applicable for many of the deliverables from non-software projects, it is important to ascertain the conditions of success. For example, the success criteria of an architectural prototype may be critical design review followed by a demonstration. The catch here is that since most of these deliverables will be implemented at a later time through a software development project, the stakeholders may defer the approval till the deliverable is successfully used in a future development project. This is equivalent to setting both the projects up for failure. What if the strategy fails? Now we have two failures to deal with: the already-closed non-software project whose strategy deliverable did not work, and the in-flight development project that cannot be continued due to an incorrect strategy.
4. Measure Business Value: Measuring or establishing the business value of the work done is a key for any non-software project. Since most of the deliverables exist as theories or prototypes, it is important to prepare a business case at the start of the initiative that clearly provides the cost benefit for this initiative followed by articulating the benefits achieved at regular intervals, preferably at the ends of iterations. Metrics for measuring these non-project initiatives should be identified and institutionalized. For example, an architecture development team may adopt metrics such as risk mitigation factor, proposed Total Cost of Ownership (TCO), proposed total cost savings and architecture due diligence rate.
5. Expect the Unexpected: Strategy or consulting projects are notorious for wild changes. This is normal as the sponsors and management are still in the brainstorming mode. Project scope, objective, and goals are liable to change frequently and drastically. Therefore, go for shorter iterations, joint workshops, paired development of deliverables, continuous expert and peer reviews, and proper socialization of theories and ideas. Having senior strategists, architects, or consultants is helpful, especially if they have a deep experience on the subject matter as well as with the overall organization.

Once the agenda is established, the entire initiative can easily be run in an Agile model using Scrum or other processes. You would be surprised to see how perfectly you can apply standard agile techniques such as pair-programming, test driven development and refactoring to non-software projects that would otherwise seem irrelevant without working software. I have been able to successfully use this agenda for many different kinds of non-software projects including strategy making, enterprise architecture development, business process optimization, process definition and institutionalization, and metrics development. A couple of these experiences are provided below as case studies:

Case Studies

Case Study 1: Development of Foundation Architecture

The organization had undertaken a large transformation program to modernize its existing legacy platforms and to move towards the service-oriented paradigm. Since all the existing architecture elements within the organization were specific to legacy, the I.S. leadership wanted an initial phase of identifying, acquiring and establishing a foundational architecture for service-oriented deliveries. An estimated timeframe of one year was provided by the senior architects to set up the basic minimum architectural elements required for service development. Being a high-risk exercise, the I.S. leadership wanted this initiative to be run in an Agile model so that risks can be mitigated early and a complete visibility is established at all levels.

As a part of drawing the vision for this initiative, I worked with the key stakeholders to establish the five-point agenda.

1. Define 'Working software': Every major deliverable was 'working software' for us. Identifying major deliverables was an essential activity since the backlogs were quickly filled with anything and everything that the stakeholders and team aspired to provide through this initiative. This included basic foundational elements to high maturity capabilities. We carefully prioritized and agreed to deliver a piece of work only if the team felt that it can add sufficient business value to the organization, and the organization is ready to accept the change. Technology infrastructure/tools set up for different environments; development of design documents; and proof of technologies/concepts were the key types of working software.
2. Define 'Customer': Since this initiative was an enterprise-wide initiative, we had a large number of stakeholders to work with. We developed an 'influence-interest grid³' that helped us understand the need for communication and potential resistance to change. We appointed one of our key 'high-interest, high influence' stakeholder as the product owner. The influence of the product owner, who was an assistant vice president for the division, helped to prioritize our backlog well and get us the socialization and buy-in that we needed.
3. Define 'Done': Each 'working software' was associated with 'done' criteria, which was fixed based on our discussion with the various stakeholders. For example, for infrastructure setup, we worked with the enterprise infrastructure support team to derive a set of test cases and sample test data that could be used to verify proper installation of the infrastructure. For design documents, we had critical design reviews by the enterprise architecture board (EAB). We had worked with the designated EAB reviewers to draw up a detailed checklist against which they would review the documents.
4. Measure Business Value: Even before the initiative started, a business case with cost-benefit analysis was presented to the management articulating the return on investment (ROI). With every sprint, we provided a high level measure (sometimes quantitative and sometimes qualitative) of how we were aligning to the promised ROI. For example, after setting up the service registry, we demonstrated the realization of the reusability value that was made in the business case. We actually created a small and simplistic service producer-consumer scenario to articulate how easily the registry can allow the service consumes to discover and reuse existing services.
5. Expect the Unexpected: This was more of a culture adopted by the team. We had four-week sprints and reviewed our goals continuously with the product owners. As a scrum master, I diligently maintained the risk and impediment logs, and had frequent one-on-ones with the sponsors. Having a product owner with high organizational influence helped since we were always updated on the high level strategy changes that is going on within the organization and could ready ourselves for the upcoming change. We also established formal organizational change management procedures, so that the changes to the project scope and goal could be properly socialized and articulated.

With the five-point agenda established, we used Scrum and a number of XP and DSDM practices to successfully complete the project in less than the stipulated time. Today, the organization is well established as a service-oriented solution provider and has been assessed at Level 5 of the Open Group Service Integration Maturity Model⁴ (OSIMM).

Case Study 2: Business Process Optimization

After delivering a number of unsuccessful updates to its value chain, this organization initiated a major program to completely remodel its end to end business processes. The first phase of this initiative consisted mainly of business architecture work wherein the target state processes will be designed. This was a critical initiative as the changes would impact almost every member of the organization as well as the end customer. As a process coach, I was asked to aid the team to execute this initiative using an Agile model.

Once again, I worked with the key stakeholders to establish the five-point agenda.

1. Define 'Working software': Naturally, the key working-software candidates were the target process models. For larger and more complex processes, the team broke them down into logical stages and each was considered as working-software. The deliverable was a process model created using Business Process Engineering Language (BPEL).
2. Define 'Customer': Although the definition of the customer was straightforward and could be identified from the actor-process mapping, the real challenge was aligning their expectations. The number of customers was large and had very conflicting requirement of the target state. While some of them wanted more control, the others preferred greater autonomy. While the marketing department wanted more flexibility of product to increase their sales, the claims department preferred products to be less flexible so that claims processing can be quicker. We knew that the product owner will be one of the business heads, but every business head was catering to a specific division and was partial to the needs of that division. For the first time, in my long career as an Agile coach, we actually went ahead with a set of three product owners to cover the entire value chain. Talk about 'being agile.' However, some ground rules were set. First, the team will always get a consistent direction from all three product owners. Second, any difference of opinion between the product owners will be settled internally and the team will be kept out of it. Third, the product owners will be time-bound with their tasks and decisions.
3. Define 'Done': This was the easy part. The product owners interacted with their own department to prepare a comprehensive list of 'features' that they required from the new system. These were then filtered and prioritized to create a shortlist. This shortlist was used by the business managers and end users during the sprint review presentation where the team presented the theoretical process flow in the BPEL tool. Any new features that came up in this meeting were added to the backlog and prioritized in the next sprint.
4. Measure Business Value: Measuring business value was not very complex to do. The existing model of current business processes came with the best, worst and average time taken for each activity. This was a part of a previous research work done by the organization. When the target model was created, we could identify definite time reductions and other efficiency upgrades. For example, if claim processing took two weeks in the existing process, we could show that in the target process, due to automation of processing rules, the processing time will drop to four days. Projecting the business value this way ensures a continuous interest from the user community and support for the initiative.
5. Expect the Unexpected: Once again, this was followed more as a culture. We had four-week sprints and reviewed our goals continuously with the three product owners. Risk and impediment logs were diligently maintained. All new requirements or changes to existing ones were welcome. However, a clear expectation was set that the incorporation of this change will depend on its prioritization. The product backlog and burn-down available for all customers to view. We also established formal organizational change management procedures, so that the changes to the project scope and goal could be properly socialized and articulated.

Conclusion

As the two cases illustrate, Agile can be used across disciplinary boundaries very effectively. Further, Agile delivery is imperative for non-software projects due to their inherent risks and complexities. Models may vary, but the concepts of Agile do not. Instead of aligning to prescriptive Agile methodologies, we need to align ourselves to the overall Agile manifesto of choosing

• Individuals and interactions over processes and tools,
• Working software (or deliverable with business value) over comprehensive documentation,
• Customer collaboration over contract negotiation, and
• Responding to change over following a plan

Remember – "be Agile, be Adaptive".

References

¹ Process Dynamics, Modeling, and Control, Oxford University, 1992

² Prescriptive Scrum – Is It the Need of the Hour or Doomsday? published in Scrum Alliance community

³ Mitchell, R. K., B. R. Agle, and D.J. Wood. (1997). "Toward a Theory of Stakeholder Identification and Salience: Defining the Principle of Who and What really Counts." in: Academy of Management Review 22(4): 853 - 888.

⁴ Open Group Service Integration Maturity Model 

About the Author

Dipanjan Munshi is a senior process and quality consultant at Tata Consultancy Services (TCS) with more than 17 years of IT experience spanning across various domains such as insurance, financial services, manufacturing, embedded systems, transportation and hospitality. He holds a Masters Degree on Computer Applications from Utkal University, Orissa, India.