Discussion: Measuring Success of an Agile Project from the Customer’s Perspective

Recently there was an interesting discussion on the Scrum Development user group trying to answer “How does a customer measure the success of an Agile project?” The important piece here is “measure”.The discussion seemed to agree that measurement of success from a client’s perspective is important, and there are various ways to achieve it. The best way to measure would depend on the situation and the customer.

Traditional development methodologies had certain metrics: quality metrics such as defects per release; schedule metrics, such as actual effort/days versus estimates and delivery milestone achievements; and cost metrics, such as budgeted versus actual costs.

However, most Agile development organizations are notoriously known for being weak on collecting metrics and reporting them back for the project. While they do have technical development metrics like checkstyle errors, test code coverage, NCSS, etc, these metrics are of little significance to the customer.

Measurement of success from a customer’s perspective becomes important because, though the team might be following the right process, the end result might not be what the customer is looking for. In fact, it might be very different.

Mike Dwyer said

You are asking me the same question that a surgeon asks. Am I doing a good job if I follow all the steps, and if the patient gets well or if they die is not my concern? My response is what is the goal here? A well patient or a successful procedure?

In this case, if the end result is what we logically expect, i.e. the patient gets well, then we would call it a successful project with a successful process.

Vikrama Dhiman provided another example:

My project could really be to make this 1000 story building which I hope to sell to United Nations - the team organizes itself beautifully and they help me realize my vision in time - but United Nations does not buy the same and I don't want to sell to anyone else. Is the [process followed for the] project a success? I would say yes. Is the end product successful? I would say no. Who is at fault here? Product Owner, most certainly.

Here the team might have got all the process right, however they probably built the wrong product. Was the product owner to blame? Yes. Was the team to be blamed? Yes. Is the customer satisfied? Definitely not.

So, is there a way in which a customer can measure the success of an Agile project? The answer does not lie in showcasing a great process as is evident from the above user statements.

A few people suggested that the customer would consider the Agile project a success if he is satisfied. Now, “satisfaction” could be hard to quantify and measure. A similar discussion on Agile India user group suggested a few subjective metrics for measuring customer satisfaction. The customer would be satisfied if:

• He got the business value that he envisioned from the project.

• He was able to change course according to business demands and get functionality delivered which makes more sense today.

• He was able to give early feedback

• He was able to prioritize the development of features/stories, thus getting the right functionality at the right time

• He was able to stop development before all the functionality was implemented, because he realized that the remaining functionality might never be used.

• He developed a successful two-way trust relationship with the development team

• He got no surprises during the project.

• He got value for money

A few others suggested that, before the team starts the project, a definition of “done” is important. The customer and the team should be able to come to an agreement on when a story is “done”. Once the team has decided on what “done” is, then customer satisfaction and the rest of the metrics flow from that.

Mike Dwyer added:

Definition of DONE simply states that when the customer accepts work from the team at the end of a sprint then that work is DONE. At this point all the effort and associated costs move from waste to value. This answers the questions, “how far along are we, against plan” (product backlog – cumulative accepted work) and how are we doing, as to budget (total planned budget – actual costs). It also answers questions like: how efficient are we (total actual costs / planned costs for DONE tasks) and what is our burn rate (average cost per accepted or DONE element).

Scott Ambler suggests a combination of objective and subjective metrics. He suggests that teams start by collecting some basic data, such as:

1. Actual cost (hours, $, ...)
2. Some sort of functionality delivered (use case points, user story points, ...)
3. Defect trends
4. Stakeholder satisfaction

In this way, the team can communicate to the business in the terms it uses (basically, money and systems capability) and still continue to track metrics that are core to development success, such as quality. A measure of stakeholder satisfaction is one of the most important metrics that a team (Agile or not) should introduce. Delivering defect-free code doesn't necessarily mean delivering valuable code.

There seem to be both subjective criteria, such as customer satisfaction, customer viewpoint and objective criteria, such as stories done, cost and defect trends, which would help a customer gauge the success of an Agile project.

The discussion seemed to agree that just following the right process is not important. In addition to the process there need to be both objective and subjective checks which would ensure that the project is going to be successful and that the customer would be satisfied. A development team could choose one of them or the combination which makes the most sense in a given scenario.