At the recent SDC conference in Sydney & Wellington Prof Philippe Kruchten delivered a talk titled “What Color is Your Backlog”. The thrust of his talk is about bringing a focus on architecturally significant aspects of software into Agile projects, along with delivering the functional components of the system.
Kruchten maintains that the Agile focus on YAGNI (You Ain’t Gonna Need It), refactoring and leaving decisions until the “last responsible moment” brings a risk of delaying important decisions for too long. On any project there are a set of key decisions that need to be made early that will set the scene for ongoing work – for these decisions the “last” responsible moment is in fact very close to the beginning of the project.
These key decisions are the ones that drive the architectural structure of the system being developed. Making poor choices (or not making conscious choices at all) can result in a system that can’t adapt to the evolving business needs. They are choices about the architectural structure of the system that need to be considered early in the development process.
He states that many Agile projects focus on the functional needs (expressed in user stories) and neglect the architectural aspects with a irresponsible “we can refactor for that later” attitude, all too frequently leaving the “later” until far too late.
He gives an example of a financial trading system that used Agile methods.
User stories were identified, releases planned and development started. The first few iterations were resounding successes – features were delivered, customer engagement was great and the functionality was exactly what the business asked for. Once there were sufficient features to constitute a “minimum viable product” a release was put into the production environment and the project hit the wall – the features which had been built worked, but the product couldn’t cope with the volume of transactions that were needed in the live environment. The customers had to revert to the old system (which had not been turned off) and the development team went back to refactor the product to cope with the live environment demands – unfortunately refactoring meant that most of the work that had been done had to be completely rebuilt, the only components that could be salvaged were the screen designs. The project was eventually cancelled, and the customer organisation was very reluctant to try Agile again.
He acknowledges the vital need to show progress – we don’t want the development team spending the first two or three iterations beavering away on “plumbing stuff” that has no visible value to the users. He suggests that it is possible to weave architectural requirements into the overall release plan of the product, ensuring that as the product is built both the architectural underpinnings and functional features are built and we can show progress both in terms of increased functionality and the ability to meet the quality goals. So, for example, the first iteration may only result in a single entry screen but load testing will show that that component of the system will cope with the likely expected demand in the production environment.
He offers a technique for making work visible and ensuring that architectural needs get addressed and prioritised along with visible feature-based work - applying the concept of color coding to the items in the product backlog.
He states that there are likely to be four colors to the work in the backlog:
- Green – Features to be delivered, the functional user stories
- Yellow – Architectural infrastructure that support the quality requirements
- Red – Defects that are identified and need to be addressed
- Black – Technical debt that builds up as the product is built and key decisions are deferred or poor work done
Figure: Colors in the Backlog
Note the two dimensions – positive vs negative value and visible vs invisible features.
The green components are the Visible Features most commonly found on product backlog lists – the User Stories that define the functionality to be delivered in the product. These will be visible to the users of the final product.
The yellow pieces of work are invisible to the users but provide significant value to the overall product. These include work done for:
- Architecture
- Infrastructure
- Common elements
- Libraries
- Frameworks
- Making components reusable
Yellow (valuable but invisible work) needs to be made explicit in the product backlog and release plan, so project stakeholders know to prioritise it along with visible features.
The red parts (Visible Defects) are inserted as the product is developed – in an ideal world there will be very few red components but it is naive to assume there will be none.
The black (Technical Debt) is what happens when the yellow is ignored; technical debt may also have accumulated from previous work where the project is working on a pre-existing code base.
When building an initial release plan it is important to plan for both green and yellow components – a single-minded focus on visible features could result in a product that looks wonderful, but doesn’t scale to the production environment, or doesn’t cope with real-world security threats. Likewise purely focusing on the yellow leads us back to BDUF (big up-front design) and delays delivery of any visible value until so late in the project that the customer engagement is lost.
He suggest building a release plan that addresses both “like a zipper” – weaving the functional and architectural components together, as in this image:
Figure: Plan your releases to merge visible and invisible value components like a zipper
The red and black (Visible Defects and Technical Debt) will not appear on the initial release plan, but they do have a significant impact on the velocity (rate of work done and value delivered) available on the project.
Defects need to be addressed as soon as they are identified as is practical and feasible; left unresolved they fester and result in significant impact when they are addressed later. After a few iterations the team will have an understanding of their likely rate of defect insertion, and can adjust the planned velocity to ensure defects are removed as close to the point of insertion as possible.
If the project starts with known Technical Debt then this must be taken into account when planning the work to be done – an older code-base is likely to have hidden defects already there, and the code may not be easily refectored so again the rate of value delivery will be reduced by the quality of the underlying product.
When building a new product there is not initial technical debt, but there is a risk of debt being incurred. Deferring defect fixes, ignoring the yellow components, deferring key decisions until beyond the “last responsible moment” all result in an accumulation of Technical Debt and a reduction in the rate of value delivery.
Where defects and technical debt are identified in the course of the project, it is important to make them visible to the project team – hence the concept of color coding the backlog. New stories need to be added to the product backlog to show the defects and work to be done to pay down technical debt.
He strongly recommends color coding the master story list to clearly show the type of work to be done – is the story a new user feature, an architectural component, a defect that must be fixed or some technical debt that needs to be paid?
By highlighting the colors in your backlog, and by making all four aspects of the product development clearly visible more realistic planning and tracking of the project will be achievable.
What are the colors in your backlog – what techniques do you use to make all work visible?
