Good engineering practices are the tools that help agile teams to deliver shippable products, argues Scrum and agile coach Mohammad Nafees Sharif Butt. Although many engineering practices have proved to be effective, they are not as widely used as they should be. Agile anti-patterns like the software testing ice-cream cone, accumulating technical debt and functional silos prevent teams from delivering a potentially releasable product at the end of a sprint, says Nafees. Anti-patterns can be addressed by inspecting and adapting regularly with the usage of systems thinking.
Nafees will talk about tools for creating potentially releasable product increments at SwanseaCon 2016. InfoQ interviewed him about agile anti-patterns and how to address them, why technical excellence is important, recommended engineering practices, and what organizations can do to establish a culture with good engineering practices.
InfoQ: What are some of the agile anti-patterns that you have see in organizations?
Mohammad Nafees Sharif Butt: From a technical standpoint, some of the most common agile anti-patterns that I have observed are:
a) Ice-cream cone test automation; where the majority of the tests are through UI and the notion of a proportional Testing Pyramid is violated. Teams/Organizations following this anti-pattern try to automate the majority of acceptance test using UI tests. These UI tests provide false negatives due to their nondeterministic nature, are costly to run and maintain, create mini-waterfalls and delayed feedback for teams and lead them to be inflexible towards future UI improvements.
b) The reckless accrual of technical debt is another commonly seen anti-pattern. This stems from the sense of urgency (that all product development groups find themselves in during this day and age of innovation) and the false assumption that we do not have enough time for design. A slight variation of this anti-pattern is also common where a code base is left to rot because the practitioners are neither following the craftsmanship nor playing the Boy Scout rule.
c) Equating continuous integration with using a build server; though tooling around continuous integration and continuous delivery has matured a lot in last 10 years, organizations continue to lose the essence of "continuous" in continuous integration. Long-lived feature branches, painful and marred with conflict merges are common symptoms of this anti-pattern.
d) Functional silos; not necessarily a technical limitation but these structural impediments have serious technical ramifications. Separation of business analysis from "development", stage-gating big architectural changes via an architecture group and N+1 design sprints are side effects of applying agile practices as cargo cult without understanding the underlying principles. One way or other, all of these result in waste - in both process and product.
InfoQ: How can you address such anti-patterns?
Nafees: There are two attack vectors to find a solution; first one is continued emphasis on Inspect & Adapt and the second one is System Thinking. Most of the anti-patterns in technical practices seen today stem from organizations' inability to see the forest for the trees and by making decisions that end up being local optimizations. A learning team/organization not only needs to have a holistic view, it also needs to understand the cause-effect relationship between its actions and the state of the system. Doing this inspection regularly and then adapting accordingly can help it address such anti-patterns.
A concrete example of this is a team I was working with a couple of years ago that started using Selenium web browser automation for acceptance testing to avoid manual regression testing every sprint for their greenfield product. Although they were able to automate the scenarios successfully, the shortcomings started to become apparent within a few sprints. During one of their sprint retrospective, the team analysed the situation and concluded that automation for the sake of it is not the objective and if these tests are delaying feedback (because they took long time to run), cause team to incur waste of rework (tests needed to be refactored when the UI changed), then an alternate strategy is required. An experienced practitioner helped by pointing the team towards subcutaneous tests which will test the API without relying on UI changes to be complete, would provide feedback faster and would not need to be refactored if changes to the UI are made. Though the retrospective stimulated an opinionated discussion among the members but they didn’t let their pride in the work that they have accomplished deter their focus on continuous improvement for the overall good and as a result were able to address the problem.
Sandro Mancuso explained why technical excellence matters in a Q&A about The Software Craftsman:
Agile was created to improve the way we deliver software. When we don’t focus on technical excellence, the quality of our software can drop to the point that it is very painful and slow to change it. At this point, it doesn’t matter which Agile process you have because developers can’t go fast anymore, causing the company to loose its agility.
InfoQ: Why is technical excellence so important?
Nafees: In the old days of waterfall we had the luxury of spending months in big up front design and architecture. We cannot do that anymore. Following the lake and rocks metaphor from Lean, the small and frequent iterations common in Agile have suddenly made ineffective practices painfully obvious. The promise of delivering incremental changes without freezing the scope up front is a big undertaking and lays its foundation on a strong product built with technical excellence in mind. Without technical excellence, under the burden of a rotting code base, the organizations would find themselves unable to keep up the promise of delivering client requirements iteration after iteration. Any such effort will reduce to become a lip service to agile and not a harbinger of true agility.
Take the concept of Clean Architecture, for example. The Dependency Rule not only ensures that the application is layered in a way that allows late just-in-time decisions but it makes it open for modifications and flexible to extend without involving nontrivial work (see OCP from SOLID principles for more). I have been involved in a healthcare insurance integration initiative where we employed Clean Architecture to incrementally create a gateway that eventually transacted 70% of healthcare insurance data in the country.
InfoQ: Which engineering practices do you recommend, and why?
Nafees: With my recent experience and focus on large scale product development, I would recommend the following two engineering practices (as starting point) that medium to large size organization really need to nail in order to stay ahead of competition.
The need to do Domain Driven Design cannot be emphasized enough (no, I am not pushing for Microservices, just plain old designing your product across seams defined by your domain and...). It is easier for start-ups to quickly deliver innovative ideas with a handful of engineers but when group size increases to a few dozen (or more), the coordination and dependencies between different components of the application by multiple teams start hurting an organization’s ability to deliver to changing market needs. A product architectured with concept of domain driven design can enable an organization even after it has crossed the chasm between a start-up and a successful business.
Another repetitive offender that I see is negligence towards non-functional requirements (Australia Census 2016 anyone?). In my opinion, if your product needs to work under a certain load with certain response time constraints, it is part of your functional requirements and needs to be considered from the get go (and not left for hardening and integration sprints). Same rigor of ATDD applied to functional requirements need to be applied to these so called non-functional ones as well. To be honest, calling them non-functional requirements and treating them as second class citizen only results in this aspect of a product to be ignored; ignored for long enough to be too late.
Earlier InfoQ interviewed James Grenning on TDD and code smells and asked him what can be done to increase the adoption of technical practices to increase technical excellence:
I think that the culture of solo work and specialization is in part at fault. If all you see is your own code, you’ll get the idea that it is good and there is nothing new to learn. Reviewing code late in the development cycle means improvement will be the hardest. Companies should encourage people to work together more closely. Pair programming is a great way to spread technical excellence, and it is a two-way street.
InfoQ: What can organizations do if they want to establish a culture with good engineering practices?
Nafees: Hire for future state. New talent almost always rocks the boat of existing equilibrium. When the current organizational culture around engineering practices is not mature, I would recommend hiring for target maturity over finding a fit for existing cultural. This way the new talent can be used to positively influence the status quo and act as catalyst of maturity.
Another important factor is to get external help. For most organizations, today’s architect is last decade’s programmer and with hierarchical structures in place it becomes difficult for organization to mature its engineering discipline even if there are enthusiasts among the ranks. Getting external help can trigger a reflection of organization’s current status even when it means challenging existing bureaucracy.
Lastly, iterative development can feel like a never ending vicious cycle if engineers are asked to implement a solution instead of given a problem to solve. Empowering the workforce, letting them self-organize (and even self-form) and providing the purpose behind it raises ownership and results in an engaged workforce. More than what traditional management would like to agree, the workforce treasures challenges and if mixed properly with autonomy and purpose they are willing to invest all their energies into it.
SwanseaCon 2016 will be held September 12-13 at the Liberty Stadium in Swansea, UK. The 2016 edition will be the second agile development and software craftsmanship conference for Software Developers, Software Architects, Project Managers, Analysts and Consultants in South Wales. InfoQ will cover the conference with Q&As, summaries, and articles.