Oliver Hankeln, DevOps and Agile consultant, presented common anti-patterns on handling failure and how to avoid them at DevOps Days Ljubljana 2015. Hiding mistakes due to a predominant blame culture prevents organizations from effectively learning from failures and becoming more resilient.
Hankeln breaks this down into four recurring anti-patterns: hiding mistakes, engaging in blame game, the arc of escalation and cowardice. He then suggested corrective actions for each of them.
Organizations where financial benefits are tied in to team-specific outcomes (such as successful sprints for dev teams or system stability for ops teams) incentivize hiding any mistakes that negatively influence. This has a snowball effect as the lack of transparency spreads vertically and horizontally across the hierarchy (especially in larger organizations) to the point where higher management does not receive sufficient feedback to actually try to improve the state of affairs. Corrective actions include being open about incidents such as a service outage (both internally to peers and externally to customers), running a post mortem and sharing lessons learned afterwards.
The blame game anti-pattern generates a defensive attitude on all parties which reduces information sharing and handicaps the ability to learn from mistakes and unveil their root causes. Focusing failure analysis on events, not individuals (as in retrospective's prime directive), leads to acceptance of (fair) criticism and helps to overcome the blaming behavior, says Hankeln. He stresses that post mortems can't be coupled in any way to performance reviews.
Escalating complaints to managers is the anti-pattern that Hankeln calls arc of escalation. It exacerbates the blame game and destroys trust among peers and teams. Talking openly to your peers about issues is the correct approach. Building trust relationships and understanding between different teams' roles can be promoted for example by spending a few days working in the other team. It also helps avoid negative confirmation bias towards them.
Transparency handling failures does not mean lack of accountability. Hankeln's cowardice anti-pattern happens when organizations hide behind formalities and boards to avoid individual responsibilities. An environment where failure is accepted leads to higher accountability, improved levels of support and faster incident resolution, which are competitive advantages for the business.
A final recommendation from Hankeln is to conduct pre mortems (post mortems based on hypothetical failures) to improve system resiliency (from both technical and people perspectives), at least in projects with high risk.
Hankeln will be presenting the same talk at DevOps Days Paris on the 15th April.