Key takeaways
|
This is a true story, based on a talk from DevOps Days London 2016:
It was a gorgeous sunny spring day, my family and I were driving through my home town of Bristol, ready for another weekend adventure. We were cruising along when my wife said quietly: “I can smell smoke”. Now, I’m a good mechanic, I used to restore classic cars. The car we were driving was modern and recently serviced. I checked the instruments, everything normal. “It must be outside”, I declared confidently. Two minutes later tentacles of smoke were curling around my ankles and my shins were getting remarkably warm.
We can learn something relevant to DevOps, and many other disciplines, from this experience.
Don’t let your own expertise be a blind spot.
When people are regarded as experts in a field, they are likely to be called upon to provide answers quickly. Humans are very good at this. Instinctive answers engage what has been called ‘system 1’: a rapid fire but slightly lazy part of the brain. It tends to reach for the easiest answer to hand, useful in a fight or flight situation, less helpful in a meeting of minds. System 1 is designed to front our ‘system 2’ which is relatively rational, but needs time to figure things out. The relationship between System 1 and 2 is like the relationship between a cache and a server request. Sometimes though when the cache can’t find an item, instead of calling the server, it just offers the quickest thing it can find. Many times our quick instinctive answer, based on years of experience, will be correct. Other times, probably when it’s most embarrassing, it won’t.
In problem solving situations and thought work we need to learn to anticipate, and reflect on, this initial, instinctive answer. A useful technique is to challenge yourself to look for a second answer, to try to refute your first. As we leap to the nearest conclusion we often ignore useful input from others, particularly if we regard them as less knowledgeable than ourselves.
The curse of knowledge speaks of a similar challenge, how to relate to people new to a field. I believe this ‘Curse of being good at stuff’ is similar, and more focused on the tendency to jump to conclusions when we feel pressure.
We swiftly pull the car over. We are in one of Bristol’s less than salubrious areas. In fact it has a reputation for tracksuits, hoodies, aggressive dogs, caps and numerous police vehicles. I step out of the car and I’m immediately confronted by a gang of youths who say: “Get your &^%*&^% car out of here, you can’t park there”. I reply “Look, my car is on fire, I want to get my family out and then we can talk”. The change in attitude was astonishing. “I’ll fetch a hose”, said one. Do you need a hand? says another.
What can be learnt from this? As DevOps practitioners we are often behaving in ways which seem unconventional, baffling or threatening to others. From outside the lads couldn’t see the fire, they couldn’t guess our context. What they saw was an ordinary car roar into their space and a stressed stranger jump out. I took time to explain the situation and they listened. Once the situation (and my motivations) were understood, they made a judgement call to help.
People are more likely to assist when they understand your motivations.
This is something I see frequently when people are promoting new techniques, be it agile, DevOps or a fresh technical approach. The new concept makes perfect sense to the promoter – in fact it’s merits are so obvious, they don’t explain context and motivation to potential adopters. Unable to understand or develop empathy, people will often resist or blithely continue with their habits.
The value of this kind of empathy cannot be understated. It applies both ways: if we as promoters of change expect to be understood and listened too, then we too must understand the feelings and challenges of others.
So the family are clear of the car, the heat has intensified and flames are coming out of the bonnet. It’s time to put the fire out. I go to the boot where the extinguisher is stored – it doesn’t open. The molten blobs of electrical insulation under the car provide a clue as to why. I take a deep breath, dive into the car and grab the extinguisher. With the flames growing I unwrap the extinguisher and leap into action – by reading the instructions. I fiddle with the pin and point the extinguisher, the foam isn’t going in the right direction. Just as I get it to spray in the right direction it sputters and stops.
My fruitless attempts to use a tool I’m not familiar at the time it most matters tell us something about our approach to unexpected situations:
Don’t just plan for disaster. Expect it, practice for it.
Operations groups are pretty good at planning for disaster, and disaster recovery plans are recognised good practice. This calls to mind the classic boxing quote: ‘Everyone has a plan until they get punched in the face’. Typically efforts focus on production systems. These alone are often not enough. Ancillary systems, with their ownership sitting in the uncertain space between Dev and Ops, are instrumental to rapid recovery. There are plenty of stories of systems recovering their compute, but being unable to recover further due to their configuration management or deployment tool servers not being available. The duration of Github’s recent outage was increased by the loss of their chat servers, something they rely on to understand system state and to collaborate in the event of an emergency.
We call the fire brigade, thankfully they arrive promptly. The team of four exchange very few words and yet they move swiftly with a sense of purpose, each contributing in a different way. Hoses are selected, traffic is redirected, a water supply is found, people directed to a safe distance, the hoses are turned on. My car drenched in gallons of water. It’s clear that even if I had practiced, all the fire extinguisher would have done is buy us time. “You need to understand the fire, see?” Says one of the fire fighters. “It’s deep in the bulkhead, very hard to reach.”
So how are firefighters so good at what they do? For one thing they follow the previous advice, they practice for disaster, they are well rehearsed. Firefighters also learn quickly. One way they learn is through investigation of failures.
Failures are rich in learning.
The approach of Western cultures to failures appears to need a massive rethink. The DevOps and agile movements lead the way, encouraging a recognition that triumphs and failure alike are opportunities to learn and improve. You are likely to be familiar with root cause analysis, retrospectives, and post-mortems (blameless and otherwise). These all encourage learning from recent events and commitment to making improvements.
We should also inspect our own reaction to failure. Do you look upon them as opportunities to learn? Or are they explained away with self-comforting logic? I often think of Danny MacAskill, a Scottish trials bike rider. He practices the same jump over and over, falling hard on many occasions, until he gets it right. Each time, each iteration, brings him closer to his goal.
The approach organisations take to failure is so significant that Westrum’s Topology of Organisational Culture uses the treatment of messengers (those who often bring news of failure) as an indicator of the type culture an organisation has built. Where reporters of failure are encouraged to speak up, their concerns investigated and corrective action taken you’ll often find a high performing organisation.
Finally we are back at home. We try not to think too hard about what happened, or what could have happened in very slightly different circumstances. We upload a few photos, friends sympathise and make summer BBQ jokes. Then, months later we are contacted by an owner of the same type of car. It transpires that this model has been suffering spontaneous fires all over Europe. None of the drivers know each other, but there are enough to form a community, enough to support each other, enough to get the attention of the manufacturer, enough to engage a lawyer, enough to prompt a global manufacturer to issue a recall. All because a few people shared their photos, and cared about the cause enough to connect with each other.
If it matters, share it, you never know who will benefit…and it could be you.
Sharing is another tenet of DevOps: share knowledge, code, metrics, styles, approaches patterns. Often though we share what interests us, what we’ve been asked to, or what process demands. I encourage what I term ‘deliberate sharing’. The act of sharing things because they might be interesting, or rather than having your default setting as private make it public. This includes the things you tried that didn’t work as well as those that did. Science has long recognised the value of this approach and many tangential findings have led to useful applications. Blogs, Kanban boards, open meetings are all examples of sound sharing practice. All these invite serendipity and others to build on what you do.
So that’s the story of the day my car caught fire, and the things I learnt about DevOps from the experience. It may be instructive to read those learnings once more and note that they are highly applicable outside of the IT world. This is one of the elements that draws me again and again to DevOps. Through it, and the willingness of the community to share, we may learn skills which serve us well in other aspects of life.
About the Author
John Clapham is an independent coach, trainer and consultant. Offering considerable experience in Continuous Delivery, DevOps and Agile, he helps teams to build great products, creating an environment which is effective, productive and enjoyable to work in. His broad experience in software development ranges from start-up to enterprise scale, formed in the publishing, telecommunications, commerce, defence and public sector arenas. Fuelled by coffee and Bristol’s inclement weather, he occasionally surfaces as @JohnC_Bristol, and has been known to blog at johnclapham.wordpress.com