DevOps is a movement. DevOps is a mindset. DevOps is devs and ops working together. DevOps is a way of organizing. DevOps is continuous learning.
The intangibility of DevOps makes it hard for leaders to come up with a clearcut roadmap for adopting DevOps in their organization. The meaning of DevOps is highly contextual, there are no popular methodologies with prescribed practices to abide by.
However, healthy organizations exhibit similar patterns of behavior, organization and improvement efforts. In this series we explore some of those patterns through testimonies from their practitioners and through analysis by consultants in the field who have been exposed to multiple DevOps adoption initiatives.
This InfoQ article is part of the series "Patterns of DevOps Culture". You can subscribe to receive notifications via RSS.
Fire Service applied to DevOps
Before servers and services became my job I worked in the fire service. During that time I received one of the best pieces of advice ever given to me even though it wasn’t offered as one. It was a challenge from one of the captains in the fire house… “What did you do today that you would be proud of?”
The fire service succeeds because it has a culture that’s balanced between pulling your own weight and being team oriented. There is a natural pressure to not let the rest of your people down and you do that by holding yourself to the highest standard possible.
Compare that with our profession where we place the focus on technology and process. You may think that attention makes sense as it’s focused on the tools we use to do our job, but there will always be some new technology around the corner to learn.
DevOps focuses our attention on the differences between the teams that make up an organization. The name itself highlights the tension that has existed between development and operations. It encourages empathy by forcing us to address communication and understanding the needs between the various teams that at times appear to be at odds with each other.
people, ideas, technology, in that order – John Boyd
Every few decades or so some new technology or process is brought along as a way to reduce the number of fire fighters needed to protect a community. But in the end it has always come down to the same fact. To do the job you need the people. And when you need people you have to have the soft skills to motivate and lead them. To have a truly successful team you need those skills in place even before you think about unleashing them on the work ahead.
Leadership, mentoring and team chemistry are the points for creating successful teams. As a fire fighter I was lucky enough to learn these by example from some amazing people. These ideas were not just part of our culture, they were expected and they’re what I want to share with you.
Leadership
Any team requires both members and leaders. Nothing earth shattering there. But what makes up a great leader? Most of us can probably think of someone who they thought of as one, but we probably don’t give too much thought into what made them a great leader.
In my view they can be described as someone who is consistent, fair, honest, direct, and aware of what is going on. The consistency provides a dependability that will help anchor a team. The fairness, honesty and directness is respecting the team by treating them as professionals and providing clear communication. Awareness provides feedback on both the impact the team is making and from outside sources back onto the team.
What we cannot have is the same pace of change and chaos that is common in the work we do reflected in the teams themselves. High turnover, micromanaging, and a high tempo of change in the daily or weekly goals are each things that will kill a team.
To counter this the leaders in each team who step-up to take on different roles need to provide stability. This include the official (tech leads, managers, and directors) and the unofficial (experienced engineers who’ve earned the respect of their peers) leaders. Teams only succeed in chaos when they’re founded on stability. Otherwise they’ve either just been lucky or they’re heading towards burn out.
John Boyd described it as essentially getting people to take action to accomplish uncommon goals. The idea being that they’ll of course accomplish their common goals on their own. – Chet Richards on John Boyd’s description of Leadership
As a leader you become responsible for your team. Their needs, problems, and failures become yours, but just remember that their successes are their own. It means that you will need to be a good listener while also providing honest and direct feedback. You will need to provide a calm and positive outlook on the work at hand no matter what difficult challenges you are facing. Your job becomes not keeping the system, site or service up, it becomes keeping your team up.
In the fire house I saw two easy ways to present that calm and positive outlook. And like most things from civil service they get broken down into a short phrase.
The first is “complain up and praise down”. You should never complain about anything in front of or to your team. You’ll only end up looking like a complainer and people will take on a more negative opinion of you. Instead if you need to raise an issue, bring it up with whoever you report to. This also leads to another common attitude that you take care of your own problems. Don’t let your actions or inactions cause anyone else to needd to “complain up”.
The second is “praise in public, reprimand in private”. Anytime you thank someone for something they did do it publicly. This can be anything from an email to the team or an “at name” message in a chatroom. But if you need to bring up an issue do so in private.. This can lead to times where help from the team is needed to deal with an issue. Anything from providing additional guidance on a technical task to helping someone adjust to working on the team. This is just an extension of the ideas of taking care of your own people and as a team handling your own problems.
The other important role a leader performs is clearly communicating to everyone the current situation, the challenges ahead and the goals for not only the team, but for the entire organization. This provides the common orientation needed if you want to be able to take advantage of the initiative of individuals to reach your goals. If you have a team of capable people and have this type of clear and transparent communication going in both directions they should be able to prevail against almost any problem they face.
Team Chemistry
One of the best descriptions I’ve seen of what this looks like is from Gene Kranz.
There, we learn the difference between the ‘I’ and the ‘we’ component of the team, because when the time comes, we need our people to step forward, take the lead, make their contribution, and then when they’re through, return to the role as a member of the team.” Kranz then went on to explain the “chemistry” aspect of a team: “it amplifies the individual’s talents as well as the team’s talent. Chemistry leads to communication that is virtually intuitive, because we must know when the person next to us needs help, a few more seconds to come up with an answer.
— Apollo 13 Flight Director Gene Kranz speaking at the National Air and Space Museum on April 8 2005
The fact that this type of chemistry has to be cultivated over time is really a completely separate subject. But the short of it is you can’t have a team that performs at that level without having them do the work, responding to outages, and training together. That last bit is something that’s sorely missing. An annual retreat or a hack day alone is not sufficient to address these points. You have to have them do the work together, encourage team lunches, and opportunities for them to share their knowledge between themselves.
This also means that when there is work to be done to let them do it. Good leaders are willing to let other peoples ideas take the lead. Let that unique mix of skills and backgrounds of a team actually take form.
I have a very specific memory of a motor vehicle accident where a college aged drunk driver hit a couple driving in a pickup truck sending them off an overpass and landing upside down on the railroad tracks below. It was a horrific accident and I was on the first of the two fire engines that arrived along with a chief and the other emergency services (EMS and Police). The couple didn’t survive the crash on impact, but we still needed to remove them from the vehicle. In most fire departments the color of your helmet indicates your role. You’re on an engine or a truck. You’re an officer or you're a chief. What I remember is a bunch of white (chiefs) and red (officer) helmets surrounding the truck trying to extricate the couple from the pickup with about half a dozen fire fighters standing watching them do the work. Let your people do the work.
Mentoring
The other way to help build that chemistry and to keep the institutional memory of an organization going is to make mentoring a key part of your teams.
This includes mentoring new members and also existing people who have stepped up into more senior or different roles. Once you’re no longer the new person doesn’t mean that you shouldn’t be the mentoree anymore.
In the fire service this is a major part of bringing in new people. It provides support and introduces them to their team. At the same time this allows more experienced members to pass on to the next generation the things you can only learn by doing the job.
There is already a bit of inertia that you will be fighting against that actually discourages all of this. As a profession we tend to stay in our perspective technology silos without stepping outside to see how other people or tools have approached similar problems. We end up disregarding the results of other smart people who already addressed the issues we’re working on. Then we have ageism continuing to bleed out knowledge from some of our peers who just happen to have years of experience.
Communication
Like any successful relationship you’ll need to have honest and direct communication. Otherwise you will be creating a false environment that both your people and yourself will be working in. If there is an issue bring it up directly. The worst thing you can do is tell someone there is nothing wrong when in fact there is an issue.. In the fire house we were more than just a team. We were in effect a family and we wouldn’t let any issue, hurt feelings, or complaint go unaddressed for too long. Things would either be dealt with out in the open between everyone or it would be addressed in private. If you don’t deal with the small stuff while they’re small you risk allowing it to fester into something much bigger. Something that could require a more drastic change to resolve it.
Each of these ideas are not only applicable to our individual organizations, but to our field as a whole. Events like DevOps Days are amazing opportunities for us to share our experiences and help mentor someone along for the little bit of time we have with them. Just remember to not pass up the chance to get some mentoring of your own while you’re there.
Recap
If you find yourself wanting or needing to improve how your development team works you will need people to lead and champion those changes and mentors to share these ideas with your people. If you don’t then it will become just another attempt to use a different approach (think Agile or Lean) to improve the output of the team. And I think many reading this have unsuccessful experiences when those types of changes come down from management as a “We’re doing this now. Here is a class or book. Now make it work.” approach.
The point is none of this is magic or new. It’s about doing the work needed and treating each other with respect. Our problem is we make things way more complicated than they need be and we routinely fail to see the obvious in front of us.
If you like this type of approach I’d suggest reading almost anything written by Leo D. Stapleton (retired Fire Commissioner and Chief of Boston Fire Department) or on Colonel John R. Boyd. But the most current example of this is “Team of Teams: New Rules of Engagement for a Complex World” by General Stanley McChrystal. Each of these provides examples of teams succeeding in some of the most chaotic and demanding situations you can imagine. All of them provide examples of teams taking a more holistic attitude where more top-down approaches have failed.
One final comment that I want to point out is these ideas don’t exclude self-managed organizations. The fire service is very structured. Pretty much you can think of it as paramilitary, but it’s made up of several self-managed teams who each have a fair amount of flexibility. You could say they are the pinnacle of the Results Only Work Environment and self-managed teams. As a fire fighter I never needed to look at a Kanban board or have a task assigned to me to know what needed to be done. The fire service along with the other emergency services succeed by taking care of their people with these ideas, after that everything else usually just falls into place. Letting them answer the question “What did you do today that you would be proud of?”
About the Author
Michael Biven is a lead systems engineer at Ticketmaster and a former firefighter who has taken his experience in leadership and emergency management from the fire service and put it to use in technology since 2000. Before Ticketmaster he helped build amazing experiences for anyone and everyone at Ticketfly, Change.org, Beats Music, and Apple.
DevOps is a movement. DevOps is a mindset. DevOps is devs and ops working together. DevOps is a way of organizing. DevOps is continuous learning.
The intangibility of DevOps makes it hard for leaders to come up with a clearcut roadmap for adopting DevOps in their organization. The meaning of DevOps is highly contextual, there are no popular methodologies with prescribed practices to abide by.
However, healthy organizations exhibit similar patterns of behavior, organization and improvement efforts. In this series we explore some of those patterns through testimonies from their practitioners and through analysis by consultants in the field who have been exposed to multiple DevOps adoption initiatives.
This InfoQ article is part of the series "Patterns of DevOps Culture". You can subscribe to receive notifications via RSS.