Key takeaways
|
When starting the DevOps journey in our organisation, we found that there is a lot of information on technologies and even a lot about the culture, but there is little information on adopting DevOps in large distributed organisations and the associated enterprise specific challenges.
In this article I talk about my experience approaching DevOps for the first time in 2014 on site at a large insurance company.
Every story has a hero. In this story it is the monolith. Let’s start with a couple of definitions of monolith:
Definition one: A great big rock.
Definition two: A large impersonal political, corporate or social structure regarded as indivisible or slow to change.
Controversially this definition could be applied directly to some of our large distributed organisations. Our organisations are very political. I love the phrase "slow to change". What strikes me in this sentence is, it does not say "unwilling to change" or "won't change". It made me think what are the features of our organisations that make them monolithic? What creates that viscosity and resistance that slows down our change so much that it nearly seems like we cannot change?
We now design our organisations’ model, roles and applications not to be monolithic. We favour modularity - but our organisations have existing monolithic features that need to be considered and understood in order to enable change.
It's all about behaviour
A video on the transformational changes that happened in Yellowstone park due to the reintroduction of a pack of wolves has been making the rounds of social media.
Wolves went extinct in Yellowstone in the 1920s. In 1995 biologists reintroduced a pack of 14 wild wolves in the park with the goal of regenerating the wolf population. The outcome was miraculous. The park flourished with the return of animal species, flora and even the rivers changed direction.
Previously, when the wolves went extinct, deers and elks over grazed in the park, eroding many areas. On the return of the wolves, the deer and elk stopped grazing in certain parts of the park where they could be easily hunted by the wolves. Indeed a short while into the clip the scientist excitedly explains that the deer and the elk changed their behaviour. Programs, arranged by humans - culling the deer and elk - did not have an impact. Indeed even the number of deers and elks killed by the wolves themselves did not have much of an impact. It was the behaviour change that did it!
Like Yellowstone, our organisations need transformational change. Like Yellowstone, it requires behaviour change.
I believe, with behaviour change you need to lead by example. Before you look at the behaviour of your teams, you need to begin with your own behaviour. Before you can change that, you first need to understand why you behave the way you do.
Characteristics Influencing Behaviour Change
In my large FinServ organisation these are the characteristics that influenced my behaviour:
Outsourcing
We outsourced a lot of our IT. Both application and infrastructure. This very much influenced how our teams were constructed. It influenced how I designed and delivered applications.
Global IT Standards
We had global IT standards that dictated what technology we used for a specific technical capability. This influenced how I designed applications and with what technology. It influenced how my solutions were governed and approved.
Business Units
We had 50+ business units, with some IT autonomy to respond to local demand or regulation, this influenced how we behaved at group and business unit level. It also cultivated a shadow IT culture. These shadow IT teams can perform very well in isolation. As these teams frequently use non standard technologies, methodologies or suppliers, the success and culture of the team exists in isolation within the business unit and is typically not replicated across the organization. Sharing and collaboration does not happen wider than the team, and frequently the team feels under threat of being “shut down”. They achieve success only in a vacuum.
Near-shore and Off-shore Teams
Whether using near-shore or off-shore teams, this influenced my behaviour as it affected how I communicate.
Different Dedicated Departments
In large enterprises we often have different departments with different mandates. There are many examples: portfolio management, sourcing, development and operations being just a few. These are departments with a dedicated remit which "own" certain activities or functions. They affect how we behave, who we engage, how we engage and when.
These are a few features of a large distributed organisation that I have worked with on a daily basis. There are many more. Not only do they influence how we behave, but they do not lend themselves well to a global DevOps or Continuous integration & delivery strategy.
To enable behaviour change - which will bring about culture change - we need to understand these characteristics of our large distributed organisations. These features and characteristics are our true monoliths.
Monoliths Influencing Culture Change
These are the monoliths I encountered while owning the delivery of a strategic initiative for my large distributed financial organisation:
Us vs. Us
A financial product is not a commodity product like a car or a handbag nor is it like Twitter or Facebook - a platform you engage on, reach out and connect to people on. Insurance is something you purchase and then you put it in a digital drawer or physical drawer and forget about it, until it needs to be renewed or until you need to use it - because something has happened to your car, your business, your health or your family. Therefore insurance companies are about mitigating risk, about protecting customers. Banks have the same challenge. When you are about protection and protecting customers, many departments crop up in your organization around protection. This might be budget protection, data protection, security protection, system protection etc. These teams are typically “always right”. They are right because they are protecting the customer which is what the organization is all about. Have you ever noticed how difficult it is to negotiate with someone who is always right? (You are thinking about your partner now, right?)
DevOps and agile initiatives (like Scrum) are being more widely adopted in our organisations. However, to successfully enable these methodologies and way of working, it is important to engage the teams and departments in the company that are responsible for "protection". Like your partner, these are the teams where you need to show the love. Without engaging these teams you are working against your own group and run the risk of creating an “us vs. us” environment.
To enable DevOps to be successful in my organisation I had to engage with many such groups. I will specifically mention the security group. Most (financial) organizations have a dedicated security group. I went to my security group with a DevSecOps story. However their big concern was around testing. How do you continuously evolve an asset in production while maintaining traditional organizational milestones around testing. To get engagement from group security we had to examine and explain our product testing and test driven development strategy more closely.
Engage your security, legal, asset management and operations release management groups. Explain what DevOps is but most importantly listen to their concerns. Their remit is important to the success of the organization. Addressing their concerns leads to buy-in. It will enable the wider organisation to adopt DevOps instead of creating DevOps in a vacuum.
Project vs. Product
Over time the platform my team was building started to mature and achieve excellent velocity. I then realized that my organisation was fully structured around projects. We had application teams building a project, go-live, and then hand the asset over to a support team. The asset was in Business As Usual (BAU) state. We had project managers, program managers, portfolio managers. All governance pivoted around projects.
As my team adopted DevOps to become more and more autonomous we moved from building a project to building a product. We were not going to hand it over to go into BAU state, but rather continuously evolve it. A project focused organisation is not structured to support this. Even simple tasks like securing funding or getting approval at stakeholder meetings became impediments due to this monolith.
Addressing how your organisation does governance and portfolio management is key for DevOps adoption outside the Scrum room in large organisations.
The Supplier
Many organisations use suppliers. Some outsource everything, others only certain functions. I experienced first-hand what happened when an organisation heavily depended on a supplier but they were not engaged with teams working in a DevOps way. Our cycle time went from 2 days to 3 hours thanks to adopting DevOps. Mainly by evolving our toolset away from restrictive global standards to a complementary mix of tools and technologies that supported our test driven development strategy and enabled continuous integration and delivery. Our asset went live, and the business was very satisfied. However, once the asset was in production for a couple of sprints, some changes were introduced around our automated release processes. All the automation remained but the supplier added their manual processes around the automation, postponing releases to production.This brought the cycle time back up to 2 days.
In large organisations suppliers often have restrictive contracts that dictate how they and the organisation engage. In this case, the supplier was being penalised when there were incidents in production. Of course, the supplier was going to be all over any team that had access to production servers.
We hear a lot about moving from Mean Time Between Failure (MTBF) to Mean Time to Recover (MTTR), but such a transition is not possible if a team, person or supplier is being penalised due to issues in production.
DevOps does not mean that you should ignore supplier management. Should your organisation have suppliers, understand their contracts and how they influence the supplier’s behaviour. How can you adapt your organisation’s behaviour to enable the supplier to support or allow your DevOps product to evolve in production? For example, contract notes can be raised on vendor contracts removing stringent penalty clauses. This is a conversation to have with your wider IT management team and strategic sourcing. It has to be an organizational decision to make such a change to enable the supplier to come on the DevOps journey with you.
The traditional application monolith
I believe, in the future there will not be much of an enterprise’s portfolio that needs to remain monolithic. Bi-modal and two speed IT should not have to apply to much of an enterprise’s application portfolio. That said, there could be monolithic applications in the organisation that will not wish to adopt DevOps or agile initiatives.
Some applications will remain monolithic in nature as they are meeting their objectives and are successful (or perceived to be!). If your product is successful it is likely that your backlog could expand and introduce stories that will need to engage with such an application monolith. This can cause problems, especially in the area of testing where test data from the monolith may not be updated as rapidly as the DevOps product requires. Solutions vary based on the application in question (automation, moving payload to a QA cloud, strangler architectural pattern, etc). However, as the owner of a DevOps-driven product, you need to understand your enterprise architectural landscape in which your product lives, and the potential two-speed-IT style monoliths that you may have to interface with.
Conclusion
This is the famous scene from Space Odyssey, the monkeys scrambling around the monolith:
Avoid creating DevOps in a Vacuum in your organisation by understanding what influences your (and your peers) behaviour. These are your true monoliths. Once understood you can start looking into changing them. By changing them, you enable pro-active behaviour change, both your own and your peers. This behaviour change will bring about transformational change in your organisation.
About the Author
Margo Cronin is an open group certified master architect and certified program manager. She worked for 10 years for IONA Technologies as a middleware consultant before moving to the FinServ sector in Switzerland. There, over the last 10 years, she has worked for Credit Suisse and Zurich Insurance in both delivery and enterprise architecture roles. Margo Cronin started working with DevOps to enable scrum and agile to be successful in large distributed organisations. She is co-founder of an enterprise architecture platform in Switzerland called EntArchs.
