Key Takeaways
- There are several internal and external DevSecOps at scale enablers; discover the right ones for your journey and link them to your outcomes and chosen approach.
- You will face multidimensional challenges throughout your journey, most of which are industry and context agnostic. Turn them into opportunities and ensure that you get the most out of your fundamental capabilities, even though those might not be in perfect sync.
- An End-to-End DevSecOps Enterprise Operating Model consists of numerous tactics, capabilities and frameworks. Tailor make yours, based on your context, expected outcomes and corporate strategy ambitions. Remember to continuously reconcile the big picture and build a coalition around it.
- There are several anti-patterns that replicate themselves in various industry and context agnostic DevOps implementations at scale. Avoid the pitfalls that we discovered at Danske Bank. Learning from the anti-patterns of others can save you time while scaling.
- Your scaled adoption will be a treasury of lessons learnt and you will be learning literally everyday. Are you curious to find out the core lessons we learnt at Danske Bank?
This article aims to provide an overview of the currently ongoing DevSecOps evolution at Danske Bank. Firstly, the DevSecOps evolution is positioned within the broader transformation that the firm is performing. Next, the main enablers and motivating factors of the evolution are outlined. Following is a selection of challenges discovered, providing a vivid picture of areas addressed as well as opportunities. The article continues with a high level overview of our new DevSecOps operating model, describing its main pillars. Concluding, anti-patterns we discovered are discussed, complemented by a not exhaustive citation of the main lessons learnt.
It is important to mention that the evolution is ongoing, looking towards the end of 2022. We are aware however that we will never be fully done with DevSecOps, as it is a continuous journey!
Is it a DevSecOps evolution or transformation?
In essence, it is actually more of an evolution, rather than a transformation, we are performing. DevOps as a concept has historically been adopted in the firm, and now we are defining its future evolution, reconciling it from a 360 degrees perspective; 360 degrees in the sense that we focus on people, operating models, technology, processes, culture and more. In other words, we take a well-rounded enterprise perspective.
The whole evolution is part of a broader transformation, which is called a Better Bank by 2023. The three main objectives of this transformation are 1) advance enterprise agility, 2) digitalize customer journeys, and 3) technology industrialization.
Main enablers
The main enablers for this evolution are several actually, and a combination of internal and external factors.
From an internal perspective, firstly it was our desire to further engineer and in some cases reverse-engineer our SDLC. We wanted to increase release velocity and shorten time to market, improve developer productivity, improve capabilities interoperability and modernize our technological landscape; in addition, find a pragmatic equilibrium between speed, reliability and security, while shifting the latter two as left as sensible into the SDLC.
Last but not least, we wanted to make the silos (business to IT and IT to IT) across the internal DevOps ecosystem collapse by improving collaboration, communication and a coalition on our DevOps vision.
From an external perspective, of course competition is a factor. In an era of digital disruption, time to market and reliability of services and products are vital. DevOps practices foster improvements on those quality aspects.
In addition, we as a bank are heavily regulated with several requirements and guidelines by the FSA and EBA. DevOps and site reliability engineering practices can support us to meet regulatory demand in an engineered and innovative way.
Main challenges in scaling DevOps
There were several main challenges; some were true challenges, while the majority were opportunities. However we definitely had a solid foundation of capabilities that were developed historically and we could build on top of them, by re-engineering in certain cases.
The main ones were (and still are to some extent):
-
DevOps misconceptions: Several times we found ourselves having different understandings and perceptions of DevOps concepts. Those led us to getting lost in translation on several occasions and basically appearing to disagree, while in essence we were agreeing. We had to eventually create a DevOps lexicon to make sure we talked about the same thing when referring to release velocity, self healing, policy vs process, and continuous delivery vs continuous deployment, to mention a few.
-
Varying maturity: We had a significant variation on DevOps adoption and maturity levels across the organization. The client-facing areas had gone quite far with CI/CD pipelines and cloud nativeness for instance, while other areas were in the bare minimum early CI/CD bar adoptions. Capturing these variations with data and baselining them was tricky. We had to know approximately how far each area was from our desired state so we could plan the evolution journey in a structured way.
-
Weak trust but verify: Our policy engineering and enforcement was not unified across our DevOps teams and technological ecosystem. Gap analysis, reconciliation and re-engineering were vital, while ensuring that “policy engineering” was embedded out-of-the-box in our technological ecosystem. It can happen that you follow a policy without realizing it, because it comes “out of the box”; compliance as code, in other words.
-
Capability fragmentation: We have been extremely innovative in enabling DevOps capabilities, though in several cases, the interoperability aspects could definitely be improved. Some good examples were enabling flawless CI/CD capabilities, enhancing the interoperability of the CI/CDs and our ITSM tool, while shifting operations and security left. A 360 degrees value stream mapping was essential.
-
Capability duplication: We discovered several capabilities that were outcomes of inhouse development of several areas individually. Standardization and harmonization in certain areas would deliver an impressive return on investment. Collecting these great stories around the bank and converting them to “as a service” central capabilities is a great opportunity that can improve economies of scale, reduce cost and technical debt, and improve operational efficiencies. It will also help us with internal job mobility, which is something we are investing in.
-
Build a DevOps coalition: This has gone simply great! But indeed, at the beginning we spent a considerable amount of time on getting the relevant stakeholders to, on the one hand, understand their role in materializing the DevOps model, and on the other hand, get their consensus that “Yes! Let’s do this”. This was maybe the most important aspect of the evolution.
DevOps 360 degrees operating model
The DevOps 360 degrees operating model is multidimensional, continuous, flexible, automated and above all else simple.
In a nutshell, it looks as simple as the below, while in essence is a little bit more complex, following the nature of enterprise DevOps evolutions.
Pillar 1 - Vision & WoW: This part is dedicated to the objectives we want to materialize, how we reconcile DevSecOps with the overall technology strategy (North Star we call it), as well as the reconciliation with our Enterprise Agility model, which is the Spotify one.
Pillar 2 - Capability Engineering: This is the heart of the model which is about simple words ensuring that the enabling capabilities are engineered, interoperable and reconciled. These enabling capabilities are purely from technological (CI/CD pipeline) to operational efficiency (ITSM engineering), as well as compliance related (SDLC controls). It is important to mention that we reconcile capabilities across the DevOps Ecosystem value stream.
Pillar 3 - Enablement: Enabling capabilities without having adopted is meaningless, hence the enablement part of the model is of vital importance. In this pillar we primarily focus on two aspects. Firstly, enabling the rollout in alignment and at relevance across the organization. For this process we use a model called TIES (Themes-Initiatives-Epics-Stories). A theme is a set of capabilities to be enabled and adopted in the organization, an initiative is the tactical adoption in the business Tribes, while epics and user stories represent workable backlog items, implementing the various initiatives. We follow up on the completion and benefit realization of our Quarterly Business Reviews. In speeding up the benefit realization visibility, we also invest a lot in analytics that can provide an almost “real time” visualization of the progress.
Secondly, we focus on uplifting our engineering skills and re-aligning our mentality and culture where needed. Hiring and incubating people with engineering skills, working in guilds to enable operational efficiency in harmony, as well as creating change agents to promote our evolution are some of the core activities.
The enablement pillar plays another vital role as well. It explains to the various utility and capability providers what their role in the DevOps puzzle is and supports maintaining the coalition as well as to checks the environmental temperature constantly.
Pillar 4 - Stay Relevant: The DevOps world moves fast and we want to stay on the edge. We read a lot, we spar with our selected vendors, we attend conferences collectively, we do as much as we can to stay relevant, inspire and be inspired. Attending the DevOpsCon Berlin 2021 is one of the ways we can do so.
Anti-patterns exist when scaling DevOps
Let’s go a little broader in the DevOps industry here and not only focus on Danske Bank. There are countless anti-patterns I have seen being repeated throughout my career in DevOps. The most obvious and repeatable are the ones below.
Anti-Pattern: Let the misconceptions be
Before starting implementing concepts and more importantly at scale, make sure you have a solid and broad understanding of them. There are several misconceptions that can lead to DevOps enablement and scalability failures. Three examples are:
- We have a CI/CD pipeline. We are “DevOps done”.
- We automate manual processes. We do SRE.
- My release velocity is 10 min. What more do I need to do?
Having misconceptions will have negative effects on your progress, and simple misunderstandings during meetings can lead you to more severe implications, like not being able to realize benefits collectively, hiring the wrong skill set, or non-compliance.
Anti-Pattern 2: Compliance is an enemy
Ok, I understand that compliance has a negative perception in most organizations. This can cause DevOps adoption to omit compliance requirements in several cases as they are seen as road blockers to production. Hence, you see several DevOps adoptions focusing primarily on speed. In my opinion, the saying that DevOps and compliance are enemies does not stand true. However, by having “compliance as code” practices complemented with a light “trust but verify” mechanism, DevOps and compliance become best friends.
Anti-Pattern 3: DevOps is primarily a developer’s job
Yes indeed, the developer is the starring actor of your DevOps adoption, but not the only one. Focusing only on developers simply breaks the value stream; it breaks the flow. DevOps above all else is about continuous flows and breaking the silos. There are several other stakeholders like operations, core infrastructure, security, enterprise architecture, and process governance that have a significant stake on your DevOps transformation. Map the relevant stakeholders to your DevOps ecosystem, and spend time on explaining and engaging.
Anti-Pattern 4: Irrational KPIs
We have all seen these right? 100% availability, 50% of regression tests automated, 10-min release velocity, automate to 0, 0 P1 incidents. Setting a good level of ambition is important; it drives behaviour, expectations, budget discussions and alignment. Nevertheless, going for isolated KPIs on the one hand, and unnecessary ones on the other hand, is a mistake. Instead, go for KPIs based on relevance in the business context, application specifics and evolution journey, as well as towards completeness by looking at your SDLC front to back. Having said the latter, try to balance innovation, reliability, security and compliance in how you set and measure performance.
What we learned
We have learnt a lot so far and we actually keep learning every day. Daily observations, corridor whispers, satisfaction surveys, benchmarking and feedback loops support us in checking the temperature, learning, adjusting and moving on. Below are the core learnings.
-
Focus on the people!
Engage, listen, explain, incubate. It is people who materialize your transformation and you need your engagement on different levels; from advocating concepts to hands-on contribution, and from decision-making to simply providing an opinion. It is very important to create an ecosystem of stakeholders who in different shapes and forms will contribute to different areas. Just to give you an idea, in our DevSecOps CoE we engage with 15 enablement areas, on top of our business Tribes.
In addition, it is very important to provide the people who are performing the “actual work” with the necessary resources to do their job. You can't expect people for example to start using your container platform efficiently if you have not trained them on containers, do not have migration and onboarding guides, and have not explained the compliance mechanism enforced on your container’s lifecycle.
-
Don't tell people exactly what to do!
By doing so you will increase frustration and kill innovation. Also, you cannot tell every single team how to do things; on the one hand, you do not have the capacity, and on the other hand you do not know their context in detail. Leave people with the space to innovate by focusing more on the outcomes they are expected to achieve. Of course, it is still important to provide them with patterns, frameworks and recipes that they can apply and build on top of, as well as to be prescriptive when it comes to compliance and security matters.
-
You are not Google, Netflix or Amazon
And you do not need to be at the end of the day. What I want to say is that it is ok to get inspiration from DevOps adoptions in big tech firms, but do not try to replicate their operating model, technology and solutions. Most probably you will end up trying to over achieve without decisive results. For instance, I am personally a big fan of the Site Reliability Engineering concept, but do we need to do everything Google does on SRE? Do we need to replicate the SRE tenets? Do we have global operations for 24/7 hours across the globe? The answer is no, we don't. Get inspired, yes! But adopt based on your context and needs by shaping and owning concepts in your own way!
About the Author
Spyridon Maniotis has 10+ years of experience in different roles within IT, both from the industry (Danske Bank & Nordea Bank) but also from consulting (Deloitte Consulting). He is well-rounded across DevSecOps, SDLC, Cloud Native, Technology Strategy, IT Service Management and Operations, Compliance as well as Agile Methodologies. DevSecOps transformations in the financial services industry, looking at operating models and capability enablement and adoption from a 360° degree perspective, have been his focus in recent years. Currently, he is leading Danske Bank's DevSecOps Center of Excellence. For him, DevOps above all is a philosophy that should be adopted at relevance and be focused on outcomes at an enterprise level. “DevOps is what you make out of it,” at the end of the day.