DevOps Enterprise Adoption at CSG International with Erica Morrison

Following up on the series of Q&A’s InfoQ has published in the last years about DevOps adoption at the enterprise level, we spoke with Erica Morrison, Director of Software Development & Operations at CSG International.

Morrison will once again share her organization’s current state of  DevOps transformation at the DevOps Enterprise Summit in San Francisco, later this year.

We took the opportunity to get to know a bit more about their DevOps journey.

InfoQ: Can you tell us a bit more about your current role?

Erica Morrison: I’m a Director of Software and Operations at CSG International. My teams provide capabilities including our telemetry system, build system, load balancer, and common, foundational .NET libraries.

InfoQ: How and when did you first hear about DevOps?

Morrison: I don’t remember the exact moment I first heard about DevOps, but one of my teams has always been a DevOps team. We expanded this to another one of my teams, the team managing the telemetry system, and that was the first time I was really exposed to the power of DevOps. I got to see firsthand the improvements we gained from joining Dev and Ops together – things like faster deployments, better feedback loops, and a more maintainable product. We didn’t yet know all of the DevOps best practices – we simply merged the teams and let things evolve. These teams were some of the earliest adopters of DevOps at CSG and my experience was very positive, which helped drive my desire to do more.

InfoQ: How did DevOps get started in your organization? What were the first steps taken and why?

Morrison: In addition to the DevOps early adopters mentioned above, we had support from a senior leader, Scott Prugh, who is now the Vice President over our DevOps organization. He has been a major driver in this initiative. Our journey is several years long. In the early days, we looked at things such as moving teams to agile practices and reducing batch sizes for releases. In 2016, we reorganized our dev and product operations teams to align under one org. DevOps initiatives have accelerated since this reorganization was done.

InfoQ: Was it mostly a grassroots movement? Or was there a top-down understanding that DevOps was needed?

Morrison: The effort was really spearheaded at the VP level.

InfoQ: Which DevOps initiatives are currently going on at your organization? Did they involve organizational changes?

Morrison: We’ve expanded our DevOps efforts substantially this past year. A few key initiatives are:

• Infrastructure as Code – We are rolling out Chef across the company and are looking to replace manual operational tasks with automation wherever possible
• Cloud – We have begun building our private cloud and have done proof of concepts in AWS as well
• Decentralizing change – We have begun holding “Local CABs” where details of each change are discussed and reviewed with team members. Only senior leaders attend “Enterprise CABs”
• Telemetry system evolution and scalability

InfoQ: How are you planning (or already working on) to disseminate DevOps in the larger organization?

Morrison: We have already disseminated DevOps as part of our reorganization. However, we spent the last year being so busy adjusting to the new world and tackling problems, that we didn’t spend a lot of time on the culture change. We are now going back and revisiting culture to ensure the entire organization understands the need for DevOps, as well as the key concepts that are most important to us. We’re now holding a monthly DevOps Leadership session where we bring together leaders from across our DevOps organizations, as well as organizations that we partner with including platform architecture and product management. We’ve aligned around service owners for each of our products and are building metrics to visualize improvements.

InfoQ: Have you witnessed any culture shocks, for example from risk management and/or security/compliance teams?

Morrison: I think some of the shock came because we jumped in and started doing, without fully ensuring all teams understood why we were making changes and how they could be successful. While some teams fully adopted new techniques, we saw others where there was less success. We are addressing those concerns now and I’d definitely recommend focusing more on this up front. It needs to be clear not only the “why”, but the “how” – what are the DevOps concepts that are most important to my company? Where do I start?

InfoQ: Which other cultural challenges have DevOps initiatives faced in your organization?

Morrison: Sharing the DevOps culture mindset with those outside our core DevOps organization.

InfoQ: What have been the greatest achievements and failures in your organization's DevOps journey so far?

Morrison: We have made substantial strides in a number of areas. Investments have been made throughout the enterprise in measures to increase the stability of existing systems. This includes a wide range of solutions such as getting away from legacy technologies and re-architecting solutions to better enable incremental change. When we do have issues, solutions are often more holistic in nature versus band-aid local optimizations of the past. We have made substantial investments in Infrastructure as Code and have started using this in production for a few key products. Our change process is evolving to put more information closer to the teams implementing change. Detecting issues earlier and being able to quickly rollback are now key elements of our change approach. There has been a large growth in synthetics and other automated tests used by teams to improve feedback loops on system health. Some of this data is being fed into visualization tools.

Overall, we have implemented many changes that are reducing risk and increasing stability. However, we’d like to see key metrics values improve at a faster pace. We sometimes take two steps forward, one step back as we continue to unveil new sets of problems as we address others.

InfoQ: In both cases which were the most important factors from your point of view?

Morrison: Enabling the good people we have with the right toolset and the right vision is instrumental to success. People need to understand where we are going and why, so they can be personally invested in the journey to accomplish our vision. The way we are accomplishing work is changing, which can mean new skills are required. We need to ensure everyone has access to proper training to gain these new skills.

When we have failures, we need to inspect what can be improved and ensure we follow through on identified actions with appropriate priority. It’s also important in failure scenarios to ensure we are supportive of our people. If a bad decision was made, what information was the person lacking when making that decision that would have made them more successful? If an error was made, how does the system protect from these errors, etc. These improvements should focus on things that can be changed vs blaming people.

InfoQ: The State of DevOps Reports suggest that investing in DevOps and Continuous Delivery practices leads to faster, more reliable delivery of business value. Do you agree? And if so have you come across concrete examples in your organization backing up that claim?

Morrison: We’ve seen investing in Continuous Delivery concepts such as Infrastructure as Code greatly enable us to reduce risk. There is a lot of value in a repeatable deployment that has been fully tested and code-reviewed ahead of time. We are already seeing this automation improve the success rate of our maintenance. Eventually, we do believe we will be able to move faster as a result.

However, there are hurdles to overcome in implementing IaC that have slowed us down before we can speed up. There’s the obvious one-time cost of converting all manual configuration into code. There are learning curves. As you start to move faster, you run into new, unexpected challenges that you have to design solutions for before you can move forward. While IaC has many advantages, if you are in the middle of an outage and could previously make a config change in five seconds to alleviate the outage, it’s tough to now go to a different model where code needs to be checked in, built and deployed. What’s an acceptable timeline? What does our “break-glass” look like and how does that align with our normal process as much as possible? I view these are exciting challenges to face because they are things we couldn’t even have foreseen without the progress we’ve made to date.

InfoQ: What kind of metrics or feedback are you collecting in order to validate the value (or reduced waste) accrued through your DevOps transformation?

Morrison: Reduced risk and increased stability are key drivers of our DevOps transformation, so providing metrics in those areas allows us to see how we are progressing towards our strategic goals. A few key indicators of that include:

• Success rate of change
• Number of incidents
• Outage impact to customers (includes analysis by product and duration)
• Mean Time to Resolution (MTTR)

InfoQ: What challenges and road blocks lie ahead in your organization's DevOps journey?

Morrison: The two biggest challenges we face are determining where to prioritize our people and growing our platform vision. As with any organization, we have a finite number of people available to work initiatives. In a lot of cases, we understand the work that needs to be done to improve our systems. This work includes things such as growing our automated test suite, building out Chef cookbooks, decoupling architectures, and improving telemetry. As teams balance these needs with the ongoing new features and functionality requests, it’s challenging to determine how to best target team members’ time to both satisfy customer requests and build safer, more stable systems designed for speed.

As we move faster, it’s clear we need to innovate with our infrastructure as well as automating our application configuration. As we begin to remove constraints, the infrastructure our servers are built on becomes a new constraint for us to work through. We are working on initiatives to move products to the cloud. Partnering with our platform team and moving faster in this arena will be a key challenge for us to work through.

InfoQ: Would you like to share what you will talk about at DOES17 later this year?

Morrison: We will share details on CSG’s continuing DevOps journey in our presentation “More Culture, More Engineering, Less Duct-Tape”. At DOES16, we talked about our major reorganization around DevOps “You build it, you run it” teams. A year later, we look at progress we’ve made, lessons we’ve learned, and next steps in our journey.

About the Interviewee

Erica Morrison’s teams provide software solutions to CSG’s 40+ DevOps teams. These solutions range from continuous integration frameworks to load balancing solutions to telemetry visualization platforms. Erica is passionate about agile and has experience leading DevOps teams where members own the end-to-end infrastructure and code. Erica also has software development experience in the defense and aerospace industries where she worked on projects such as the replacement for the space shuttle. She lives in Omaha, Nebraska with her husband and two kids.