Transcript
Reinhard: I'm really excited to be here and talk with you about change, because change is at the core of modern software delivery and at the core of our work as leaders. All of us are here to talk about optimizing our organizations for speed, and how to improve velocity and flow of our teams. Putting DevOps into practice helps us with this, and I want to share with you how you can utilize it for your teams.
When we talk about DevOps culture, we fundamentally talk about making change easier. DevOps culture is fundamentally about change. It's about reducing friction to improve our ability to change, and change continuously in service of improving our products for our customers. As leaders in modern software delivery organizations, we choose continuous improvement, continuous change over being satisfied with the status quo. We do so for the benefit of our customers, our business, and our teams. We're also humans, and sometimes change is hard for us and for our teams. From many conversations with leaders, I know that the last year has brought many difficult changes for teams and leaders as well. It may have been difficult for you too, and I find that important to acknowledge. Yet for us as leaders, change isn't just an annoying side effect of what we do. Change is the work. Initiating, responding to change, that's what we do every single day. As leaders, we want confident teams that thrive and deliver business great value regularly.
CircleCI
I want the same thing for my teams too. CircleCI sits at the heart of helping teams manage change. We've been around for almost 10 years, we have over 500 global employees. We've always been distributed. We just celebrated our 100 million Series F funding round, two weeks ago. We power some of the Earth's best engineering teams at companies like PagerDuty, Spotify, Ford, Samsung, and support open source projects like PyTorch and ELEKTRONN. This means that we have a front row seat to how teams move code from idea to delivery. How teams handle change, and also in helping leadership teams support their teams in moving to better practices. My engineering team has grown quite a lot. Our department overall has grown 5x in the last 3 years, from 45 to over 200 employees. We use our own product and other modern software development practices and processes. This means that we lead change ourselves, we help other teams do the same. We've learned some things along the way. This talk is not just about frameworks and principles, but it's also a case study. In just one of those years, we doubled our engineering team and increased our customer facing deliverables by 2.5x. I'll sprinkle a lot of examples of how we've done that throughout this talk.
The Spirit of CI/CD
I want to get us started by looking at some principles in CI/CD, continuous integration, continuous delivery. Our mission at CircleCI is to help teams quickly and confidently validate and seamlessly ship change to end users. Some of the core principles can be summarized in the word VALUE. The first part is visibility. I really think it's crucial because it means visibility into changes for everyone. Accountability is another one. Continuous feedback and transparency, and encouraging teams to take responsibility and hold themselves accountable is a crucial part of CI/CD. We lead with frequent and small changes that everyone contributes to. Obviously, we do all of that in service of user value delivery for quick feedback loops with our customers and ultimately continuous enhancement.
I want to pull out here that it's not just about change, but it's about changing confidently. Our job as leaders is to be confident in change and help our teams be confident in the changes that they're making. Confidence isn't in a trait. It means being conscious of our powers, but I also think there's an action component to it. Confidence is what greases the wheels. It turns our thoughts into actions. It makes us go from, can we do this? To, let's give it a shot. It's not just the state of being, it's action oriented. It means we're willing to take risks, struggle, fail, and eventually achieve mastery. That's really the mindset at the heart of CI/CD. Change isn't just a detractor for confidence, also, it's a prerequisite for it. Confident organizations and teams they act on opportunities, rise to new challenges, take control of difficult situations, and take responsibility when things go wrong, ultimately to learn from this failure.
Let's translate those core CI/CD principles into leadership strategies and frameworks that you can use with your teams. We'll cover visibility, accountability, leadership, user value delivery, and enhancement. I'll show you some frameworks and tools, as well as how we utilize those in my team at CircleCI. Your team and your company may be in a very different place, so take what works for you or make modifications to that. Ultimately, again, it's all about continuous improvement.
Accountability and Ownership
We'll dive in with A for accountability. I want to pull accountability up first, because I think it's foundational. If you don't know who's accountable for which work, your efforts will inevitably fall short. A lot of accountability because of that ties back to ownership. We want to empower teams to do the best thing for the business and drive impact to make the right decisions. A crucial factor here is that teams have to have clear ownership of metrics and feel connected to those. This is done when you connect team metrics to business goals, strategy and business impact. Accountability also means that teams own their progress and reporting. Only if we're owning our progress and mistakes, we're able to actually develop this confidence that we've talked about.
There are a couple ways that we've done this in practice. Like many of these other things that I'm going to talk about, we're continuously learning, continuously improving in those areas. That's a key factor that I want you to keep in mind. Metrics ownership for teams is a really important part of this. I think this ownership is key. First of all, we've involved our teams in defining the metrics for them. Their subject matter experts use their input for figuring out what the best metrics for them will be. Use business goals and strategy to help support those conversations. Clarify roles and responsibilities. Make sure that it's clear who owns what and who's ultimately responsible, but also accountable for which parts.
Another aspect is, clearly define service ownership. We have service ownership for the teams defined, but we also regularly review if the boundaries that we have in place are still working, and if the interfaces and interaction patterns between teams are aligned with what we want them and our business to do. We need teams to have the right amount of context, but also the right interaction patterns to be able to really own their domain. Then, I think it's not just about service ownership, but ultimately end-to-end ownership. The service ownership is just a way to get there. I think the team shouldn't just own services, they ultimately should own the user experience, because in the end, that's where the real impact is. Then we usually have a mix of metrics that we end up with between SLIs, SLOs, as well as some primary business metrics that we're looking for the team to impact. Those metrics are all aligned with our corporate goals. Transparency in all of that is really important as well, we'll cover more about that in the visibility part. Teams should have access to metrics. They should be able to review their progress report on it, and also make changes to their execution based on it. It's an important part of the empowerment component in the accountability.
Leadership at All Levels
Next, let's look at leadership. I believe that there's a difference between management and leadership. I like the definition that management is about coping with complexity. Leadership is about coping with ambiguity and change. I believe that organizations need leadership at all levels, because it helps us carry our organizations forward and ultimately succeed. This is not about a dichotomy between leaders and followers. Leadership at all levels is about growing people at all levels, who can handle ambiguity, who can drive change. This will mean that your organization can take a much more active role overall in leading change. Ultimately, it will accelerate your ability to improve for the sake of your business and customers.
Let's talk about a couple practical ways how we do that. One big part is context. I and many of the leaders in my organization regularly share context with our respective staff about the meetings that we're in, the things that we're hearing, the initiatives that we're driving, or just observations across the organization. This really helps with the bigger picture and exposes people to larger amounts of ambiguity, to higher complexity, and to topics that may not have easy or straightforward resolutions. This context can really help in building out this ability to deal with ambiguity and take leadership.
Another part is sharing problems or objectives instead of projects. Delegating larger stretch opportunities is a really good way to grow leaders, but making sure that those aren't delegated as basically a task list but as outcomes impact that you're looking for this person to have, and having them figure out how they're going to do that. Similarly, we usually have projects in the domains or organizations that people run. Usually, those projects are again tied to impact and objectives that we're looking for people to achieve. We have those at different levels, and both for people managers, as well as technical leaders in any level. It means that people can go from having impact in their team, or in part of their team, to larger impact across several teams, to then organization-wide. Another aspect is a growth framework. We have a competency matrix in place, one for engineers, and we also have one for managers. Those kinds of frameworks which outline expectations and roles again, can help support growth conversations. Leadership skills are an entire section within this. There's something that we expect, again, from engineers at all levels.
Leadership: Shared Goals
Another aspect of fostering leadership is communication specifically around goals. I believe in high transparency when it comes to goal setting. Business goals should be visible to everyone, the same for team goals. It helps with alignment, but also makes sure that teams have a better view of what's going on around them and how to stay aligned across. Communication around those goals as well as strategy is key. Communicate your purpose, mission, strategy, operating plan, and do so regularly. Especially in times of change and uncertainty, it helps people stay connected with the purpose. It also builds confidence in your plans and your leadership. Communicate your strategy continuously at a high frequency. I believe there's no such thing as talking about strategy too much. Then utilize the power of connections. Don't just share strategy, but help people understand how it relates to them. Draw the connection from, here's the strategy to here's what this means for you, for your domain, your team, as well as your individual job every day.
A couple ways for how we do this are, we regularly communicate strategy. We have kickoff meetings in the quarters. We have strategy overview sessions. At the domain level, we host different calls to connect those higher level with team goals as well. All of those things also get recorded. It's a small detail, but it also means we can share those contents with new folks as they're joining us. We use OKRs as a goal setting framework to help us with all of that. We lead with the company direction and strategy. Then, the teams align their goals on this overall direction, as well as in alignment with the metrics that they own that we talked about earlier. This helps the teams with alignment while also opening up space for the teams to contribute and prioritize work that they know that matters in their domain. Again, we're relying on teams as subject matter experts.
We do OKRs on a quarterly cadence. They're visible in share places for the entire company. We review progress against them regularly with the entire company. I know that people in our industry, there's a lot of opinions about OKRs and how to do them the right way. I believe that most of all, like any framework, it's about how you use it. We have been continuously iterating on our use of OKRs. I'm 100% sure that we will continue doing so, because ultimately, it's never about the framework but about making it work for you. We also do individual goal setting, and those goals again, are aligned with our OKRs. We can draw the connection between what the company is looking to do and what individuals are doing on a quarterly basis. Ultimately, all of this instills confidence because it helps teams know that we're doing the right thing.
Visibility: Defining Metrics that Work
Let's talk about how to define metrics that work. I've often encountered two different problems scenarios here. One, in an organization where there's a strong data driven culture, or people who are very keen on observability, there can be a huge temptation to measure everything. On the flip side, sometimes there are cultures that aren't at that point yet, and then it can feel really daunting. I'm mentioning a lot of metrics, and you may be really unsure where to start. I want to give you some tools to establish good metrics with some examples baked in for how we approach that. The first point I'd say is, measure what matters and measure what you want to know very regularly. In my team, we incremented our way towards the dashboards that we're using today. We started with a handful of metrics that seemed like they'd be good to keep an eye out for on a weekly basis, and a very simple spreadsheet, and built that out over time.
The second point is figure out how you will know what's working or not, and how you will know quickly. My favorite example there is our hiring goal. Ultimately, for me, the biggest goal is having people join my organization, have their first day, and then succeed in their onboarding. That's usually a couple weeks or even months out from when we start hiring for a role. I look at metrics that are much quicker and tell me much faster whether we're succeeding or not. For example, time to a certain interview stage, or time to application review even. Another part is the impact. Focus on impact. For team goals, for example, you can convey team goals by measuring features, or by measuring the impact that teams are having, when you focus on user adoption or other much more user centric metrics. Align teams on business metrics that you're looking for them to drive. It's much more motivational because it connects them to meaning, big picture, and to users. It also helps with accountability because it gives teams a clear sense of ownership and purpose.
Then, use a good baseline. There is a lot of industry baselines out there for DevOps metrics, hiring metrics, team health metrics, and other metrics where you can also look at cohorts or companies similar to you in size or stage. These baselines add meaning and direction, and are really helpful context for any metrics that you set. Then, just a word of caution, avoid driving the wrong incentives or encouraging counterproductive behaviors. There are a lot of anti-patterns in our industry around metrics, and a couple that I just want to caution against. One, individual metrics. I believe they're not helpful. We're all looking to build great teams and organizations, and not incentivize hero culture. The second, avoid metrics that pull teams in opposite directions. A classic example is between product and engineering teams. I run a product engineering organization, so there's always some tension, which is healthy, but in many organizations, there's conflict. Oftentimes, it's because metrics are misaligned. The third part, teams should know what they're measured on. Don't run shadow measuring systems in the background, not great. Then also, fourth point to caution against, avoid metrics like lines of code. I think they're not really helpful. They don't really entail meaning or really capture the impact of work.
Visibility: Metrics Used by Successful Teams
There is no one-size-fits-all approach in metrics. Every team is different. Pick what works for yours. I'll show you just a couple metrics that I found useful. We see a lot of teams handling change, and move from idea to value delivery every day. We've observed patterns on our platform, especially from top delivery teams. These show patterns that suggest benchmarks that other teams can use. There are four key metrics that show team performance. Those execution metrics in CI/CD have been mentioned by many others, Puppet and DORA have been talking about them or flavors of them for a while as well. You can see the medians that we've observed in our platform, as well as suggested benchmarks in this chart. First one is for throughput, so the average number of workflow runs per day. Second is for duration, so length of time it takes for workflow to run. Mean time to recovery is for the average time between failures and their next success. Then there's the success rate, so passing builds divided by the number of runs over a period of time.
These benchmarks we suggest for teams, these are the ability to merge at will. Then builds that run between 5 to 10 minutes. Time to recovery under an hour, and a 90% or better success rate on your default branch. The interesting part there is also that high performing IT organizations outperform their peers in those significantly, with 200 times higher deployment frequency and 24 times faster recovery from failure. A much higher success rate and shorter workflow duration. If you're using the CircleCI product by any chance, these metrics are also all visible in our Insights tool, so can be accessed there by anyone on your team, again, helping with the visibility part.
A couple other metrics that you could use if they work or are useful for your team, engineering quality. You've looked at DevOps metrics. There's also availability, which is obviously important for customers. Then for the delivery metrics, like lead time, cycle time, how long does it actually get us to go from idea to value delivery? Investments, we look at how we distribute our time between technical investments and maintenance feature work or escalations for our customers. Another is of course goal progress. OKR status and progress towards other goals that we may have. Hiring metrics, time to get to a certain stage, or pass-through rates to identify if your process is working. These metrics are all only useful if you have clear ownership. They need to be visible to help with transparency. Metrics are only a part of the story. It's always context and complexity that matter. I ask my teams not to share just their metrics but also something around them, their projection, their confidence in those metrics. Because anyone can throw numbers out there, but it gets really interesting when you understand the story behind them, and your teams take accountability for sharing that story with you.
User Value Delivery: External Users
Let's talk about user value delivery, because that's come up a few times as well already. Because ultimately, everything that we've talked about so far impacts this part. Confident teams will result in happy users. There is the external actual value delivery to users. Incremental value delivery allows us to get user feedback quickly, address it, and build on it. Ultimately, increase our team's confidence through continuous learning. In my team, we use a modified dual track agile delivery process to help us not only deliver in increments, but also combine discovery. Learning about our users and their needs in our daily work. There's also the more leadership facing part of user value delivery. If we set up our team in such ways that they're continuously learning, growing, and changing in increments, it's much more manageable, because it helps continuously enhance the value that our teams are delivering.
Enhancement: The Real Failure Is Not Learning from Failure
This value delivery is really tied closely to enhancement, the last letter in our VALUE acronym. Because I believe that the real failure is not learning from failure. Getting feedback regularly from customers and key teams is crucial. I want to give you an example of that, because I feel like this is much more convincing in concrete terms. A couple months ago, there was an issue in my organization, and I had a couple discussions with my boss, our CTO about it. Ultimately, any issue in my organization is to me to own up to. The conversations we had were not about blaming, finger pointing, or anything like that. That's not how we work. My boss wanted to make sure that we are learning from this, and that we have the culture, the tools, and the visibility in place to prevent something like this issue happening again. The focus of the discussions that I had with my teams was, do we have the right structures, tools, visibility in place? Do we have the right culture? Are we able to own up to mistakes that happen? Are we able to learn from them? Because that's the part that matters, not how we got there, but what we do with where we got going forward. I believe that having this culture, these open conversations is crucial for teams. Another part of that is skill development, having an open feedback culture, hosting team retrospectives, and open incident reviews.
That segues us nicely into how we do these things in my team. We run regular incident reviews, and they're blameless. We also hold retrospectives for teams and projects. It's nice to just have a cadence for learning. We have different training and development programs. I also use these kinds of check-ins regularly to figure out if we have the right visibility, accountability, learning, user value delivery, and enhancement. Are we actually learning and are we continuously improving?
Confidently and Continuously Delivering
Ultimately, these continuous integration approaches to engineering leadership, they mean a comprehensive system where all parts are woven together and highly connected. We empower teams and build confidence when teams own their metrics and their learning, and we give them the tools to do so. We use metrics that work for our teams, and organization. Metrics that are visible to everyone so that we know confidently that we're working on the most impactful thing for our customers. Our goals are all connected, and they're visible for everyone. It means we're confident, we're doing the right thing, and that it's aligned with our strategy. Everyone on our team contributes and leads, which helps us thrive in times of ambiguity and change. We make incremental changes for our teams and for our customers to continuously improve. Ultimately, all of those are about handling change with confidence and delivering value continuously. From all these ideas that I've shared with you, pick what works for your team, and use it to help your teams become confident. Lead them towards being a team that delivers value confidently and continuously.
Resources
If you're looking to learn more about the metrics that I mentioned, and other data backed benchmarks for delivery teams, we have our state of software delivery report on our website as well.
Questions and Answers
Shoup: As you start to introduce KPIs and SLOs, how do you make sure that people feel confident in their failure and not get too risk averse?
Reinhard: I think that there's a lot that goes into that. I would want to tackle two parts here. I think where the one is about which metrics you set and also how you introduce them. The other is then setting the right environment for people to be comfortable with failure. I think they can't be separated. I think the first part is really, go slowly. I've mentioned a couple times like building your way towards metrics that work. I don't think it's helpful to overload a team with several metrics that they have to observe all at once. Introducing those things slowly, incrementing your way towards what you actually want can be really helpful, because it also helps people get comfortable with this. Because, ultimately, it's a culture change. It's a very different way of looking how teams operate. That just takes time.
The other is also, making sure you're not setting the wrong incentives. I've in many cases seen that you basically need at least two metrics in order to balance and get to the impact that you actually want to have. Let's just say you would measure a team on the customer impact that they're having. At the same time, you want to make sure that they're not neglecting technical investments, because you need those two for scalability and for actually supporting more customers. In those cases, you would probably want to actually have both, because you want to make sure that there's a good balance. Thinking about, what's the outcome that you actually want and the impact that you want the team to have? What are the best metrics to get there that help drive the right behaviors in people, and not set the wrong incentives.
Then the other part is really the comfort with failure. I think that one's a really tough one, because it's often really hard to measure why people may be uncomfortable with. I think digging into it with the team and having some open conversations about that can be one start. The other factor, I'd say, is definitely looking into how does your leadership team talk about failure? How are issues being addressed? How is criticism being voiced? How are the public discussions that are happening in chats, or in larger meetings, around when things go wrong? What are leadership reactions? There's a lot that can go between the lines. I think asking people what their impressions are can be a big factor. Then, in combination with the right metrics, you'll have a good foundation, at least.
The last point I'd say on the comfort with failure is, do you actually have an environment where you can learn? Because the failure is ok, if people understand that it's an opportunity to learn. In order for that to be true, you have to have an environment where that learning is the focus and not the failure. If you're getting too focused on the KPIs, it can easily become not about the learning anymore, but with an over-focus on failures, basically. Looking at how can you use the KPIs that you're introducing for the team as a means to help with the learning for the team as well?
Shoup: Iterating on the metrics, the countervailing or balancing metrics, or the so important, and making sure you set the right incentives, everything is so good.
What happens when teams don't want ownership? Then there was another person that asked about, this DevOps stuff, but I don't want to be on-call, or team X doesn't want to on-call.
Reinhard: This is so interesting, because DevOps transformation and culture, I think, has often gotten slightly reduced in some conversations. I love the nuance that is going on in the conversation that everyone's having now as part of this track. I do think understanding that sometimes people are concerned about the complexity or about all the things that may change as part of it is really, I think, foundational, and digging into just what's going on. Obviously, I don't know the teams that this person was talking about, as I can only ask some questions, but things like, what's concerning people. Are the systems set up in a way where if they were to move on-call, they'd probably get paged every night? What's the risk that comes with moving to a different approach like this? Are people concerned about the change or are they concerned about probably having to own more than they might be able to? What's the context of the team boundaries? Is the area that the team owns quite large? Is there a huge responsibility? Do the teams actually have the right staff for that? In those kinds of hesitations, I think there's a lot to learn about the organizational culture and how it can be improved. Because, ultimately, I shared a lot of how we do things here in my team, but all of that was a learning as well. We had to increment our way towards it. A lot of that happens through learning from folks and digging into their impressions and addressing those things.
Shoup: I would just also say, just from my own personal experience, the companies that we look at, where they're like, "Those guys are amazing at DevOps." They didn't all start that way. A lot of them started with the places where a lot of traditional organizations still are in terms of like a very classic Ops team is separate from development. Even the industry leaders, very much to your point, Lena, have iterated their way toward learning about how to do this.
I love that framing of management is about complexity, and leadership is about ambiguity. Say more, because that's really neat.
Reinhard: First of all, I think a lot for me has to do with the time horizons that you're looking at. Management in a lot of ways to me is about the short term. It's about taking a piece of work, breaking it into smaller chunks, handling those. Making sure that there's pure scope, that there's deliverables to certain timelines, those kinds of things. It's a very execution focus test. I'm not saying that in any diminishing way, it's also really important work. It's very tactically focused. Whereas the leadership aspect has a lot to do with creating space for the uncertainty. Thinking about the longer term. Setting direction, making sure people understand that direction. Having a vision that they're aligned against. Having different approaches for how you're handling those kinds of situations, or conversations about that longer term, depending on who you're talking to. Modifying leadership approaches, and then connecting people with all of that.
In a lot of ways, I think it is opposites or dichotomies, because it means that in leadership there's no easy answers necessarily. There's a lot of things that don't get resolved immediately. There's a lot that you're just collectively holding as a leadership team. I often tell my staff that it's part of our job just to collectively hold some ambiguity for the organization and still show we're still pushing our way forwards in it, and we're still working through it one week at a time, but there's a lot of things that we just can't answer. That in itself, that ability to handle ambiguity, the ability to think about the longer term, the implications, the systems that are involved, that's something that I think everyone can develop. I think it's not exclusive to leadership roles or management roles. I find that a really important part because the more people you have in the organization who possess those traits, who are able to think about that bigger picture, and about the strategic level, the more you'll be able to also move the organization forward.
Shoup: When I was running engineering at Stitch Fix, I participated in an internal training that they'd been doing, before I even joined, about leadership. I love this framing too, which is very analogous to what you're just saying, leadership is making something happen that otherwise wouldn't. It's not dismissive to say, everybody's a leader. We expect everybody to make something happen that wouldn't otherwise.
Reinhard: I love that because it also opens up this space, I think, where management is more reactive. There's this creative, in the literal sense, creating a component of leadership that really helps with all that. That's a good way to put it.
Shoup: Yes, 100%. It's not fun to be in a job where you're just order taking, as opposed to like, "Here's a problem to solve. I'll help you."
If you could elaborate more on your experiences with building a healthy culture, I think it would be very insightful.
Reinhard: Start with values. Think about what you actually want to drive towards. I love this definition of culture as the behaviors that we reward and punish. Culture is something that happens every day, and we create a set culture through any interaction that we have. Being cognizant of that is a really good start. Then having values that you can align people on, that everyone shares, and that you can also use, again, to reward behaviors that you think contribute to the culture that you want ultimately, is another. I think a lot of the aspects that I covered in the talk around leadership, and the visibility, accountability, basically, everything that's part of this VALUE acronym, all of those things are ultimately driving culture. Because they set a standard around transparency, a standard around ownership. They foster a certain leadership approach. All of those things contribute to culture in themselves. I think culture happens with every interaction, basically, keep that in mind. Then start with what you want to drive towards.
Shoup: Can you share any situation where you had to impact culture change across teams and divisions? How did you go about that? How do you deal with things that cross organizational boundaries, like stuff that you need to do?
Reinhard: It depends. There was this model of impact, influence, and soup. Looking at which areas you can actually change yourself versus which parts you may be able to influence which are just completely outside of your control. I think a lot depends on the culture change you're driving towards. If you're looking to introduce DevOps culture in an organization that doesn't have any of it so far, my recommendation probably is, start small, show impact. In a lot of ways that is a really smart approach, because you can demonstrate that you're able to do things and they're impacting things in the positive. If it's about other types of culture change of how teams operate, or things like that, partnerships, relationships are often a really good foundation. A lot can also be used with metrics where you set metrics for how you expect to partner, what turnaround times look like, or things like that. It's a squishy topic. I think visibility in all cases is a really important part because it means you're showing impact. You're communicating what expectations are, and how teams are doing. That helps.
See more presentations with transcripts
