The Ongoing Challenges of DevSecOps Transformation and Improving Developer Experience

Transcript

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I'm sitting down with Adam Kentosh. Adam is a field CTO of digital AI and, Adam, a good starting point is who is Adam?

Introductions [01:03]

Adam Kentosh: Thanks for having me, Shane. Really, really appreciate the opportunity to be here. And just in regards to who I am. I'm a technologist at heart. I've spent over 20 some odd years, or close to 20 some odd years in technology, really cut my teeth on Hadoop, large scale Hadoop clusters, managing those with automation and purpose built for custom search algorithms. From there, went into the consulting business and had an opportunity to work with some Fortune 20 companies around the area, especially as they were transitioning from things like VM based architectures into cloud, containerization, Kubernetes, et cetera. From there, I went into HashiCorp. I was able to join HashiCorp very early on and got to experience just the really exciting growth that happened at that organization.

And now I'm at Digital.ai as a field CTO, and I'm helping to bridge the gap between the field teams and the product and marketing teams, making sure that I'm taking back anything that I've learned in the field to the product teams for potential enhancements or improvements, and then also back to the marketing team as we're thinking about how we can have messaging that resonates with our customers.

Shane Hastie: We're talking about shift lift and DevSecOps and so forth. Isn't this an old, tired topic that's dealt with already?

DevOps is a well understood approach, but still not implemented in most organisations [02:27]

Adam Kentosh: It does feel that way. We've been talking about this, I think, for quite a while. Even back in 2009 when I was still fairly young, I was doing work and we had conversations around Agile and what does DevOps mean to us? And at a point in time in my career, DevOps was essentially me sitting next to a developer as an operations person. Certainly not probably what we would think DevOps is, but just the fact that they moved us into the same room, they thought, "Yes, we've done it". So I think we have seen obviously good evolution here. Just one interesting thing that I find fascinating though is I was reading the state of DevOps report in 2023 and they mentioned that 80% of companies are still in mid-transformation. And I just find that really interesting. I mean, I think we started talking about this 2009, 2010 timeframe, and a lot of companies really kicked off those initiatives between probably 2010 and 2012.

We're now 10 to 12 years in on DevOps transformation and we're still talking about it. So I think that just begs the question, why are we still talking about it? And for me, that's a very complex, I'd say, question and complex problem. I think we've seen DevOps transformation happen at the team level, but I think that sort of enterprise-wide transformation is still very elusive to a lot of customers. And I'd probably pin the reason on technology changes and disruption. If we think about what's happened over the last 10 to 12 years, not only did DevOps come around, but we also had really the just maturity of cloud. And then we've also had mobile devices become the number one way for customers to interact with applications these days. And so in that situation, I think that we offered up some autonomy to teams and we said, "Hey, go pick your tools. Figure out how to take advantage of cloud. Go pick your tools, figure out how to create a mobile application".

And with that, we gave them what we would consider a DevOps team, and we paired dev teams with operations teams and we said, "Go forth and conquer on the technology that you choose". And I think that was great for a time. It really did help people pivot. It helped people accelerate and have kind of speed to market and get better velocity. However, as you think about trying to scale that across the organization, well now you're very challenged. It works for maybe five to 10, 20 teams. But when you start talking about hundreds of teams, we have to figure out where we standardize, right? Do we standardize at the tooling level? Are there processes that we can standardize around? Is there team makeup or structure that we can standardize around? So I think we've just had a compounding problem that was getting worse simply by, I'd say, the introduction of new technologies over time.

Shane Hastie: How much standardization is enough? As a developer who was given this freedom, now you're taking it away. How do I come to terms with that?

The balance of standardisation vs team autonomy [05:35]

Adam Kentosh: I see two options, and I've seen this in practice at a lot of organizations and done it personally at a few. But number one, you don't. And you build a interface layer on top of that and you've got something that basically integrates with all of these different technologies. Because really when we start thinking about standardization, it's not so much that we necessarily care about what tool they're using, it's that we care about the metrics that we can view of that. How are my teams performing? Are they delivering at the rate that we would anticipate? Are they having more or less bugs than other teams? So I think the thing that really matters for me is sort of the metrics that can be driven out of that platform and then being able to level teams up, "Well, if we have a bit of an underperforming team, how do we take them from a two to maybe a five?", for instance.

And I think DORA has done a great job of setting a framework and a context for us to have a level up type methodology. And so we've got four or five metrics. We can take a look at how that performance is across all our organization, and then we can focus in on specific areas. So I think the first approach is you don't bother with standardization and you just care about the metrics and you figure out a way to gather those metrics and then help your teams perform better. I think the second approach is probably what we're seeing materialize, is something like platform engineering where, "Hey, let's create developer-oriented tooling. Let's create standard interfaces and APIs that can be leveraged so that we can obviously make things either more productive or we can reduce just some toil from a developer standpoint".

I feel like either one of those are viable options in many cases, though certainly I think as a business there's probably more viability in getting some level of tool standardization. They still have to operate as a business, they have contracts that they have to worry about. There's a lot of negotiation and renewals and things that happen to enable those contracts. So from their perspective, if we can get it down to a reasonable amount of pool set that does get the job done, I think there's probably usefulness in that.

Shane Hastie: So finding that quite difficult balance. The other thing that certainly you and I mentioned in our conversation earlier, but also I've seen a lot is as we have done the shift lift, we are seeing more and more cognitive load on developers today.

The impact of cognitive load on developers [08:04]

Adam Kentosh: Definitely. It's very interesting. There's been a couple of reports that have come out. GitLab has published a report just around global DevSecOps and GitHub has done a lot around developer experience and what that means today. And in both cases, the numbers were somewhere between a quarter to 32% of developers time is spent actually writing code and time is spent elsewhere now in terms of maybe improving existing code or doing administrative tasks or meetings or testing or securing your application. So I think as we think about shift left, what we're really doing is we're putting a lot of strain and stress on a developer community and we're asking them not only to write code, but understand how to test that code and then understand how to secure that code and then understand how to go release it and also maintain the operations of that code over time.

So for me, it puts a lot of pressure on the development teams, and I think that's why developer experience is becoming so important these days. I've heard a ton of just conversations around developer experience. We work with customers that now have titles for the VP of developer experience and their initiatives is to go make sure that their developer community is productive, that they have good impact and that they're satisfied. So I think a lot of the reason that we're seeing developer experience sort of come around today is simply because we are putting a lot of strain and stress and pressure on the team as we continue to progress through these shift left motions.

Shane Hastie: How do we reduce that but still get the benefit?

Adam Kentosh: That is a very good question. I'd say ways that I've seen it be reduced and probably the most effective means of reducing it, is truly having more of a product centric approach. So we've seen two things, I'd say, happen over the last few years. Number one, we asked the developer to do everything, and that obviously breeds challenges. Number two, we continue to have isolated silos for testing and for security. And that again, causes challenges. Maybe we see a reduction in velocity because now I have a handoff between development to testing and then there's a handoff between testing and security. So depending on which way you go, you're having issues no matter what.

And I truly feel that we probably need more of a product-oriented approach, meaning that the product teams have to have these capabilities on the team themselves. We still need testing, we still need security, we still need development, but I don't think one person needs to do all of those things. And if we can get them closer working together, more collaborative, I think we're going to be in a much better situation.

And if you look at some of those surveys that I mentioned, it was really interesting because a lot of what the developers were asking for was better collaboration. And I think this would breed not only better collaboration, but probably a more productive product team.

Shane Hastie: But haven't we known that collaborative cross-functional teams are the way to do things, since 2001 at least?

Adam Kentosh: We have definitely known that. I would totally agree with you there. I still feel the challenge is the organizational structure is matching the team structure, and so we're seeing a dedicated testing organization or we're seeing a dedicated security organization, and now we run into the people process aspect of our challenge where people don't want to maybe give up control of the teams that they're working with. And so the idea of blending teams means that we have organizational changes that must take place, and that, I think, is probably what's restricting or limiting, I think, this type of approach. Inevitably we get in a situation where, like I said earlier, we either fully shift left and we have developers owning more of that delivery or we just continue to deal with our silos because it's almost easier than trying to reorganize a large enterprise.

Shane Hastie: What are the small nudges we can make that move us to a better developer experience?

Small nudges for better developer experience [12:22]

Adam Kentosh: The best thing that I've seen work for nudges is really still the idea of trials, targeted use cases, where we can actually take a product-centric approach to a team and let them go be successful. And if they can be successful and we can validate that their numbers are what we would anticipate, that they've got great cycle time, that their bugs in code are not there or they're infrequent, I think that gives us justification then to say, "You know what? This is a working model. This is an opportunity to work together better, and it's a way that we can improve our overall performance as an organization".

So for me, it's always in doing. I think, in getting out there and actually trying out new models and then making sure that it's a model that can scale. So come up with some best practices and actually be able to take that back to the organization and show real results and real numbers behind it. I feel like that's probably the best way that I think we can continue to nudge, otherwise you're in a position where you're trying to nudge at an executive level, and that can be maybe far more challenging.

Shane Hastie: We've known this. As an industry, we've figured this out. We've seen organizations that have done this well, and yet 80% are still mid-transformation. We're still facing a lot of the same problems that all of our methodology changes and everything that we've tried to do over the last couple of decades. Why don't we learn?

Finding the optimum delivery frequency [13:58]

Adam Kentosh: Well, first off, I'm racking my brain to say, "Have I seen an organization that's done it well?" And I suppose that the flagship examples are maybe your FAANG or MAANG companies, I guess now, but to me they're operating at a different level and they also seem to be producing a lot of this content around some of the more progressive types of development approaches. So if I think about that, and I'm getting off-topic a little bit, but we'll get back around to it, but I think about that, they're pushing and asking for continuous delivery. Let's just be able to deliver on a continuous basis, small incremental chunks. And I think for the vast majority of organizations that I work with in terms of financial services, insurance companies, even some level of gaming, definitely airline, I mean, those companies don't necessarily care about delivery frequency.

They're purposefully pumping the brakes on delivery because they want to make sure that they're compliant, they want to make sure that they're secure, and they want to make sure that they're not introducing a poor customer experience with the next release that's going out the door.

So in reality, I'd say there are companies I suppose, that do it well, but there are other companies that just maybe don't need to do some of this. And so being pragmatic about what is useful and what will resonate with your organization, and truly at the end of the day, what outcomes you care about. If you care about new features and keeping pace, then certainly maybe continuous delivery makes sense for you. But if you care about stability, well, maybe it doesn't. Maybe there's the better way. And I'm not advocating that people go back to a completely waterfall style of delivery and that we take six months to get a release out the door. That's certainly not, I think the case here, but I think technology has enabled us to take a more, let's call it reasonable approach to delivery, and then also still be able to get better quality, more secure applications that can go out the door.

So I know that was sort of a little bit of a segue or a long way away from what your question was, but just something I was thinking about as you mentioned that. Now back to what you were asking, "Why don't we learn?" I think that the challenge is from a developer standpoint, and you'll see this too, if you talk to any developer, one of the things that they really enjoy is just the opportunity to learn new things. And so when you go to a developer and you say, "Hey, we want you to take on testing". "Well, hey, that's interesting. I'll go learn that new thing for a little while". Or, "We want you to take on release pipelines". "Oh, interesting. I'll go take on that new thing for a little while". So I don't think they're shy about saying, "No", or I guess they're happy to say, "Yes", rather and say, "Yes, I want to go learn new things".

So for me, I'm not going to pin it all on the developers. It's certainly not all their fault, but we're asking them to do the more, we're giving them an opportunity to learn new things. That means professional development, it means new skills. That's something that they're always after, so they're just going to go do it and they're going to solve a problem. And I find that true for most engineers. I talk to my wife and she always wants to just kind of talk at me, but I always want to solve the problem that she's talking about. And so it's hard for me to stop trying to solve a problem, and sometimes I have to recognize I just need to listen. But I think that's just an engineering mindset. You're always looking to solve a problem. So maybe we run into a situation where the problem's been identified and nobody's doing anything about it and they just go fix it.

So why don't we learn? I think that's probably the biggest reason from an individual standpoint. From an organizational standpoint, to me, this could be controversial, but I just feel that maybe organizational structure has not evolved the way that we need it to evolve to support IT. And I know IT is supposed to support the business, and we're supposed to ultimately support the outcomes for our customers. And I definitely recognize that. But internally to the CIO on down who owns testing security and other things, maybe there has to be an evolution there that helps support better collaboration and maybe a more product oriented or product focused approach.

Shane Hastie: One of the things that struck me when you were talking about the contrast is those companies that are held up as doing this well, they have a product mindset and they were built from the ground up with the product mindset. So are we asking our "traditional" organizations, these banks, insurance companies, airlines and so forth to fundamentally shift their business model?

The challenge for non-product focused organisations [18:50]

Adam Kentosh: It sort of feels that way. Now, you can say that those companies are obviously wildly successful, so maybe it makes sense for them to try to shift their business model. But to your point, when you're not built that way from the ground up, there is organizational structure, of course, organizational politics that are going to come into play, there is certainly a level of skill from a just perspective of people that are working there that you have to take into consideration. And so with that, I think that we might be asking companies to act in an unnatural way. And so if we're asking them to act in an unnatural way, I think that's where you get... And we've seen people get into the situation, as I've mentioned, where shift left is now the term, and DevSecOps is definitely still a term in trying to get everybody to work together in a meaningful way without actually maybe changing who they report to or putting them on the same team, truly together.

So the way that they're trying to evolve is maybe not necessarily conducive to the way that some of these organizations just grew up from the ground up.

Shane Hastie: Adam, there's a lot we've delved into here. What have I not asked you that you would like to get on the table?

Software engineering intelligence [20:07]

Adam Kentosh: I'd probably say, what is the role of something like platform engineering or DevSecOps and how important is Velocity truly to companies and organizations? And I think Velocity has ruled for the last 10, 15 years. Everything was about getting code out the door faster. And in reality, I think taking a deliberate approach to software development is probably more meaningful. And let's focus not so much on Velocity, but let's continue to focus on the outcomes that we want to deliver for our customers. And again, that kind of goes back to, what is my end goal? Is it a stable customer environment or is it a stable application environment for my customers to operate into? And if so, well then let's just be maybe thoughtful about how we deliver software. And then from a platform engineering standpoint, I think there's value in the idea of platform engineering, but I actually think there's probably almost more value. I shouldn't say more. There's equal value in the idea of something like a software engineering intelligence platform.

And Gartner coined this term back in March, and it's kind of interesting as we start to think about it. But if you look at software development, you've got to plan what you're going to build. You've got to code it, you've got to test and secure it, you've got to operate that. And historically we've purchased solutions inside of each one of those areas, and we have engineering processes inside of each one of those areas that have to happen. And that all generates data into its own database. And that's great for business intelligence. It gets us some information from an agility standpoint or an Agile standpoint. We can look at things like cycle time and figure out how our teams are doing.

However, when I want to ask real questions as a business leader, when I want to understand how teams are performing across the organization, when I want to understand what changes are coming down the line that are really risky or when I want to understand what teams have the biggest impact to cycle time across the organization holistically, I can't answer those questions with that type of approach.

So for me, I think the next evolution that we really need to see, and it's going to set up AI initiatives inside of an organization, which is why this is also important, is the idea of unifying all of that data into a meaningful data lake and then putting engineering processes in place so that we can link the data together. And what I mean by that is if I'm doing a release and I'm not opening up a ServiceNow ticket or some type of tracking for that, well now I can't tie an incident back to a specific release. But if I could, that would be very helpful. And if I can tie a release back to specific stories or features that are actually getting into that release, well that's also very helpful. I still talk to customers every day that spend hours on change advisory board reviews, and they spend hours on tier one resolution calls that they have to jump on, and they've got 80 developers on there. So if it's a situation where we can reduce some of that, that breeds value to an organization, certainly.

So I think the important thing is being able to measure where we're at today and being able to unify that data to answer meaningful questions about my organization, then that sets us up for truly applying artificial intelligence and machine learning on top of that data set to help make recommendations. And that's really where I think people want to get to over the next two years as we talk about AI and ML.

Shane Hastie: A lot of good stuff here. If people want to continue the conversation, where do they find you?

Adam Kentosh: So you can find me on LinkedIn. You can definitely reach out to me as well. I'm happy to have a conversation and talk anytime. I love chatting about this stuff. I'm also very happy to be disagreed with. So if you disagree with what I'm saying, I'd love to hear a counter perspective. I'm fortunate, I get to talk to probably 50 to a hundred customers every year, if not more. And a lot of those are in that large enterprise space. So hearing other perspectives outside of that is extremely valuable for me and helps me understand what are other companies doing, how are they doing it well? And so Yes, LinkedIn is definitely the place to go.

Shane Hastie: Adam, thanks so much. Been a real pleasure to talk with you.

Adam Kentosh: Thanks, Shane. Appreciate the time today.

Mentioned:

• Adam Kentosh on LinkedIn