Thoughtworks’ Tim Cochran on Developer Effectiveness

Introductions [00:05]

Good day folks, this is Shane Hastie for the InfoQ Engineering Culture Podcast. I'm sitting down across the miles with Tim Cochran today. Tim is a technical director at ThoughtWorks and is starting to get known for talking about developer effectiveness, which is going to be the subject of our conversation. Tim, thanks for taking the time to talk to us.

Tim Cochran: Hi, Shane. Thanks for having me.

Shane Hastie: Probably a good place to start. Who's Tim?

Tim Cochran: I ask myself that every day, but my title is technical director at ThoughtWorks. What I am is a consultant and a developer. I've been with ThoughtWorks for over 15 years, and I've worked for lots of different types of companies. Right now I'm a technical director, but still very, very hands-on and still very close with technology.

Shane Hastie: What brought you to where you are today, particularly thinking about developer effectiveness?

Why focus on developer effectiveness? [00:53]

Tim Cochran: It's an interesting journey. Over the around 25 years we've seen a lot of changes. So, we used to have to spend a lot of time evangelizing about agile and then DevOps and caring about automated testing, CICD, all those kind of things. Now we have to do less, because industry concepts and everybody wants to do it, and everybody is, everybody's investing in these concepts and various tools that are popular, cloud and all that kind of things. And I've been working at different organizations and the interesting thing about ThoughtWorks is that we don't just work at slower enterprise companies that want to transform, we also work for cutting edge companies that are about as good as you can get on the DevOps spectrum and on the productivity spectrum. Living in New York, I've had the opportunity to work at some of these companies.

And so as I contrast these companies, trying to put my finger on what is actually the difference. When I've managed them, managed the accounts or worked on the accounts, I can see that there's a massive amount of productivity difference. Just like when we analyze or we show, what have we done for the last six months or three months with similar sized teams, just the difference. And of course you can say there's certain things that of course are going to be true, like legacy technology, you're not on the cloud and all those kinds of things, but it isn't necessarily a platform or a tool or a technology. What are some of the investments or the cultural aspects that some of the higher effectiveness companies are working on? That's where I ended up talking about this concept of developer effectiveness and this concept of feedback loops. And I think what I'm seeing is a lot of the higher effectiveness are really investing and intentionally, almost for a business strategy, definitely technology strategy and a business strategy, to invest in the feedback loops that I call out in the article.

Shane Hastie: Let's explore that. When you say a feedback loop, what do you mean?

Feedback loops are a key differentiator [02:48]

Tim Cochran: So, as a developer or a product delivery team, most of what you do, is you're making a change or you're doing a bit of functionality. And what we see is that what that team wants to do is to validate or to get feedback that what they've done is correct and valuable. And that's a very abstract idea of this feedback loop, but you can apply that to a lot of different things. So you can actually apply that to things that the developer does throughout the day, you can apply that to what a team does, and then you can also apply to more organizational, larger concepts. To give some examples of some of these feedback loops, some small ones are like, as an engineer, I write some code and I want to see that it works, but in a personal situation.

I write a little bit code and how do I know that it works? Well, I run it, essentially. I run my compiler, it compiles, and then I deploy it to my personal environment. That's probably my laptop or maybe a personal cloud environment and I run it through my application server and I see that it works. I also probably are using a set of unit tests that maybe I wrote, or my colleagues have written, to make sure it works. That's a very small feedback loop and that can be run, depending on the team, a developer might run that ten times a day or might run it a hundred times a day. I think that the high effective ones are the places where the developers can run that hundreds of times a day. That's a very small feedback loop, but those other things that I mention is, as an engineer, I change a component and I want to know that I haven't broken anything that depends upon that component.

Exploring the real value in CI/CD [04:26]

And conversely, I want to know that the things that I depend upon haven't broken me. So, that's another example of a validation of a feedback loop. And the reason why I think that some of these are important is sometimes a lot of it gets grouped together into, we talk about a CI or a CD pipeline, which is incredibly important. But sometimes we miss, well, what is the value of developers getting out of that pipeline. Why are we doing that? What happens then is then you end up getting into these discussions about tooling and then we forget, we get very dogmatic about doing something, like test coverage is a good example, we're very dogmatic about a hundred percent unit tests. But then we forget, okay, well, what are we actually trying to do?

Tim Cochran: We're trying to speed up, we're trying to make our code high quality, or we're trying to speed up product development. And that's where I think that what I've been talking about with those feedback loops, it's really focusing on the value to the engineer. And if you do that, then you can think about, what are the investments and the tooling and the organizational and process changes that I should be making to improving effectiveness. And thinking about that value going across organizational structures, team boundaries, thinking a little bit outside the box, that's where I think these things are valuable. So it's less about, how do I deploy Kubernetes or which CI server should I be using, what is the value I'm providing to my developer and my development teams?

Shane Hastie: How do we measure this? You've said, at the macro level you can see this in terms of how much a team produces over a period of time, but that's quite a lagging indicator. At a team level, at an individual level, how do I start putting this into practice?

Different contexts require different feedback loops  [05:59]

Tim Cochran: What I recommend is thinking about what are the feedback loops that are important for you in your context, and I list out some ones that I see very common. An example, being one, a developer joins a team, I want to become productive. There's some technical elements to that, but a lot of that is cultural and process related, so that's one example. And it really depends, so some teams, especially fast moving, agile startups, if you work for one of them, you know that these companies reorg every three months or six months. So most important thing is change, so the ability to very quickly join a new team, work in a new technical or business domain, and become up to speed quickly. It's important, but not every company is like that. What I recommend is, think about what are some of the feedback loops that you're really trying to optimize for your company, and then you can measure it.

What I suggest is, you define, I describe it as the trigger and the outcome for those feedback loops, what is the starting point and the end outcome. And then you can measure how long it takes, you can measure how much steps, how much automation, how much manual, how much back and forth. But then you also got to think about the outcome. What we see sometimes is, folks are very focused on automation and speeding things up, but not like, am I actually getting what I want from it? So for example, again, back to the automated testing example, often regression tests, there's a lot of focus on automating everything, but sometimes the output, the test results, are too difficult for a team to actually understand that they disregard the test.

Use value stream mapping to make the flow of activities visible [07:29]

Tim Cochran: What I'm suggesting is you come up with your list of feedback loops, you think about the trigger and the outcome, and obviously you can measure. What I normally suggest is using a value stream mapping exercise to measure the different steps involved and to see the different people involved, to see the friction, and then look at the outcome where developers are actually, or not just developers, technologists are getting the outcome they need. But some of it is going to be automateable, you can measure over time, some of it's really just a matter of getting people in a room and mapping it out. The other part to it is, it's not all just quantitative, it's also getting the opinions and the feeling of the engineers involved and you can do that by surveys and interviews and things like that. So, to get back to your actual question, you start off by finding the feedback loops, you map them out, you find your baseline, you can start to automate it.

Over time you can see progression or you can essentially find a way of measuring on certain cycles. What I would say is, just be a little bit careful about when you measure stuff. Quite often, I see teams that they measure what is easy to measure and they measure what the tooling gives you. So for example, in typical continuous delivery, if you're trying to map the lead time, you want to measure when the software is actually in front of a user, but that might span across a number of different tools and what typically might happen is, the easy thing to measure, is measure my CI pipeline or something like that. But often those are just set up to deploy to the first environment, which can be a bit misleading, because software is not valuable to anyone until it gets in front of a user.

What I would say there is, of course over time you want to have sophisticated measurements, but maybe to begin with you can do it just getting the right folks together and going through a value stream mapping exercise. That is measuring the feedback loops themselves. I would also say that there's a lot of indicators and lower level metrics that are worth capturing. I haven't gone into them into the article, but that will be coming up in future articles, like looking at some of the leading indicators associated with some of this stuff.

Shane Hastie: In the article, which is on the Martin Fowler website and we will include a link to that in the show notes, you actually give a list of some of these feedback loops. So first one, validate a local change works, and you've got a table there that talks about the difference between low effectiveness and high effectiveness. This one intrigued me, that even down at the level of the validate a local change works, a high effectiveness environment is five to fifteen seconds and you do say, depending on tech choice, whereas a low effectiveness environment is two minutes. Why such a big difference even at that level?

Effectiveness factor: Time to validate a local change [10:07]

Tim Cochran: So, it's an indicator of a lot of stuff. Some of it could just be technology choice, some of it could be what you're going to see at a low effectiveness is that the company may have more legacy technology that they're using. But quite often it's an indicator that the company hasn't invested into improving that. Typical one that everybody knows is the monolith. Often monoliths are set up in a way that when you build or compile or try to deploy it, it builds everything, rebuilds every library, every module, and probably to begin with when you, first year or so, it was fine. But when monoliths exists for ten years, unless they've done some exercise to componentize it or to break it out into separate microservices, it's going to take a long time. The technical reality is something like that.

You may have multiple teams working on this legacy monolith, that's maybe an old technology or something like that, but that's a technical reason. But the interesting reason is, what's the organizational factors that allowed this to happen, which is more interesting. In those sense it's like, well, there's an indicator that, that company is not investing enough in technical debt. They're not allowing their developers, because I will guarantee you go into all these companies and no developers is going to say, I like sitting for two minutes every time I want to see my change work. And by the way, two minutes is not the worst we've seen, we've seen way worse, but it's more that either what we see sometimes is that there's a Conway's law problem that they're not empowered through organizational structures to be able to optimize it for themselves.

It might be that there's a separate team that does that, or to make a change is going to affect so many people that they're not able to. The company may be too focused on deadlines and trying to get new functionality out and not enough time to actually focus on that technical debt. I also mentioned this idea of learned helplessness, It's just over time, they just become resigned to it, that's just how things are, and that's a sad state of effect. It could be that maybe the company hasn't had much change in personnel, that's something else that we've seen.

I think nowadays with evangelizing with dev ops, with all the companies that talk about their good stories, most people are trying to do better, but some of the stuff we've seen historically as to why they haven't optimized it. On the high effectiveness one it's either that the teams have a good process around technical debt, they have people that are really motivated on the team, they spent time up-skilling on dev ops and understanding that. The other thing that we've seen sometimes is that, depending on how the technology is laid out, but you may have a separate team, it might be a developer productivity engineering team or developer experience team that is actually, their KPIs is to keep those numbers down and they're providing tooling and they're making sure that the application developers have a fast environment.

Shane Hastie: So let's explore some of the other metrics in your table, if we may. Find root cause for defect, high effectiveness one day, low effectiveness four to seven days. What's happening there?

Effectiveness factor: Find root cause for a defect [13:01]

Tim Cochran: I think this is not going to be a shocker to most people listening to the podcast, if they've been in the industry for the last 10 years or whatever, and this has just changed with all the really good observability tools that have come out. The root cause for defects, again, in the low effectiveness example, a lot of it's about empowerment and information. What we've seen is, bugs happen, it takes a little bit to get triaged by an ops team and then typically the engineer will not have enough access and not have enough information to actually find the solution to the problem. What I've seen is lots of phone calls to developers and operations phoning and saying, "Hey, can you type this command into the terminal somewhere?" because they don't have access, or there's just not enough logging or telemetry data in actual to find the bug it requires a number of changes, more like trial and error.

They don't have enough information to really figure out what the problem is, so they try a lot of different options until they get there. That's a low effectiveness. The high effectiveness now is obviously a lot of teams operating in a DevOps way. So a lot of the teams themselves have all the access and information they need. They can very quickly recreate, they can have access to production data, or sanitized version of production data, to recreate those problems locally. And then they have very sophisticated observability tracing of requests through the various systems. I think one of the other problems, why a bug might take a long time to find out is a little bit of, "It's not my fault."

Sometimes it's a little bit of like trying to figure out, especially if it's a distributed system, which team is responsible and in the low effectiveness, everybody's trying to, "It's not my fault, it's got to be someone else." But in the high effectiveness, everyone understands that we're in it together and it may be my fault, maybe not, it actually doesn't matter, we're going to figure out what's the problem is we're going to help. Nobody's really to blame and we're going to work it out because we're one company and there's a user somewhere that's experiencing a bad experience and we're all hurting because of that and we're going to try and figure it out. And then afterwards you try and make the system better, but collectively, and not with a blame culture.

Shane Hastie: If I look at some of the other metrics in there, I'm going to guess that this culture thing, and of course this is the Engineering Culture Podcast, what are the culture elements that are going to make a big difference? So you've touched on the, it's not my fault, who can be blamed, but what else? What underlies that? If I'm a technical lead, if I'm a developer on a team, how can I influence this?

It’s more about culture than technology [15:31]

Tim Cochran: So it starts with what I mentioned, the blameless culture, understanding that whatever was done historically was done for the right reasons at that point in time, based on the information that that person had and nothing nefarious is happening, everybody's trying to do the best they can. So if you take that attitude and apply that to all conversations and practices, it changes the way you think. There's also a certain amount of mindfulness, making sure that people are heard and people's ideas and listened to and people have a chance to express that. Some of the examples of that in practice is the agile concepts of continuous improvement. One practice that is really good is the blameless post-mortem, that after something goes wrong, we're just going to replay what happened, just facts, it doesn't matter who was involved, but just the facts and then talk about how could we improve it.

And sometimes you don't actually have to improve it. Of course, you always want to like, oh, everything should be automated, we should have better log in and all that stuff, but sometimes it's just a matter of understanding what happened. Just understanding will probably reduce the chances of it happening next time or being able to debug it quickly. That's some of the cultural aspects. As a team, I think some of it's about what I've seen in the culture in the high effectiveness places is just really helping others. So really understanding that if you've been at a company for 20 years and you're a top developer or whatever, of course you can come in and do it faster than anyone, but that's not particularly scalable and understanding that perhaps you might be more valuable in helping others and trying to bring them up to speed. Those kinds of things, some examples of what I've seen is when there's been a problem, people offering their help without having to ask for it, things like that.

Shane Hastie: When we were chatting earlier, you touched on the concept of developer happiness and how that is linked to productivity and effectiveness. Can we explore that a bit?

Developer happiness directly impacts effectiveness [17:21]

Tim Cochran: Everyone is. If you're motivated at anything, whether it's cooking a meal or going for a run or reading a book that you're interested in, you're going to put more energy into it. So it's this concept that if you're interested and more motivated and happy, then you'll be more productive. That shouldn't be particularly unusual to people, but I think how that helps a little bit to think about is making sure your developers are happy. And a lot of it's, sweat the small things. So if you do in your one-on-ones or your surveys or the things that you do to actually understand how developers are feeling, make sure that there is a way that those opinions, that there's a forum that they can actually be helped and addressed. A lot of the small changes can help a lot.

Examples of this can be internal tooling. We invest so much in external tools, but if a developer spends so much time trying to make the e-commerce site as fast as possible so on Black Friday it responds in two seconds, but maybe the knowledge base system is underpowered, it doesn't have enough RAM or something and the developer has to look at that many times a day and it's just, even if it's 15, 30 seconds for the page to load, it can be very frustrating. And those are some of the things that are hard to really quantify, and sometimes the developer themselves may not even know, but they just know that when they've worked in a different environment, they were happy. And it's these little frictions, these little things that just over time, disturb your flow, your motivation. When you talk to someone, they go, "Open a ticket," or instead of, "How can I help you?" we'll be talking to someone they go, "Okay, go and read this hundred-page document and it'll be in there somewhere." Those little things that can really upset and de-motivate somebody.

Shane Hastie: Thanks for that, Tim. If people want to continue the conversation, where can they find you?

Tim Cochran: Well, I worked for ThoughtWorks, The easiest way is via my email. I have a Twitter so that they can find me. I am continuing to research on this topic, the first article was really an introduction. I think I touched on some of the details that are coming, but myself, some of my colleagues at ThoughtWorks and some collaborators in the industry in the original article, we worked with Spotify and Etsy who obviously are some of the leaders in developer productivity thinking and DevOps. We're continuing to get more details and also to work with other companies. It's not true, but you might think, oh, it's easy when you start as a digital native, but actually looking at companies that maybe started as a brick and mortar enterprise company and success stories about how they've actually been able to improve culture and to improve developer effectiveness. So all that'll be coming up in future articles on Martin's website and other places like InfoQ.

Shane Hastie: Thanks so much.

Mentioned

• Thoughtworks
• Article: Maximizing Developer Effectiveness
• Tim at Thoughtworks
• Tim on Twitter