Developer Experience is a Critical Issue for Organisations Today

Transcript

Shane Hastie: Hey folks, QCon London is just around the corner. We'll be back in person in London from March 27 to 29. Join senior software leaders at early adopter companies as they share how they've implemented emerging trends and best practices. You'll learn from their experiences, practical techniques and pitfalls to avoid so you get assurance you're adopting the right patterns and practices. Learn more at qconlondon.com. We hope to see you there.

Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today I'm sitting down with Abi Noda. Abi is the founder of DX and comes to us because of his research and interest in developer experience. Abi, thanks so much for taking the time to talk to us today.

Abi Noda: Thanks for having me. Really excited to be here.

Shane Hastie: The place I like to start is who's Abi?

Introductions [00:59]

Abi Noda: My name's Abi. I've been an engineering leader for the past decade, and over the course of the past five or six years, I've been really focused on the problem of helping organizations measure and improve developer productivity. So several years ago, I started a company called Pull Panda, which arose out of my own challenges as an engineering leader struggling to have concrete data about how my teams were performing or where they were experiencing friction. And Pull Panda was a product that pulled a lot of data from primarily Git and project tools like Jira and provided various metrics like cycle time and pull request throughput, things like that. That company was acquired by GitHub a few years ago where I sort of continued this work. I worked with Nicole Forsgren who had just joined GitHub at around the same time our internal DevEx group and engineering leaders to both understand how to measure and improve engineering velocity within GitHub internally but also how to build solutions to help customers measure and improved developer productivity.

And that journey led me eventually to starting my current company called DX which focuses on helping software organizations, measure and improve developer productivity and experience. And really the journey, I suppose, has been spending years trying to figure out this elusive problem of how to measure developer productivity to really come to the eventual realization that there's not even a consensus definition of what developer productivity is. But I think what leaders, developer experience teams, infrastructure teams, what they all do agree on is that there's a very tangible amount of friction and pain that developers encounter in their daily work. And if we can improve that, we improve developer productivity. We improve business performance. And that's really the problem I'm currently focused on understanding and solving.

Shane Hastie: I want to pin you down on that. What is developer productivity is? It's such a nebulous term.

There is no consensus on what constitutes developer productivity

Abi Noda: Well, so while I worked at GitHub, I actually gave a talk titled My Elusive Quest to Measure Developer Productivity. And the keyword there is elusive. I think everyone's been trying to figure this out. Everyone has an opinion, a strong opinion on what developer productivity is, how it should be measured. There's some really interesting research that has come out recently on this topic. One recent study by my colleague now, Dr. Margaret-Anne's Storey was on how developers and managers actually defined productivity in their own work. And so they went out, they surveyed developers, interviewed managers. And one interesting thing they found was that developers actually... The most common way they viewed their own productivity was in the terms of number of tasks they completed. When they asked developers, "How do you think your managers define productivity?" they similarly said, "In terms of the number of tasks completed."

When they went and asked managers, "How do you define your team's productivity?" they said performance and quality of outcomes. So I think what's really interesting about this study is that even within a team, there are not only differing conceptions and definitions of productivity but misperceptions of what other people define as productivity. So it's really a difficult thing to pin down.

Of course, another recent paper on this topic was the SPACE framework published by Nicole Forsgren and Margaret-Anne Storey and some other researchers at Microsoft. And they broke down productivity into a number of different dimensions, which included things like satisfaction and collaboration. And I think one of the really interesting things about this paper, it definitely didn't give you a single measure. In fact, it argued that there is no single measure of developer productivity. But I think one thing it highlighted was that in addition to looking at the activities of developers and the output of your team, it's also really important to measure how developers feel. And I think the shared belief by almost all leaders and researchers... I don't think there's a shared definition of developer productivity, or there may never be, but what is agreed upon is that we really do need to move beyond what's traditionally been a pretty narrow managerial view of productivity defined as number of tickets completed or pull requests submitted sort of assembly like metrics. Right. So, that's kind of my take on it.

Shane Hastie: So it still remains a nebulous thing.

Abi Noda: It is. I have a funny story. When I worked at GitHub, I actually tried to find a consensus definition within the company. So I actually asked a lot of executives around the company, a lot of managers, "What does developer productivity mean to you?" And one of the funniest definitions I got... And actually this is from the CEO, this is from that at GitHub at the time. He said, "Profit per developer." And I remember he said like GDP. And I remember thinking, "Man, that sounds very agricultural, but that doesn't seem like the essence of what we're trying to get at here."

I remember that night, I went on like Googled the definition of productivity, and I won't pull it up because I don't have it on my browser, but literally the first definition that came up said the amount of something that's produced, yada, yada. But it said semicolon, especially crops. So it was agricultural. So it was really funny. So yeah, no consensus definition. And I think that's okay, right, because productivity, even just outside of engineering what does productivity mean, I think as an industry, we know that we deal with a very complex art and practice and profession, and we know that we need ways to understand how effectively we're working, how effectively we're managing the resources that we're investing. And I think all those questions kind of umbrella into this word productivity but really mean different things in different contexts.

Shane Hastie: You mentioned the tangible friction and pain. What are some of those tangible frictions and pains and how can we address them?

Developer experience can be impacted by a range of categories [06:59]

Abi Noda: Well, if you ask any developer for complaints about their work, they'll be happy to provide them, right? And that's one of the things we did in this research study where we published this framework. Called it the actionable framework for understanding and improving developer experience. But we did a qualitative study where we went out and interviewed professional developers to understand what are the things that are top of mind for them that affect their day-to-day experience in work. And what was really interesting is that similar to the SPACE paper that defined productivity in terms of these different dimensions, we found something very similar with developer experience. We found roughly that there were a number of different factors that broke down into the high level categories of development and release, product management, collaboration and culture, and then developer flow and fulfillment. Within those categories, there's different factors.

So for example, things like code-base health, right, the quality of the code, the amount of tech debt, how easy it is to test and release code. Within the product management category, there were things like having clear requirements for what you work on, having things that are reasonably scoped down, having reasonable deadlines, having an influence, let's say, on how things are built, right? And then for collaboration and developer flow, there were things more team dynamics and collaborative things such as code review process or documentation, knowledge sharing. The ability to just have uninterrupted time at work to be able to focus on tasks in a deep way. That's something, I think, we all struggle with these days at work. And so there were really whole array of these different factors that affect developer experience. And what we found is that... Well, first of all, there's no single attributable cause of these challenges within the developer environments.

Developer experience is often not a priority for leaders [08:45]

But I think one common trend you see is that developer experience is often not the priority of leaders. It is not invested in because it is not always eminently visible. It sort of exists within the day-to-day experience of developers, but it's not something that the business has awareness of or has ways to measure and understand the costs of. And as a result of that and because engineering organizations, their primary focus is on delivering things to customers, developer experience is kind of like technical debt. It just tends to sort of accrue and pile up until it explodes, right? Until something terrible happens or until it just reaches a boiling point. And what's funny is that's actually what happened at GitHub. While I was at GitHub, this is public, GitHub wrote published blog posts about this, but GitHub paused the feature development for a quarter to focus on internal developer experience.

And this was based on the business case of, "Hey, we are moving slowly because of problems in the developer experience. And not only are we moving slowly but we're losing people." Developers are leaving the company because of frustrations with their ability to get things done at GitHub. And so, developer experience is so important for businesses. It's not just about making developers happy and excited, which are really important things but really this comes down to the amount of waste in your system. How much time is being wasted by developers in their day-to-day work? How engaged are they in their work? And these things boil down to the bottom line of a business, right? We know things like engagement. Of course, time waste, you can attribute a monetary value to that, but these are all important things. So one of the things I've been focused on since this research study is then how do we help organizations gain awareness and understanding into what these problems are within their own organizations, what the cost of these problems are, and then of course how to improve them.

Shane Hastie: What are some of those costs?

You can calculate and show the direct and indirect cost of poor developer experience [10:45]

Abi Noda: So one thing we've looked at recently is this thing called the cost of delay model. And your audience is probably pretty familiar with it, but essentially it's a measure of how change in time affects change in money, right? So it's often applied to sort of prioritization decisions around product releases. But you can apply it sort of, if you will, at unreleased things as well, sort of a crude things like technical debt. And very common and visible example of this would be something like automated tests. If developers have to run tests many times a day and each time they do that, they have to sit and wait for 15 minutes. And then assuming they can't really context switch while waiting and do other productive work, you can put a dollar value just based on head count value, right, on how much money that is costing the organization.

So a lot of the ways in which we try to help companies come to an understanding of cost is through estimates of time waste, then projected out or multiplied out by frequency of occurrence or headcount, things like that. So it's really a fairly objective calculation. Now, there are other areas of developer experience that are much less concrete. So for example, having an influence on roadmap and priorities. How do you quantify the costs of that, right? That's like cultural thing or certain organizations just involve engineers more in discovery and planning than other organizations. How do you prove that having a say on roadmap priorities matters? Well, one way we're doing that is through research. So we're doing statistical analysis of industry data to look at how things like having a say on roadmap and priorities correlates with developer engagement, developer productivity, business performance. So at a meta industry level, we're looking at this information.

Within the context of a single team, we try to find ways to help turn these things that are soft into things that are objective, right? So having a say on roadmap on priorities in of itself probably doesn't necessarily have a clear attributable cost to the business. However, rework, right, requirements churn, those are things that have clear time and costs associated with them, very common areas of waste in software development in general. So finding ways to sort of link these different phenomena to one another and use statistics to understand them and just use sort of cost of delay to understand the cost of these things to business, or some of the different types of approaches we're using to understand for ourselves and understand for customers and understand for our industry why developer experience matters.

Shane Hastie: Having identified that it matters, having identified that there are some things we can measure, what can we do?

Platform teams can help improve developer experience [13:36]

Abi Noda: I think this is a challenge that a lot of organizations are facing right now. I think in the last couple of years, we've seen developer experience become an area of interest for a lot of organizations. And pockets within those organizations attempt to try to champion efforts, form teams, make investments in developer experience. And I think everyone's trying to figure out how to do that successfully. I've yet to sort of find the silver bullet to that, but I can talk about some things I'm seeing across the industry that I think are working and not working. So I think one common pattern is the platform team. They're called different things across the industry. They're called enablement teams or DevEx teams, infrastructure teams, DevProd teams, have many names but fundamentally these are internal facing teams that are focused on improving the tools and processes across the organization. They're focused on highly technical areas such as addressing monolithic code bases and tech debt and complex build and test systems.

They may also be supporting the organization in more cultural and process oriented aspects of how they work to be more effective. And their North Star metrics tend to be things like time waste in the system in the SDLC as well as developer happiness and engagement. That's a really good trend, and I think we're seeing that investment growing rapidly across the industry. You look at a lot of the most innovative companies out there like Stripe and Chime in. They're investing heavily in their internal developer experience functions or engineering operations functions.

The role of engineering managers should be creating great developer experience [15:09]

I think one of the thing that I hear from leaders of these types of platform or developer experience teams a lot and that our research also validates is that a lot of the problems and challenges with developer experience tend to be local, meaning it's very difficult for a single dedicated team within an organization to A, even understands all the problems and challenges across the organization but then to fix them. They're often just not resourced well enough to do that, and they don't have the expertise to go understand the unique challenges of every team and improve them.

And so, I think one thing that's not yet happening that I think needs to happen for organizations to really solve and make meaningful strides in developer experiences is that developer experience becomes a shared responsibility. It becomes not a problem that's delegated to a single platform team within the organization. But while that's still important, developer experience becomes a shared responsibility of all teams and all managers within the organization.

And I think that shift will happen. We're seeing that shift happen with certain companies where executives have made developer experience a top priority. Even large companies in Silicon Valley, Airbnb, for example, they, a couple years ago, were dealing with a lot of attrition and they knew from their exit interviews that developer experience was the number one cause of attrition of developers leaving. And so it became an executive priority from the CTO. I know Google, for example, has... I don't want to share specifics, but they have C-level KPIs around developer experience and an enormous investment in their developer productivity organization that from what I last recall is over 4,000 engineers that are dispersed throughout the organization almost as solution engineers embedded into teams to improve developer productivity.

So I think as organizations begin to get a better understanding of developer experience as the industry looks at companies around them to see the investments other companies are making in developer experiences, developer experience sort of becomes more accepted as a thing that all organizations should be investing in. I think the end state will be that if you're an engineering manager and someone asks, "What's your job?" today most people would say, "Ah, keeping my developers happy and shipping things on time." I think in three years, people say developer experience. They'll say, "My job is to create a great developer experience for my team and make sure we're delivering valuable things to our customers."

Shane Hastie: How is the developer experience different in this remote space today?

Developer experience in remote teams [17:47]

Abi Noda: That's a great question. There are certain aspects of developer experience that, I think, are talked about in general, not just specific to development but things like work-life balance or connectedness. Anything having to do with collaboration, right, has become often more challenging in a remote and hybrid world. I think what's actually the biggest challenge with the shift to hybrid or remote is actually the awareness of developer experience, the visibility into the pains of developers.

I don't think that was ever easy for companies even in a co-located model, but it's certainly easier to sort of get that pulse on areas that are frustrating developers or seeing the fact that developers are in meetings all day long, five days a week. In a remote and hybrid world, these problems are even more elusive and invisible. And I think that's one of the greatest challenges for companies today, and that's one of the reasons developer productivity is also a more elusive problem in this environment is that we've taken this black box art of software development. It was already really hard to really discern and understand and observe and assess in this remote and hybrid world. It's even more difficult to even get visibility into what's happening or how people feel. And so, I think that's the greatest challenge is really awareness.

Shane Hastie: So if I'm a technical lead, a middle manager, maybe a newly promoted manager in an engineering organization, where do I start?

Advice for new leaders [19:19]

Abi Noda: I think there's a lot of different journeys, but I can kind of tell you what I think is the prototypical one. So let's say you're an engineering manager at an organization, and you've heard about developer experience. You've maybe read this paper we're talking about, and you look at your organization. You say, "Wow, we have a lot of problems, or we have a lot of room to improve." Let's put it that way. What do you do? And let's be specific. What do you do beyond your own immediate team, right? Because addressing the problems on your immediate team, at least you probably have some scope of influence over. We know that a lot of the problems in developer experience are local. So if you're a manager just want to create a great experience for your team, you probably have some means to do so. Now if you're trying to improve developer experience for your entire organization, this is very similar to the problem that has plagued developers for decades around things like technical debt that it just doesn't get prioritized.

And so, you have to approach this from a sort of business perspective. You need to present these problems. You need to present developer experience in a way that makes it a priority for the business. And I think one of the most common approaches I've seen is to not shout out from a cliff and tell everyone we need to do developer experience, we need a developer experience team but to focus on just one problem, identify a problem that's crosscutting, affecting a lot of developers across the organization that has clear ROI if solved. It could be technical debt. It might be tooling. It might be the development environment. It might be just process. I would say that that's probably one of the harder ones to influence in a cross-cutting way. And pitch a project. Don't form a team, just pitch a project. Create a proposal around this problem you want to solve and attach clear business metrics to it.

Attach, go look into cost of delay, right? Go attach a monetary value. Go attach a bottom line KPI to it around overall employee engagement or attrition, which is more and more often becoming a top level OKR or KPI for technology organizations. I've seen a lot of managers, engineering directors have success with this approach of bootstrapping a developer experience function or a developer experience investment by first focusing on just one project or one singular problem. And then using the successful delivery of that, converting that into an ongoing investment in developer experience. A permanent function within the organization that's focused on taking this sort of customer first approach, looking internally, treating the developers within the organization as their customers and figuring out ways to improve their experiences.

Shane Hastie: So lots of interesting points and good advice. If people want to continue the conversation, where do they find you?

Abi Noda: I post a lot on LinkedIn. I have a newsletter that I post a lot about on LinkedIn. I'm also on Twitter. Check out our website getdx.com and always excited to talk to anyone who's trying to learn more about or solve this problem developer experience within their own organization. So feel free to reach out.

Shane Hastie: Abi, thank you so much.

Abi Noda: Thanks, Shane.

Mentioned:

• SPACE Framework for developer productivity
• Developer Experience Framework
• DX company
• Abi on LinkedIn