Using Cognitive Science to Improve Developer Experience

Introductions [00:49]

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I'm sitting down with Hans Dockter. Hans is in, it was Berkeley you said?

Hans Dockter: That's correct, yes.

Shane Hastie: And we're going to explore the hard side of soft skills, developer experience, culture, all of the stuff that of course is deeply germane to me and to the Engineering Culture Podcast. Hans, welcome. Thanks for taking the time to talk to us.

Hans Dockter: Thanks for having me. Nice to meet you.

Shane Hastie: My normal starting point on these conversations is who's Hans?

Hans Dockter: I am a trained physicist, I have a master's in physics, but was always was kind of first computer kid generation. Started all with the C64. So programming was my passion from a very early age, and I picked that as a career, being a software engineer. And I got immediately intrigued. When I started my career, that's when test-driven development started, that's when Agile started, and I was a big proponent of that, introduced that in many teams. And when you do testing, you get automatically dragged into automation. What's the point of having a test when you don't have automation? That's the whole thing of fast feedback cycles. And so I got really involved with build systems, which are the primary tool for automating the software development life cycle, or at least the development process. And I was pretty unhappy with the tools that were available without going into the details.

So I created my own tool called Gradle, which was very soon joined by the co-founder, Adam Murdoch. And the two of us, we developed Gradle to a very successful open source project used by millions of developers, and then we created a company around that. So I evolved into the CEO of that company, great company. And our focus is developer productivity and developer happiness when it comes primarily to the part, continuous integration, building, testing. How can we make the life of developers better regardless of the build system you're using? The problems that we are primarily now addressing are the ones every developer in the world has, and I'm happy to talk more about where we see those problems.

How culture contributes to developer experience [03:05]

Shane Hastie: So let's step up a little bit. Gerry Weinberg is known for huge contributions to our industry, and he had the perspective that anytime you think it's a technical problem, it's actually a people problem. And no matter what, it's a people problem. This is the Culture Podcast. How does culture contribute?

Hans Dockter: Yes, I would say there is definitely a chicken and egg problem here. So I like your challenge here, although I think it goes hand in hand. So let me answer with an anecdote before I get into my more technical kind of perspective on that. There was this fireside chat quite a while ago between a Wall Street executive and a Netflix executive. And the Wall Street executive said something like, "If we only had the quality of your engineers, we could do all this amazing stuff." And the Netflix executive was saying, "Well, guess where we got them from? From your companies." They were going to every user group on the East Coast telling, "Hey, we have a productivity culture. We really care about productivity and a lot of developers ... Life is too short. We love our craft. We want to work in a company that really appreciates productivity." So I think that's where it starts. If you have a productivity culture, you attract a certain type of developer, if they believe that you as an organization, take that serious to provide them a productive environment.

A productivity culture is a culture where the organization is willing to invest into removing bottlenecks for developers [04:34]

Shane Hastie: So what is a productivity culture?

Hans Dockter: A productivity culture is a culture where the organization is willing to invest into removing bottlenecks for developers and not saying, "Yes, we can do this later. Get this feature out. Oh, sorry, again, let's focus on the features. Let's do this next year maybe, or maybe you can do it on the weekend if you like." So productivity culture is where you do not rely on heroics of developers to create a productive environment, but where it's a conscious decision, by a software development organization to really invest into this area and where the developers care about that. If you have an organization that invests in the productivity, but developers don't care about it. So those two things go obviously to together because it's quite abstract. Let me give you just some examples that fascinate me and that gives some more kind of meat to the bone here, what I'm talking about.

So for me, there are three big areas where I think we have tremendous immaturity and without which you cannot have a good productivity culture. So the first starts with, so okay, when you write a line of code, it's a hypothesis. Many people outside of software don't understand that. They think, "Oh, it's mathematics." No. It's a hypothesis that for all practical purposes cannot prove that it's doing what it's supposed to do. So you need, as a developer, philosophically, you need to have a dialogue with the tool chain, like a physicist needs to have a dialogue with nature when they have a hypothesis via experiments. Developers is having that with the tool chain. "Hey compiler, what do you think? Hey, unit test. Hey, functional tests. Hey, performance test. Hey, vulnerability test. What do you think about my change? Is this hypothesis valid?"

So that is the bread and butter, I think we all agree in the life of a developer to formulate a hypothesis with code and then get it validated. And that is where the crux is in various areas. First of all, when I started out with my first basic programs, the feedback was always immediate, in milliseconds, small programs. Now when you enterprise developers, we still have organizations where it takes a small change, you take 24 hours to build everything, to test everything, to get the feedback, "Hey, this is working." So this waiting time for the feedback is a huge problem and we can talk about it also from a cognitive science perspective, but that's one area where a lot of people do not invest. It just goes up, goes up, goes up, goes up, goes up.

And then the second one, I think emotionally and for the culture, even more poisonous, is you get a signal from the tool chain, "Hey, something is wrong with your change." But there's a lot of flakiness in the signal and now you are like, you cannot trust the signal from the tool chain. And for developers, you have a pull request build, it fails. What do you do? I think almost every organization in the world does when there's a failed pull request build. "Let's run it again, let's see if it passes the next time and maybe one more time." And then, "Oh, I don't know what to do. Let me talk with the CI team." Unless it's an obvious reason why it fails and it devalues writing tests for example, the whole testing culture. When you cannot trust the tests, why to write them. When you start ignoring them, why to write them. So I think it's hugely problematic for the testing culture that things are very slow because you don't run them often and that they're very often flaky.

And there's another culture problem here. So what we see very often, the pull request build fails. I, as a developer, have no idea why. It worked on my machine. The pull request build fails and they say, "Hey, CI team, platform engineering team, you need to look into this." Platform engineering team is looking into this. "What the hell? It's not our test, it's nothing to do with us. You have to look into this." So we know CI teams that get 10,000 issues per year because a pull request build fails and developers don't know why, but most of those issues they hand back to the developers. It's this finger pointing instead of collaborating, there is quite a bit of friction because no one knows who's responsible for it. Very bad for the culture between those two teams.

And then for me, the last thing, so I usually do rhetorical questions when we're talking with an organization. I ask them some very basic questions, I said, "Hey, how many feedback cycles your developers have per day? What is the average time they take? And how often do they fail? And what are the major reasons why they fail?" And I know already the answer, "We don't know. We don't know." For me, we are in a similar situation like with application performance management. In a way, when it comes to the developer tool chain, before we have application performance management, a customer had to complain, "Hey, I cannot check out my shopping cart." Today, completely unacceptable. I want to know that there is a problem. With the developer tool chain where they have the feedback with the tool chain, we have exactly that situation. It's not instrumented, no data, no observability.

And then from a culture perspective, now what does it tell you as a developer when your feedback cycle time gets longer and longer and longer? When the developer infrastructure creates so many wrong signals? It tells me, well productivity doesn't seem to be the highest good here in this company. So it's a subliminal message about how we are developing software and what we care about in my opinion. It's like a kitchen where the knives are not sharp. What should you think about that kitchen, about the food they're producing?

Approaches to instilling a culture of productivity [10:02]

Shane Hastie: At a leadership level, how do we fix this?

Hans Dockter: So that's a really good question. So it's pretty wild. When we do surveys with developers, how much more productive would you be with a more efficient tool chain? And we just had the GitHub survey where waiting for builds and tests was the most complained point. Developers say 80% say we would at least be 2 times more productive. Many say three or four times. I mean this is wild. Two times really, three times. And I think developers, I mean it's a gut feel but I don't think they're directionally wrong with that. But then why are we now saying, "Okay, let's fix that." So the problem with the leadership is it's the gut feel of the developers, the gut feel. They have no data, nothing is instrumental. And in leadership they're like, "Are the developers whining again? Is it really that important?" So they understand that you can improve the situation, but the developers think it's a transformational opportunity and the leadership thinks it's an incremental improvement opportunity.

The importance of understanding cognitive science [11:09]

And that's a big mismatch. And how do we translate the gut feel of the developers into business cases? That is where I think data and cognitive science comes in. And I'm curious on how you think about that. So the way I look at it, when engineering leaders, when you look at leaders in other industries, let's say chemical industry or car industries, I would say that the leaders there at the manufacturing level, let's say they're really good chemists, but they also understand everything about large vessels and alloys and what it takes to produce chemicals. Same in automotive. There are car people. At the same time, they understand robots and all of that. So our leaders, they very often understand a lot about software, but what is our means to produce code? It's our brains. And what do they know about brains? At the end of the day, that's what you're paying us, the developers, for. You're paying us for our brains to produce output. It's output from our brains.

But if you don't understand at all how they work, how do you make now decisions? There is a lack I think, of understanding that we really need to bridge from cognitive science to make more educated decisions. And it's even worse that sometimes you are aware that you don't know something. But a lot of leaders, they have very naive models about how the brain works and think that's the truth. So they don't even know what they don't know. And yeah, I'm happy to talk about that, but I think that's where I throw an analogy to other industries. And I think where we have one deficiency at the leadership level, it don't understand enough about cognitive science in as far as it is relevant for knowledge workers like software developers.

Shane Hastie: So let's dig into that cognitive science stuff. What does cognitive science tell us that we're not picking it up?

Hans Dockter: And again, I'm not a cognitive scientist obviously, but cognitive science is the very important concept of cognitive control. There are two ways our brain produces output. One is when you have established neural pathways for things like, I don't know, when we're walking, we don't need cognitive control for that. We have the input and we have all the pathways to know what we have to do as an output. And I think a similar example would be professional chess players and chess opening. You can wake them at 3:00 AM, they make the right opening. So different for me. So the interesting thing is the learned effortless routine is not trivial stuff. Let's say when we're walking through a forest and not running against trees and falling over, this is complex at a mental level. This is complex stuff, but we never get exhausted from this. We get exhausted in our muscles, but not from what we have to do at the brain level.

But for activities where you need to apply cognitive control, cognitive control means you have the inputs and you know what you want to achieve, but there is no pathway yet to get to this goal. That's the miracle of human intelligence that we are able to do this. And that's when you look at chess players, again, that's the middle and the end part of the game at a certain point, the end part, not anymore, but the important part where the games are lost and won. And that's where cognitive fatigue sets in, where after four or five hours chess players make mistakes they wouldn't make two hours in the game. And that is something ... I mean no one is surprised by the concept of cognitive fatigue, but we are completely ignoring this in software development. And when you look at Elon Musk, hardcore, just work harder, don't sleep. It just doesn't work that way.

Managing cognitive load [14:45]

So when you now look from a software perspective, cognitive fatigue is not a bad thing. Ideally I start my work, I produce a lot of great code, and then at the end of the day I'm cognitively fatigued and I go home. Great. But the problem is with software that there's too much unnecessary stuff happening that accelerates cognitive fatigue without getting the output. That's the problem. And one other very important part is to understand the problem is that context switching is accelerating cognitive fatigue. And neurologically it makes sense when we have a certain problem we want to solve with cognitive control, we really have to create a dynamic pattern in our brain that takes energy to kind of build that to get in a state to solve a particular problem. And when I switch the problem, I have to rebuild the inputs, the outputs that I want to achieve or the goals. But I see a lot of naivety around that.

I'll give you an example. Leadership and developers. I had situations where we worked with the developers and we said, "Wow, we can make the feedback cycle twice as fast. Get it down from 10 minutes to 5 minutes." And developers are, "Wow, this will be so good for productivity." And then there were leaders that said, "Well, why is that important? While they're waiting for the one thing, they can do something else. We are not losing any time here." And when I told this to the developers, they were dumbfounded. How can someone look at this this way? But they do. I can tell you. "Well, do we really get business value by making things much faster?" And they are right. If there were no cognitive fatigue and context switching wouldn't cost anything, then they're right. You just switch back and forth, 100 things in parallel. But that's not how the brain works.

And now my theory, and this is kind of speculative, a lot of people that moved into positions like I have, we have to do a lot of multitasking. We are not very often in a state of flow. That's not what we're getting paid for, which is a problem. But there are a lot of experiments in cognitive science. So multitasking is super interesting, which is very related to context switching. People say, "Oh, this is a skill everyone needs to have. The young generation is good at multitasking." My wife always says, "Women are better at this than men." And it's something you can learn. The reality is this is all not true. Multitasking is not effective. There's so many studies and it's very easy to measure. One group does certain tasks sequentially, the other does them in multitasking. And the time it takes to solve the task and the quality of the output is substantially lower.

But the fascinating thing Shane is if you ask the multitasking group, "How productive were you?" They said, "We were great." So now think about leaders that are pretty confident about themselves. They have to do, for example, context switching, multitasking all the time. How can they not be good about it? We have this big title, and that's another thing in psychology to get a little bit more serious. When you do something all the time, and it's culturally interesting just because you do it all the time, you think you're good at it. But those are two completely separate things. You never are really good at multitasking. You just think you're productive, but you are not that productive. And what I'm saying, multitasking and multiple streams of works is not the same way.

Let's say an author, they often work on multiple manuscripts. They work on an introduction for this one story, and then they kind of get stuck. Said, "Okay, I need to let this rest for a while. I want to work on something else." That is very intentional, working with different levels of focus. We are talking about you're deeply immersed in something, being your pull request build fails or you've got a flaky test to deal with. And I think cognitive fatigue, it's almost like how much fuel turns into motion versus dissipates. It's the same thing, how much cognitive fatigue of your developers you really get into code versus stuff that is wasteful.

So I want to mention one more thing. Not all software development tasks require a lot of cognitive controls. I can do trivial refactorings, I can add new lines to my code, there's a lot of things to be busy. There's an infinite amount of busy work I can do. So people, let's say, especially with the whole productivity paranoia we have nowadays, people working from home, you see commits, you see activity. Even when they have cognitive fatigue, they're not going home, but they're not doing the stuff you actually pay them for because you have accelerated the fatigue. Now they have to do other stuff. So there is no easy way to detect that. "Oh my people are now in a state of cognitive fatigue." But what is clear is the number of context switching incidents they have to do, wait time, everything that takes them out of a flow. Cognitive science is very clear about that. That is an indicator for cognitive fatigue, acceleration and you have to minimize that. That's one of the most important thing you have to do for your developers.

The real impact and value of AI tooling to reduce cognitive load from simple tasks [20:00]

And now the last thing I want to make is think about Copilot and things like that. The busy work will be done more and more by AI. I think we can all agree on that. So now imagine the competitive advantage for a company is really good where the cognitive fatigue sets in very late. They don't have to do the busy work anymore. They can just work on the value-creating activities. And now the organization that is not investing into that, there's a huge gap. I think that's what engineering leaders have to understand. That's where things like the space framework I think also helps us finding terminology and a joint language to talk about this stuff.

The disparity in investment on developer experience [20:39]

Shane Hastie: One of the points that you made in our conversation before we started today was the big mismatch in investment that companies are making in developer experience. For some organizations, it's as much as 20% of the developer workforce focused on removing the friction. And for others it's less than 0.1%. Why does that mismatch exist and how do we fix it?

Hans Dockter: It's a very good question. I have a couple of thoughts on that. When you are responsible for budget and then someone tells you, "Actually, out of our $10 million budget, we now need to invest a million into productivity from, I don't know, $10,000." It's like ooh because it is a serious investment and we don't have the culture yet for that. So people feel like, because not everyone else is doing it yet, "Do we really have to do this?" And it's like you want to have your cake and eat it. "But I want these features in it. So I want to use that money for getting new features out. I don't want to deploy resources I wanted, but I don't want to give anything up in the short term." I think it's that. So it's pretty immature, but human, I would say.

So for example, when you look at a chemical factory. If someone in the factory would have an idea to improve the efficiency of something by 1%, they could call the manager at 4:00 AM at night. "Oh, amazing idea. Let's do this tomorrow." Because they have such a culture around that. There’s no question, "Is this really relevant?" It's deep in their DNA. So it needs to get into our DNA. I think one thing that is super helpful is that the Big Four, well I would say Google, Microsoft, Facebook, Amazon, they're taking that serious and they're now starting to write scientific papers on that topic. How for example, every second they save on feedback cycle time can be attributed to more code that is produced. So they have the means of a large sample data, their workforce and all the skills to do this kind of work. And it's already working. They're talking about this and I think that is already changing the perspective of people.

So I think that's very important. We need to have more clear evidence, not just at a generic cognitive science level, but really, "Hey, let's do this experiment and let's compare two groups and things like that" so that people really no longer wondering, "Do we really get business value out from that?" And then I think it's just the typical inertia thing. Organizations doing it more and more. And then you start having FOMO at one point. "Oh, they're now investing into this. Now we better also do that." So I think we are on the right track, but there is still a lack of conviction.

And final thing, sorry, I think that is absolutely instrumental. It has to start with instrumenting all of that. When you don't know how long your feedback cycle takes, how can you even improve it? How do you even know how big the problem is? How do you even know where to start? So that is the first thing. Just get the lay of the land, instrument your tool chain, the local NTCI workflow for developers, the test workflow, and look at the numbers.

Shane Hastie: Those numbers will be uncomfortable.

Hans Dockter: Yes, they will be, absolutely. But at the same time, the good thing is we're now at a point where you can improve a lot of those problems significantly. So yeah, they should be uncomfortable, but then there's hope, a lot of those problems are solved. And then we come back to the culture and I don't have a good answer yet because this has to work together with the developers. Let's say if you have developers to say, "I don't mind my 20-minute feedback cycle time, I can have a lot of coffee that way." Then of course that will not go together well. I think it's important that you have a culture where you have developers that really love their passion and want to get things done. And then a leadership that says, "We do everything we can to get the bottlenecks out of the way." And I think it will become more and more self-selecting. I think the more companies provide better productivity environments, the lesser not so productive companies will be able to keep their good software developers.

Shane Hastie: Hans, a lot of strong points and really interesting things to explore here. If people want to continue the conversation, where do they find you?

Hans Dockter: Yeah, they find me on LinkedIn. That's where we're doing nowadays, a lot of our stuff. So look for Hans Dockter, D-O-C-K-T-E-R. We also have written a book, it's still a beta version called The Developer Productivity Engineering Handbook, where we're saying, "Hey, we need, similar to chemical process engineering and automotive engineering, we need a discipline."

I mean, I have one funny story to tell. So in those other industries, the people who take care of the efficiency, they have the highest respect. You can study this, you can make your PhD, you can become a professor. I mean, those are disciplines, they have a deep foundation. Not long ago, I would say 5 or 10 years, the people who worked on the software development process were the people considered not good enough to write a production code. And it was the same way I know back in Europe where basically when you failed as a developer, they would put you in the testing team and then you could redeem yourself. So I mean, it's a similar thing. So there is that cultural aspect. The software development process is complex now with all the AI possibilities, we need outstanding people working in that area and not people we think are not good enough for writing production code. This is all changing, but it's not that long ago that it was different. So I find that kind of funny if you think about it.

Shane Hastie: Yeah, I worked for a company where the developer who had the most bugs this month got to be the tester next month. Hans, thanks very much for taking the time to talk to us today.

Hans Dockter: Yeah, thanks for having me.

Mentioned:

• Gradle Inc
• Hans Dockter on LinkedIn