The State of Engineering Management in 2024

Transcript

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today, I'm sitting down again with Andrew Lau. Andrew, welcome. Thanks for taking the time to talk to us.

Andrew Lau: Shane, thanks for having me back. It feels like it's been a minute, but I love it from last time. So thanks for having me here.

Shane Hastie: So the reason we're back is the 2024 State of Engineering Management's Report.

Overview of the 2024 State of Engineering Management Report [00:30]

Andrew Lau: Yes, I love it. We've been able to do this four years in a row, it's our fifth annual report, and for me it's actually super interesting because we get to see changes over time, and this year we surveyed 600 full-time professionals in software engineering, but not just managers, but contributors and executives. We saw the whole layer there. So it gives us a cross-cut in seeing how people perceive the world there.

Shane Hastie: So the 360 view of the engineering manager.

Andrew Lau: Yes.

Shane Hastie: So what's the state of, in a single sentence.

Andrew Lau: When I step back and look at this, I think it's a reflection of how the world is still in change, right? We are in a world where hiring is still in flux, right? That there's still a shortage of candidates in the market out there. I think that there was a theme of AI continually showing up here and really being enforced and actually, frankly, that there's still a gulf there around what people think is actually happening there, what's actually happening on the field, but also, again, back to change. It really comes down to also around how we think about how we do work today. It's a time which in the last few years have been pretty tumultuous, right? So you also see how we work and how we actually engage with our teams, it's just causing a shift through the system, right? So I actually think the state of engineering management is really just a snapshot at a time where we're all still trying to figure it out, right? That's the theme I see.

Shane Hastie: So let's dig into some of these themes. Hiring in flux. There have been reports of job losses and gains, and confusion would be what it looks like from where I sit anyway.

Burnout is increasing [02:14]

Andrew Lau: I would agree, and one area that I saw that dovetails with this is 65% of respondents said they experienced burnout in the last year. So this is dovetailing into staffing shortages, change of work, change of medium that we do work, right? You and I are lucky that we can do this over Zoom, but we're now, what, three years into this hybrid environment and we're still learning, right? I think we've proven that we can do it, but is it the same warm and fuzzy we were used to? Unclear, right? And you've got economic uncertainty out there, but at the same time, you've got a change and there's still a talent shortage out there, where it's like, what a juxtaposition all these things are in.

Shane Hastie: Burnout, 65% of people reporting burnout. What are organizations doing to help this, or is it just being ignored?

Andrew Lau: I hope we're helping to it. So when we actually looked at the survey, we were looking for things that people identified as issues. In this format, we don't fully have all the remediations in play. So I think our role in this is to point out this is an issue. We're still an issue, we're increasing it as an issue. I think it's on us as a community to figure out how we're going to remediate this and how do we help. Our contribution is to identify this as a problem, if you don't point out the issues in that way, it's unclear how you'd ever resolve it. And I think this is one of those things where as individuals, it's hard to perceive sometimes the burnout because we're all in this journey together.

We've gone through a pandemic, we've gone through economic crises, we're dealing with work idiom changes, we're dealing with AI and movement, and changing the way we work. And so at the very least, each of us are crossing these thresholds alone. Some of us may have resolved it and figured out how to negotiate the waters, but this is a reminder that 2/3 of our collective community has not figured this out, and answer to your question, how do we solve it or how are we going to solve it? I think step one is, talk about it. Step one is to figure out what's working for who and why, and then maybe we can find solutions here.

Shane Hastie: And that's embracing AI or not. You made the point, the gulf between perception and reality. What is the perception and where's the reality?

The perception vs reality of AI in the developer workplace [04:39]

Andrew Lau: Yes, so first of all, your audience knows this, but we're at a time where generative AI, LLMs, copilots, agents, they're all in flight. And I think all of us are dabbling with it, if not fully in it, right? You've seen it in the media. You've seen this thing, and I think intuitively, I think everybody thinks there's going to change things. I think on a positive side, you look at some of this stuff and you're like, "It's magic. It's amazing what it actually can do." But there's also a fear side of it, which is what is it going to do to me in my role, and where do I fit into this stuff, right? There's this strong juxtaposition of it. So I think deep down we all intuit this is happening and so there is a gulf out there, right?

So we actually surveyed executives like, hey, are you using generative AI here? And 76%, 3/4 of folks said, "Yes, we are." Okay, in that same environment though, we actually asked those related engineers, are you using it? And the answer was only half, 52% said, "We're using it." So there is a gulf there around perception of usage and actual usage on that stuff, and that is the gulf that I'm talking about. But I think if you ask the whys, it actually goes back to my first comment around what's actually happening here. There's this fear juxtaposition going on here. So I think if you talk to an average business person, I think they look at this as an imperative to be using AI, right? So I think there's two things going there. As a business person, there's fear and greed in play, right?

So the greed part of it is like imagine what could happen. I see all this magic happening. If I could apply this magic pixie dust to my company, my business, either I'm going to get more efficient or magic things will come out of this machine. So there's a, let's push into it just for the greed value,  let's try it, see what's going to happen. And then the fear factor on that is if I don't do it, my competitor or my industry's going to do it and we're going to be left behind, right? So I think there is a business imperative to capture, bottle, utilize whatever's actually happening in this world of generative AI. Now, on the other side of it, you look at the folks making, the engineering team, and they're the ones that use this. They're often the ones that identify the magic, see it first, but there is a time of juxtaposition, and change, and fear.

What is this going to do with my role in this? What does this actually mean? Is it actually any good? And these things, and so I think that is the goal for the reluctance in that space, which is the intention and who's affected by this is some of this catalyst here. My observation though is it's not like this change isn't coming. It is because I assume, Shane, and it's like what do we do about it as an industry here? It's true, but you and I have seen how this movie goes. You can't stop the progress of technology in this stuff. It's coming. So I think to me, the worst thing to do is put our fingers in our ears and just pretend it isn't happening.

I would say instead, you have to acknowledge and capture that fear, and figure out how you're going to utilize it as opposed to letting it happen to us. I think we have to drive it to be conducive to us in that space. My intuition is it will drive change. I think everyone's intuition is it'll drive change. I'm just saying, just putting your fingers in your ears isn't going to actually stop the change. We have to embrace, engage and drive it to a thing that is actually conducive to helping us make progress here, is my two cents on it.

Shane Hastie: And how do we do that? And this is outside of the report, but how do we do that ethically?

Andrew Lau: Well, ethically in terms of who, right? So you could talk about roles changing, you could talk about rights holders of the content. There's many layers of what ethically can mean, and I probably have tackled all of them, but which one were you referring to?

Shane Hastie: A broad, general view on it, but also making sure that the change is good not just for the individual, for the organization, but for society as a whole.

Making change ethically [08:34]

Andrew Lau: Ooh, that's some deep thinking there. Look, I hope that we're making progress in society as a whole. I think there's a lot of dark things that people could imagine on that stuff. We have to look at it in the spectrum. I think it actually, I still fit into the same statement here. I think if you fear tough things actually happening here, still the answer is to not put your fingers in your ear, ignore it, right? Because someone else will capture it and drive it forward, right? I think this is where technology progress happens. I would say, engage with it where you actually steer it towards the good use cases, I think.  And however you define good in this case and away from the bad, because if you just ignore it and hope the bad doesn't happen, actually unfortunately, I think there's a bit of that that does happen. So if you were talking about people abusing the usage of it or hackers, well, maybe engage with it to try to figure out the counter case and how to make it to prevent that, right?

You look at AIs that detect AI usage, or also look at it the other way. So I think again, you've got to lean into it to actually help solve the problem. You didn't jump on it, but I'll say the other couple of things that I have heard, I posited too. One is actually rights holders and all this stuff, and I think every nation state is going to be putting on rules in place in this stuff. And you're seeing actually in the US here, some of the litigation start to actually happen on the media consumer side with respect to this. And I think that'll blaze the trail and answer that question there. So I think that it'll be rules set in place by how legal treatment, of what you can train on, not just what can you actually synopsis. This is going to be things that are answered in the consumer with big media perspective.

The potential change in roles as AI is adopted [10:17]

The other one is, I think we have ethically around change of roles, and I think that's probably the one that is probably closest to and most germane to the community that you and I are part of here in this. And I think there is changes coming here where, for example, our collective roles will start to shift. And so let's look at a software engineer. I actually think that, increasingly, it's not binary, but it'll shift, right? The act of raw coding will start to change and start shifting, where more of the code generation will be coming from these agents, and copilots, and things. And so the role will shift to more to review. And furthermore, I think we're going to up-level more and actually probably live in a world where software engineer is more bubble architecture, and maybe as software engineers we're going to become more like product managers, because if we're doing bubble architecture, then you actually ask if the code's doing something and how are we actually going to fulfill whatever goal we're trying to figure out.

We have to ask here, what's the goal of this thing we're making, right? Is it fulfilling that person, or that need, or that system, regardless of whatever your aspiration is in that space. And so software engineers, maybe we start becoming more like product managers in some ways, and again, I am cartooning this a little bit, right? And I had a friend we were just talking about this weekend, he and I probably started out maybe a touching assembly, a little bit of C, and in some ways we were actually having the same rhetorical discussion maybe 30 years ago here, around what's going to happen here? Is this skill going to go away? And maybe there's echoes of that too, where actually what happened in that evolution?

Well, frameworks came into play and if you look at the way that a lot of folks are developing today, it's a higher level construct than we're actually doing down in assembly or C. And now it's actually just even going further high-level in that case, and it's more of an orchestration, product manager, bubble architect kind of way. But it isn't to say that in today's world, if you actually still coded assembly in C, you're probably a special skill set that is actually pretty sought after too. So it's possible that you see a parallel shift here in that case. And if that's true, then if you draw that analogy, if you believe that analogy for a little bit, then maybe we should learn the new framework today. We shouldn't engage with it, we should actually... Or dig in and decide you're going to be that C assembly-level programmer and go deep in that space. But I think there might be lessons from history here, that we can learn here, that might play out. Or you can believe me or not, I could be making up stories here. I don't know.

Shane Hastie: Well, I certainly, looking back on my own career, like you, I was in Assembler, and COBOL, and C, and yes, stepped into frameworks and haven't written much code in a while, but at least still enough to understand. One of the things that does concern me, makes me wonder, and this is a theme I've asked a few of our speakers recently, is as we step into these frameworks, experienced engineers can do that easily. How do we help novices to have the base knowledge or do they not need it?

Andrew Lau: Yes, okay. It's interesting. I actually probably take an opposite posture. I would actually argue that it is much, with these tools, look, I am sure you play with the ChatGPTs and all the... It's actually pretty easy to engage with it because the language is very easy to access. Or if you look at copilots, you're already in the ID, it's just coming back to you with the Lingua Franca. I would actually argue the engagement vehicle is actually easier today than it was. I'll posit it a more interesting thought, and someone brought this to me earlier, which is, look, if you look at the way that you and I might have started our careers, there was a little bit of an apprenticeship path, right? You fix the bugs first, right? You fix someone else's bugs, and then you got better at doing that, and then you started writing your own and maybe you reviewed, the guy or gal that was next to you, reviewed their code. And you started at the lower-level pieces and then you moved up to eventually being an architect. You orchestrated a few modules together in these things.

There was a pathway of learning on that part of it. The question is if the rudimentary, low-level bug fix or the basic block of code is no longer the thing you need to do, what is the training path for us to become more seasoned over time? And maybe the answer doesn't matter. Maybe you just start at the higher-level construct and it's fine because we're so frameworks on up. But it is interesting, right? We come up through a profession which is seeing that evolution and what do we do here?

Shane Hastie: Interesting thoughts.

It's going to be fun to watch things play out.

The gulf in expectations and understanding [15:25]

Andrew Lau: Yes, fun is right. Look, at the same time, we see this actually in this discussion, you see this gulf around how business people think actually we should be. Executives be believing the team has embraced AI and then 3/4 there and only half of the engineers think so. So I said the gulf was an intention.

The other one too is I actually think that we haven't agreed upon what's actually going to happen here, right? So you and I are riffing on what's the theorized, big picture, longer term part of it. But I think we're actually not able, super well today, to even measure and talk about what are the effects today, right? So one reason is it's fast moving. This is an area that we spend some time on and thinking about. Right now, there's no widely adopted way to measure and manage this AI adoption part of it. So we're spending a lot of time in that area because it's so fast and changing there, but also with the out of actually the empirical way to actually do this, I think you end up with these gulfs because now we're just clear [inaudible 00:16:23]. What are we doing and why are we doing this? And so I think that's a rapid regression we're going to be seeing down the path here, right? We're spending some time in there around measuring copilot impacts and stuff like that.

Shane Hastie: Digging back into the reports and the metrics- what are useful metrics that tell us about engineering and engineering management today?

The importance of empathy in managers [16:47]

Andrew Lau: Well, I actually think back to the actual shortage of candidates, right? And I am jumping back to that a little bit, right? So the two call-outs for me, around burnout and candidate hiring, are to me a little bit the surprising part of it. 2/3 of folks are still burnt out. I think it's important, if you're an engineer manager, if you're in leadership in that space, to realize that 2/3 of your folks are baked. And to a certain degree we think that, oh, we're through this pandemic. We're doing all these things, everything's fine and easy. It's not true, right? We're all getting pounded, constantly, pounded with changes, pounded with other crises, pounded with changes in patterns. So as a manager, be empathetic. That's a take home. Ask, are your people doing okay? What do they need help with? This is classic management stuff, but it's a time that's really important. So I think that is one that, to me, is a little off intuition, that I think leads to some action here.

The other one is around qualified shortage of hiring. So we just saw, in the last four years, pandemic and so hiring stopped and then a huge surge of go go, '21 hiring. And then I think some more economic challenges recently. So it's a pullback and I think a lot of us are in the trenches actually in the hiring, and you're like, "Oh, it seems like the hiring's at a pullback, so it can't be a thing." But we're still seeing six out of 10 companies are still reporting a shortage of the right candidates there. And so this is again an off-intuition thing, and so that means that, look, you're investing time in your team, in training them, investing in them, building them, building a team together, right?

So aid, again, to be empathetic to your team members. Don't let them burn out and walk out on you. The other one too is, to invest in those folks and grow them because it's still hard to find folks. Even though you might think the economic climate shifted, it might be easier in that case. It is still hard out there and the team members too, to stand up and to have some pride in themselves too. They're doing something special there. And so for the right candidates, it's still a candidate's market in that regard. The people still are not feeling like they're finding the right, best people always out there. To me, those are things that ring home around being a manager in the state of today, which is in some ways both of those are off intuition and neither of those comments have to with AI, right? And I think they are actionable and they come back to, I think, respecting your team, right? I think that's a simple message there.

Shane Hastie: And pragmatic, practical ways of improving or supporting high performance in teams. What can the manager do?

Pragmatic actions managers can take [19:40]

Andrew Lau: I sound old school here, but I think it's ask your team, clear roadblocks, but also to communicate to make sure that everybody's on board around what direction you're going. What are you trying to get done? What is the business impact we're here doing? And actually, as much as we're talking about the dichotomy of executives, manager, we surveyed the whole ball. In some ways, it's an interlinked group of people. I'm just separating it up for this discussion here today, in that part of it. And I think it's important to remember that this team is there to actually accomplish business goals, right? I think it's easy to talk about these things at odds. They're not at odds. We're still trying to actually make this. And the one thing that actually was really important, that I saw there, is that over 90% of folks actually said that their engineering team is actually shaping business strategy. That's huge, and so look, as someone that started out as an engineer, is in the engineering industry, I love it.

They said software ate the world. It's true, right? 94% of teams actually said that their engineering is helping the business grow. Okay, so software ate the world. Every company's in software engineering today, they're looking to the software engineering team to define business strategy, a strategic partnership, okay? I'm proud because we've gotten to where we should be. This is the big show. We're driving the business, but that also means that we're pushing for a business outcome. So I think actually as a leader, back to your earlier question, what do we do here? We have to make sure high-performance teams know what they're playing for. It's not just blindly doing stuff. It's not like CD for the sake of CD. No, we're actually trying to accomplish a business objective and make a dent out there.

So congratulations engineering team. You've got the hand on the wheel. You are setting business strategy today. Okay, now the pressure's on. You've got to steer the ship, right? You can't just press the pedal down for kicks and go fast. You are going somewhere, right? And so as a leader, as a manager, you're really the bridge to the business, and so really setting that stage and figuring out how to point the team. So I actually think to be a high-performing team, you have to be going somewhere, and as a leader, you're articulate and connect, where are we going? And that's the ultimate measuring stick, which is, it's not that you're blindly going fast, are we're going fast to our business objective? I think that's the biggest role that a leader has in this.

Shane Hastie: A lot of really interesting points, had a wide-ranging discussion, which I really appreciate. What's one piece of advice for the engineering manager today?

Andrew Lau: Oh, I think in this time my simple one is to figure out how you're going to harness AI, not AI harness you.

Shane Hastie: And if people want to continue the conversation, where do they find you, Andrew?

Andrew Lau: You can find me @amlau on X and you can drop an email. I'm happy to do that, or catch me as A-M-L-A-U on LinkedIn too, all three of those work.

Shane Hastie: Thanks so much for taking the time to talk to us today.

Andrew Lau: Shane, thank you for making time. This has been great.

Mentioned:

• State of Engineering Management report
• Andrew Lau on LinkedIn and X