The Evolution and Challenges of Engineering Management

Transcript:

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast.

Today, I'm sitting down with Peter Gillard-Moss. Peter, welcome. Thanks for taking the time to talk to us.

Peter Gillard-Moss: Yes. And thank you for inviting me.

Shane Hastie: Our normal starting point is, who's Peter?

Introductions [00:58]

Peter Gillard-Moss: So, at the moment, I've worked for a company called DeepL, which does AI language translation, and I'm working there as a senior engineering manager.

But if I go all the way back to the beginning of my career, I actually started in IT support back in 2000. And back then, you could become a developer if you could write a web page. So, I was really lucky that it was quite an easy route into software development back then.

So, I got involved in lots of side projects and then got into software development. And then, had to kind of very ... Like many engineers at the beginning of their career, moved around a lot. Went into media, went into banking, and then eventually I ended up at somewhere some people might know called Thoughtworks, is a consultancy. And I was there for 16 years and for about 10 of those I was in a leadership role.

And at the end of my tenure at ThoughtWorks, I was responsible for over 400 engineers across the globe. Now it's a bit less, it's a bit of a smaller team, which is what I wanted after such a big experience. Most people are in Europe. But it's also been a great opportunity to get back closer to the teams as well, rather than being more far away in the more VP roles.

Shane Hastie: When we came across each other is through QCon London. And you gave a talk, Help I Didn't Realize I Was a Manager Now. What's the hypothesis there? What's the key message there?

Recognising the importance and value of management roles [02:33]

Peter Gillard-Moss: The thing that I found is I've been coaching ... When I had these 400 engineers to be responsible for, there were a lot of leaders going through over that 10-year period, and a lot of people going into engineering management. And I just found a continuous pattern of people I was coaching, and even still when I'm interviewing people for engineering management positions, I often ask, "How did you get into engineering management? How did you find it?" And I still often see the same pattern, which is a lot of people fall into engineering management by accident. It's not an intentional choice. It's often they're the most capable or seen as the most reliable pair of hands in a team. And then, I'll give them the job and they're often not sure what that job is.

And strangely, one of the things that I think happens over that journey for a lot of people is they suddenly learn the manager word and the management word and they're like, "Oh, there's all of these responsibilities that I have to have, that aren't to do with technology". And I think as well, there was especially kind of post-Agile movement, a little bit of disdain, should I say? For the word manager. People didn't like the idea of management. It was seen as something very heavy. And I went through that experience myself of being kind of adverse to management concepts for a while.

But then, laterally in my career I realized that these were actually important things and there were responsibilities there. And so, you need to embrace that. So, really, there were two things is, one was it is really important that experts lead experts, in my opinion. And having engineers and people who have come from an engineering background or been involved in engineering. So, even people who may have been say, QAs, QA engineers, going up that management track and being responsible for engineers, I think, is really important. And then, the second part of it is we need to support that journey much better because otherwise, engineers are not going to want to do it. They're going to keep walking away from engineering management.

Shane Hastie: All too often we see the pattern of we take the best engineer and we make them the worst manager and give them absolutely no training and support, and wonder why they walk away from that role.

Peter Gillard-Moss: Yes, exactly. Exactly.

Shane Hastie: So, over your 20 years and working in leading and bleeding-edge organizations, what have you seen the evolution of the role of the engineering manager?

The evolution of the engineering manager role [05:14]

Peter Gillard-Moss: Yes. When I first started, I mean, IT was the department that most developers probably started in and IT tended to be a very, very separate organization, not so connected to the businesses we have now in product orgs and big tech orgs. So, they weren't at the heart, they were kind of at the side.

And back then, most of the teams I worked in, you reported into IT, which may have been part of operations. And the first engineering manager type job I had, my boss was a project manager, not another engineer. And actually, what was kind of weird is the project manager would be there temporarily, my boss, whilst I was on that project. And then when I was on another project, I'd report into another project manager.

So, whilst my responsibilities at the time may have been for my engineers and doing the people management stuff, I wasn't responsible for delivery. That was a project manager. And then, you had other roles like BAs that would rotate in and out. And I think that was the structure for a while. And then, when Agile came along, the people management stuff kind of went away a little bit. So, there was this period of time when the project manager took over the people management in some teams for all of the developers, and who was the most technical senior technologist was just called the tech lead. And the tech lead was probably closer to what we call a staff engineer now, really. And therefore, you had this period where the most senior technical person had no people management responsibilities.

And then, I think that shifted again when product kind of teams came along and you got this kind of big tech which was actually engineering managers should be engineers, they should do people management and they should do delivery responsibilities. So, you see things like project managers disappear from teams and then become something outside of the team, and you see all the people management happen directly through the engineering manager.

So really, I mean, the size of teams I worked in at the beginning of my career, they were very big. They were pools of people that moved around on projects. Then moved into big teams that had scrum masters, product owners, project managers, BAs, QAs. And now they've shrunk right down to being product managers, designers, engineers. And I think that's changed the responsibilities of the engineering manager a lot over that time.

Shane Hastie: What makes a successful engineering manager in today's realm?

What makes a successful engineering manager? [08:02]

Peter Gillard-Moss: I used to say it was people, process, technology were the three pillars. But I was interviewing someone for an engineering management position and they said product, people, process, technology. And so, I think they were really right. So, I now add product to that list. In some departments that might be business rather than product, but it's having some sort of business or product acumen. Like to be a successful engineering manager, you have to understand what the goals and the objectives are for your customers, for your users and for your business. So, you have to have some business savvy.

The people part is being responsible for the team, making sure the team is high performing, that people are working well together, and things like, even simple things, even the manager-y aspects of people management, like making sure that stuff can keep moving when someone goes off sick or goes on vacation. There's even that kind of logistical part of people management. And then, there's the growing of people and making sure that people can engaged and motivated and that's really important. So, to have a high-performing team, you have to have engaged and motivated people.

From a process perspective, it's really ... I mean, you can look at it two ways and it depends on the organization. It might be delivery from a more project management kind of delivery, like being able to take what product you're trying to achieve and break that down into work that can be delivered. But also from a kind of day-to-day process. So, more and more, people are talking about things like cycle time and DORA metrics, and it's the engineering manager's responsibility to understand those things and make sure that they can move those things in the right direction.

And then, of course, you've got technology, where most engineering managers should have come from and be comfortable with. So, you're talking about technology choices or helping product understand how technology should be used to obtain their goals, all the way down to technical practices and what are good engineering practices as well.

So, it's a very, very broad set of things, but generally an engineering manager, I don't think you're going to ever ... There may be a few unicorns out there that are excel on all four, but in my experience, you generally find engineering managers are more like graphic equalizers, where you'll have some that are very good in two or three particular areas and then the other isn't their strength or isn't their passion. So, you also have to know as an EM how to compensate for those weaknesses that you're going to have naturally as well.

Shane Hastie: That's a wide list if we think about it. Products, delivery, people, technology. There's a lot there, a lot of cognitive load. You made the point of the graphic equalizer metaphor, which resonates with me, but I can't afford to have one of those dials fully at the bottom.

Balancing the four areas of responsibility: product, process, people and technology [11:16]

Peter Gillard-Moss: No. No. As the EM you are responsible for, from the team perspective, making sure that it appears that all of those dials are at the top. And I think that's where the people aspect comes in a lot. And it really depends on who's around you. So, that shouldn't just be your fellow engineers, but let's say, for example, you're an engineering manager who's ... Technology is not your strength. Or, it might just be that at this time technology is not where you should be focusing as an engineering manager.

So, you need to find the person ... And this is where that delegate word comes in. You need to find the person you can trust and rely on and be able to pass those things over. So, if you've got a strong senior technologist, then you want to rely on them and make sure that they can take the lead in the technical decisions for you. If you haven't got a strong senior technologist, then is that a hiring decision you need to make? Or, do you need to ask for some external help from someone else or something like that, if that's not where your strength is?

Likewise, being strong on product is something you need, but a lot of that will be down to the dynamic with your product manager or your product owner, depending on how things are working. If they're very, very strong themselves ... And I mean we're assuming that a product manager has very strong business acumen, but if they're a really good communicator and they're really good at coaching, then that's something that maybe you don't need to be as strong as, because the team's just going to absorb that, and then you just need to repeat the key messages and learn the key phrases that need to be passed on. So, you only need a superficial understanding.

But I've seen situations and I think it's fairly common, where sometimes an EM ends up in a team without a product manager or a product manager is looking after multiple teams and they don't have all of the time that they can give. And in those cases, that's when the engineering manager is going to have to try and fill that gap somehow themselves and learn more than they would have had to if they'd had a strong product manager there.

But really, I think the two ones that you can't sacrifice are the people and the process. I think those are the ones that EM will live and die by, shall we say? Because they're the ones that tend to be more externally noticeable. So, when the morale of the team or the team isn't performing very well for whatever reason, people will notice that outside your team. And they often notice that if delivery is not going well, so if your processes aren't in a good healthy place or you are having challenges with delivery, it'll be very, very clear to people outside the team that there's a problem.

So, generally, I would say the people and the process are the two that you have to make sure are covered. But again, you can delegate process as well. There might be someone in your team that's really, really passionate about methodologies and may be a natural project manager. Well, get them more involved as well. But ultimately, you're responsible for it, so you have to make sure that those two are working well.

Shane Hastie: The team today is truly the unit of value delivery for our organizations. How do I, as an engineering manager, build great teams?

Building great teams [14:56]

Peter Gillard-Moss: There are two things that have to be in place. I would say the first one is trust. You can't build a great team without trust. And that can take a long time or it can happen by magic sometimes. So, I'm sure a lot of people have heard the kind of storming, norming, forming cycle, which I think is a good heuristic, though not a scientific model. It's a good heuristic. But generally, understanding whether people trust each other, and what does trust really mean?

It means that they can speak openly, they can talk about their challenges, but in a way where there's a level of safety there. So, you haven't got trust when everyone's just direct and annoying each other. That's kind of that norming, storming stage you'll be in. Yes, you've got lots of people who are outspoken, but the other people, they're just upsetting each other when they're doing it. And so, you really need to make sure that the team trusts each other but that there's that psychological safety. So, that's number one. Once you've got a team that trusts each other, they can do a lot more.

The other thing is motivation. So, motivating people in the right direction is what an engineering manager has to do to get a high-performing team. If people aren't motivated or they are motivated towards the wrong things, then that's a real challenge. And sometimes, like if we talk about motivation, there's the classic having a discussion about what language should be used or what framework should be used in a team. And that can take a lot of time and attention. And I think that's where that product aspect or that business acumen aspect comes in. Where the engineering manager has to work with the team to decide whether now is the right time and whether those decisions are the right things to bring the business direction in and get the outcomes that we need, and make sure that the team feel that connection.

So, in those kind of technology debates, the team's motivated towards the right outcomes but also understands the impact of those decisions on those outcomes. And they're not just motivated towards, I would say, selfish concerns. I don't mean that in a ... But we've all been there. We all want to work with that new cool technology. We all want to try that new thing. And that might not be right for the team or right for the problem at hand. And that's the engineering manager's job, to help the team go through that journey and then be comfortable with their decisions and that they're making the right decisions.

Shane Hastie: So, there's a strong facilitation message coming through there.

Facilitation as a leadership competency [17:47]

Peter Gillard-Moss: Yes, definitely. And I think that really depends on where the team is, what style an engineering manager might have to choose. But when you've got to a good, healthy place, you're going to be relying a lot on facilitation.

There's kind of an irony here, right? There's different situations you could be in. So, number one, when you're in a challenging situation. So, I used to remember, I liked watching those kind of survival programs when they abandoned a load of people on an island. And what you often find is at the beginning, the person who is the strong leader is more directive, more decisive, and they get things done very quickly and they get the team up and running and in a good place. But then when the team is comfortable and their food and their shelter is organized, they often usurp this more dictatorial leader and then they get the more facilitator- type leader, the more democratic leader, if you like, in. And I think there's a lesson in that is that democratic style sometimes doesn't work at the beginning because you are in a survival situation, you need to make fast decisions.

I also think there's an irony is that when you build trust, your team are more likely to trust you as a manager to just make decisions on their behalf as well. So, I found that be a really weird thing. I remember having a conversation with an engineering manager and they were asking me what to do in the team, and I was just coaching them through. And they were like, "I trust you. I know what you would do. Can you just make the decision here and I will go with it?" And I was like, "Oh, that's very odd". They were very comfortable trusting me with making a decision.

So, yes, there is a facilitator role, but also, you need to know and need to be able to sense when you don't need to facilitate and for whatever reason, for speed, for urgency, you need to come in and make the decision and say, "I've made this decision. I'm sorry I haven't involved everyone and got their opinions, but you trust me because you trust that I know you. You trust that I'm going to do the right thing for you and the team and the business". So, you have to choose your style based on the situation.

Shane Hastie: And we lean into models like situational leadership.

Peter Gillard-Moss: Yes, exactly. Exactly.

Shane Hastie: One of the things that bothers me when I look at a lot of organizations is how do we train, build, empower these strong technical people to become good engineering managers?

Enabling new engineering managers through peer support and coaching [20:28]

Peter Gillard-Moss: I think the most easiest and the most successful way of doing it is to be surrounded by other good managers. So, that's the easiest way. Unfortunately, that's often not the case. And I don't believe that's any organizational or systemic problem. That's just because lots of organizations, especially over the last few years, grew very, very fast, and so, had a lot of people who were new into these roles and were learning. And that was my experience as well. I was going through these roles and I was learning as I went.

The other thing is coaching helps a lot. So, when you haven't just got a natural set of engineering managers that everyone's copying from, then having coaches who can coach, that helps, but it's not terribly scalable. So, the issue there is you are relying on a few set of individuals. And actually, coaching is a long-term investment, not a short-term investment. So yes, it's a positive, it can work, but it is hard.

There are other things I think work really well. So, having communities of engineering managers that can learn from each other. So, I see, personally, engineering management as a craft like any other craft. So, it's the same as being a techie. If you're in a team of newbie engineers, you will learn off each other just by having conversations about the work. Actually, most of my learning in my career was through my peers.

Talking to them, facing problems together and then going through that learning experience and them sharing material and ideas, like, "I read this thing from the internet. Let's have a read. Let's have a try this out". And that's how I discovered things like TDD and how I discovered much of my engineering practices were from other people. It didn't just suddenly spring from me. So, I think treating engineering management as a craft and encouraging engineering managers to talk to each other, to talk about the challenges they're facing, and see themselves as part of a peer group, that's very important.

The importance of defining the role clearly [22:46]

I will say, the other thing is about defining the role and setting the expectations of the role and making that clear. And actually, that can become really, really detailed and is specific on the organization. I'll give an example. The other day I had a conversation, and it was a real people problem. It was in our engineering management community and one of the questions came up, "I care about delivery and making sure we've got delivery in place, but how should I handle when people are asking for leave and that leave might be clashing, and I'm concerned that if that person takes leave at that point it's going to jeopardize the delivery? What do you do, as other managers, when people raise these requests?"

And it was such a simple kind of people question, but the important thing to come out of that was the discussion and then be able to say, "Hang on a minute, we've had a really, really good discussion. That's something that we should write down for the next manager that's going to come along and have that same problem. And give them some guidance and say, 'When you're in this situation, this is generally what we found worked.'" So, you are clarifying these things. And then, over time, your organization can build a kind of a guide and an idea of what an engineering manager should be responsible for.

The responsibilities of a manager of managers [24:11]

I also think in my responsibilities when I was a VP of engineering and now as a senior engineering manager, one of the things that I believe is the responsibility of the managers of managers is to make the job as easy as possible. I'm a big fan of Bob Sutton and he's got The Friction Project. And I think that is just a fantastic guide for any manager of managers, which is the role of the manager of managers is to remove friction from the job. You want to make the job as easy as possible. And I think if you've got that focus, you want to make engineering management the easiest job it can possibly be, because it's already hard enough, as we talked about, it's already hard. So, what can you do to remove as much friction?

And also, Bob Sutton makes the really good point, "Put friction in where friction should be as well, because you want to use that as guardrails and you want to make specific things hard". So, then building processes where you may want friction. So, I know a lot of organizations, and I've done as well, is put in things like having approvals for architectural decisions when they impact more teams. That's a great mechanism that puts friction in, but is also a process that makes it easier for the engineering manager to facilitate a decision, but puts friction in the right place so that they don't make bad decisions or they get the right input into those decisions.

So, yes, so I think there's a lot of different angles all from coaching, but a lot of it is around putting systems in place to make the job easier and make the expectations clear and make the role well-defined for people, and build those communities.

Shane Hastie: Delve into your crystal ball. You've got 20 years, you've looked backwards, you've seen the evolution of the engineering manager role. Where's it going next?

The future of engineering management [26:08]

Peter Gillard-Moss: I think the way of answering that is looking at how teams are evolving rather than the role itself. So, as I said, when I started my career, I mean, you could be in an organization where you were in, in inverted commas, "a team" of maybe 50 or 100 developers that were all working on one project, and there wasn't really any team structure. You were in a pool of developers, you were told to go and work on something and then you had pools of project managers, pools of BAs, and that's what life was like. But what we've seen is this trend over time to long-lived teams that are leaner and have the least amount of fluff around them as possible. And I think that trend will continue.

I believe that the answer to high productivity is smaller teams with a high ratio of engineers to non-engineering roles. So, I think you are typically seeing teams maybe between four and six engineers, maybe a little bit bigger, but you've got 5, 10, 15 ... You've got Dunbar's number in place. So, a lot of organizations are structuring teams around Dunbar's number. So, that's more smaller teams than we've seen, and leaner. So, I think that will continue.

So, I think what you'll see is engineering managers having more of that whole team responsibility. I also believe that because generally there's a shortage of specialist roles like product managers and designers, you are going to find more and more engineering managers in the position where they might be in a team that doesn't have a PM or doesn't have a designer. Especially if they're on a technical project like a platform project, or they're sharing a PM amongst several teams and that's going to stretch that engineering management role more.

On the plus side, they are smaller teams, so there is less to do. I think what we need to see actually and needs to come out the industry, and I believe we are starting to see signs of it, is actually more of a support and more structures that go around an engineering manager to make their job easier. I do think that that is missing. So, I think making delivery easier, tooling that makes delivery much easier and more predictable, so that their engineering manager doesn't have to be an expert project manager. They're able to manage the work in a way that's fairly easy to do, but they've got the systems and tools in place that give them the data.

I would say that's another evolution that I'm definitely seeing is that engineering managers are becoming incredibly data-led, and I see that escalating. When I first started my career, there were no metrics in engineering apart from things like how big a method was or lines of code and things like that, and they were all very, very poor. Then you've got things like velocity come out and story points, but generally, the engineering managers just disliked those things and argued against them. Whereas now, you're seeing this trend towards engineering embracing metrics like the DORA metrics, the SPACE metrics, and we're seeing even more sophistication coming in, in metrics as well, and being able to use those metrics appropriately for their team. So, I think that's going to be a massive evolution that we'll see.

A lot of the engineering managers I'm interviewing, I'm surprised that over the last year, a lot of them will talk about metrics in the interview from the beginning, which is a massive shift from a few years ago when those just would not be on the table. So, I think there'll be a lot, lot more data-led engineering managers. And I think there's risks there, to be blunt, because metrics are a powerful but dangerous tool as well, if used inappropriately.

Shane Hastie: How big a team realistically can an engineering manager look after? How many people?

Span of control – how many people can an engineering manager lead? [30:27]

Peter Gillard-Moss: I think that depends on the experience of the engineering manager and I try and look at multiple dimensions of the team to understand. A more experienced engineering manager will be able to manage a bigger team. The problem there is a lot of it comes down to how many people you have to coach and how many people you have to mentor or performance manage. So, generally, there's a limit to how many people we can give that amount of care and attention to, which is probably for many engineering managers, when it gets above like six, they will struggle because people management will be taking too much of their time.

But I believe there are structures you can put in place. So, I very commonly use structures where you have a team which has what I often call streams, two streams in them, or two domains inside the team. So, outwardly, they look like one team, but inwardly they're organized where you might have a more senior aspiring manager take over each one and then the engineering manager looks over both of them. So, they're kind of doing a more junior manager job within the team. And that can work as well. But that needs an experienced engineering manager to be able to do that structure. It's no good taking an engineering manager earlier on their career and then saying, "Now you've got to train up two managers".

The other thing you have to consider as well is the technical dimensions. If the area that the team is working in is very, very technical and has a lot of technical complexity, you are going to find that the engineering manager is going to have to step in a lot more into technical decision-making and be responsible for that. So, that will shrink the team naturally, because you can't have a big team tackling gnarly and kind of technical problems and have an active engineering manager at the same time. So, I think you see that a lot in domains where it might be more security based, or even now with the rise of AI where teams are maybe working with machine learning or other things, or data projects as well.

But I generally think Dunbar's number is a really good guide. Actually, as teams go above 10, I think you start to get into problems naturally, to be honest. Unless they've got a really clear domain, the technology's not too gnarly and the people were already in a really good place, yes, you can maybe get above 10 in the team. But I've found most of the times when there's a fire with a team, that team has often gone bigger than 10.

Shane Hastie: Peter, lots of great advice, lots of good content here. If people want to continue the conversation, where can they find you?

Peter Gillard-Moss: I'm active on LinkedIn. So, they can come and search for me on LinkedIn. I'm really easy to find because I'm the only Peter Gillard-Moss in the world. So, if you search for me, you will find me. So yes, that's where I stay most active. So yes, please reach out and connect on LinkedIn.

Shane Hastie: Peter, thank you so much. It's been a pleasure having you on the podcast.

Peter Gillard-Moss: Cheers. Thank you. It's been my pleasure too.

Mentioned:

• QCon Talk: Help: I Didn’t Realize I Was a Manager Now!
• Bob Sutton:  The Friction Project 
• Peter Gillard-Moss on LinkedIn