Continuous Delivery, Psychological Safety and Inclusive Leadership in Software Engineering

Transcript

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I'm sitting down across many miles with Hannah Foxwell. Hannah, welcome. Thanks for taking the time to talk to us today.

Hannah Foxwell: Hello. Thank you for having me.

Shane Hastie: We met because you gave a really interesting talk at a recent QCon, the Dangerous Dichotomies of People Management. But before we get into that, who's Hannah?

Introductions [00:34]

Hannah Foxwell: I've been really active in the DevOps community here in the UK for over a decade now. So I refer to myself as a bit of a DevOps elder, but the reason that I became fascinated with DevOps and continuous delivery and that type of engineering practice that was only just emerging a decade ago, even though it feels very commonplace now, is because of the positive impact it had on the humans involved in that endeavor. So I've really been focused for the last 10 years of my career on seeking to identify the engineering practices that make life better for the people involved in building software.

And that's taken me to a number of places. I've been the director of platform engineering at Pivotal and at VMware, and most recently I moved across into product trying to take a little bit of that customer empathy with me into the product sphere where I've been leading Snyk Container.

Shane Hastie: The engineering practices that make life better. That's intriguing. Aren't engineering practices about making software better?

Engineering practices that make life better for the people involved in building software [01:49]

Hannah Foxwell: Well, yes, they do that too. They do that too. So I'll sort of tell you a little bit about my origin story, which I've told many, many times because it was my grounding in software development. But earlier in my career I was a release manager and I was the human interface between the developers and ops team. And to deliver a new release of an e-commerce website into production was many, many months of effort and planning and coordination across a lot of teams. We did releases about once a quarter. We did about four a year. That was normal at the time. It's not normal now, but it inevitably all went horribly wrong.

And so I was the face of failed changes and I could not understand why we couldn't sort of plan and itemize our way out of risk in that context. But also because of the complexity of delivering a large amount of change in a single big bang, the pressure it put on our colleagues to meet business deadlines but also to do so safely and not impact our customers was incredible. I was suffering myself because I was pushing to get these deadlines met. That was my objective. But in order to do that, we were working on bugs and triaging bugs over the weekends. We were working late nights.

The test team and the ops team were always the last people to know about deferred release or a change of plan, and I just thought there must be a better way. And so that's what I've been doing. I've been looking for those better ways, reducing risk by actually delivering more frequent changes, but smaller ones, continuous integration, continuous delivery, automation. Also, things like test-driven development and pair programming are so underrated in terms of just building quality in and shifting all of that risk left in the process. So I'm a huge advocate for anything that makes the process of delivering software all the way to the hands of your users in production, a less stressful and more delightful experience.

Shane Hastie: Let's dig into some of those technical practices. How do we make them, I want to say safer, easier. I'm an advocate of pair programming. I'm a strong proponent of TDD. I've seen why those and other XP practices just do work, but I also see a lot of reluctance.

Hannah Foxwell: With pair program, it always feels counterintuitive to a lot of leaders to put two people, or sometimes if you're doing ensemble programming more than two people on a single task. It's like, "Well, surely we can parallelize and get more done." But it's a false economy. What you end up doing is you end up moving those sorts of interactions and those quality gates to maybe a PR review. That's very common practice at the moment, but that person who's doing the PR review doesn't have the context, hasn't followed through the thought process of how that software was being built and can end up creating a bottleneck. I hate to say it, I've seen PR reviews create bottlenecks in organizations needlessly.

So I think it feels counterintuitive to do less work in progress, but actually a lot of organizations find that it produces higher quality software and the team work far more efficiently. It also creates, I think, a really great collaborative environment for learning. I know of managers who've bought in interns and junior engineers, and that's how they have learned. That's how they have learned new technology. That's how they have learned to work in professional environments, and it's because they were in an environment where pair programming was normal and expected and they always had somebody to talk through their ideas with.

There was no pause, there was no hesitation. It was expected, and you learn so quickly if you got an expert by your side, don't you? And so there's efficiencies there as well, which I think are sometimes underrated. Like how you can level up your whole team and how you can really scale the impact of your more senior experienced people through these things. And spending time with each other breaks down those barriers as well. You're going to have a lot more psychological safety in your team if there's more dialogue and if you know each other as human beings rather than communicating solely over call requests and Slack channels.

Shane Hastie: The stereotype though is that that's how technologists prefer to communicate.

Find the healthy balance between solo- and pair-programming [06:54]

Hannah Foxwell: Yes. Isn't it? I think it can be quite exhausting for an introvert to spend all day in a one-on-one dialogue. I do think there is a healthy balance to be found between soloing and pairing. If you know what you're doing and you feel confident in what you're doing, then soloing is fine. If you are exploring or experimenting or iterating, I think having more people on a task is very valuable. I say all this as a huge advocate for flexible working hours as well. So am I going to go ahead and contradict myself immediately because the flexibility that a lot of asynchronous companies have, it requires developers to be incredibly independent.

You have to be cool with those bottlenecks and things like that, but I do think flexibility in working hours and distributed teams are really emerging pattern for success as well. I'm very curious actually. I'm very curious if anyone has been able to bring in some of the benefits of XP and pair programming into that very asynchronous or remote culture. I haven't worked anywhere that managed to do both things.

Shane Hastie: I have certainly seen remote pairing work.

Hannah Foxwell: Absolutely.

Shane Hastie: But the asynchronous side, as you say, there's a lot of evidence that it seems to be working and definitely all of the exploration studies quality is not suffering from asynchronous work. I do wonder how collaboration is going.

The impact of large language models [08:32]

Hannah Foxwell: Yes, yes. Now we have a new evolution at our doorstep, which is the impact that GPT models and the large language models are having on software development because you are interacting and you're learning from the AI rather than seeking that from your colleagues. The AI isn't always right as well, but your dialogue is somewhat less human now and you may not need to reach out to your colleagues as frequently. And what does that do to shared understanding and collaboration? All of those things you sort of take for granted in a team environment.

It's really early days to figure out how these Gen AI auto code complete and co-pilot type tools are going to impact collaboration, are going to shift how we do software development. They're incredible tools. They're so, so powerful, but they've changed the game completely.

Shane Hastie: So in that changing game, how do we keep the human stuff running smoothly? The psychological safety is more than just a buzzword, but has almost become one.

Ensuring and protecting psychological safety [09:43]

Hannah Foxwell: You still need to have some kind of human interaction. I haven't seen any team with very high psychological safety that didn't have very strong interpersonal bonds. They don't have to be a complete network. It doesn't have to be to everyone thing, but that support network is really powerful. I think one of the things that really helped in one of my previous teams was when new joiners come into the team, they're given a buddy, a person, not their manager. Their manager's got a whole load of other duties that they have to do to help them on board, but their buddy was the person who they could go to with their ridiculously stupid questions.

Things that they would feel uncomfortable asking anyone else and I think that's sets a very strong precedent for in this place. We can ask any question and your body would almost indoctrinate you into the culture of like, "Well, I don't know the answer to that. So shall we throw it in the team Slack channel and see if anyone knows the answer or shall we ping this other team over here? They might know." I think that's a really powerful kind of learned behavior. And I think the buddying system really baked that in from day one for new people joining the team. There are no stupid questions here. If something goes wrong, you can ask for help.

And we talked about pair programming, but that's a really good way of working through a problem. It's like if someone flags to their team usually through a Slack channel or something like, "I'm really struggling with this." The culture was we help each other. It wasn't like my task is more important than yours. It was, "Oh, here's a person that needs help. I'm going to jump on a Zoom call and work through it with them and see if we can break this down." So I think that's certainly helps with psychological safety. Psychological safety, it's about failures will not be punished, they'll not be blamed.

SRE practices and error budgets [11:33]

And I think if you are in a team which has got on-call responsibilities. If you are looking after your users in production, that can be an added dimension to it as well. And so I really strongly advocate for SRE practices as identifying an SLO, your service level objective. And it will never, ever, ever be 100%. And so when you use that language of error budgets in the SRE space and in the Ops space, what you're doing is you're not catastrophizing every single incident that happens in production, you have an error budget. Failures are expected here.

And when we have failures at a rate that is intolerable to our customers, we have a commitment that we will do something about it. We don't just have to accept that every day is a firefight to keep the lights on. Again, it's technical practices that build more psychological safety and a shared language about failure in your team, which means that people feel safer to own up if something has gone wrong. People feel safer to take risks, people feel safer to share ideas. It all sort of builds from that place where you feel safe and you don't have fear in your workplace.

Shane Hastie: What's the role of the manager or team leader in that?

The role of the manager in visibility and enabling safety [12:58]

Hannah Foxwell: Visibility is for sure one thing. I think these SRE practices work really well when they are very visible, whether that be a synchronous call where you look at how you're doing against your SLOs and how you're doing against your error budget or whether it's a weekly post into a team Slack channel, recognising that we are doing good. Sometimes if there are no incidents and nothing's on fire, it can be a sort of neglected topic. I always talk about security in that context as well. It's like the absence of bad things is actually what success feels like.

It's like the absence of bad things and you can easily forget to recognize that, like we've gone through a week and we haven't had a single incident like, "Hell yes, well done team. This is an amazing achievement. Let's keep it like that." So I think visibility and recognition when things are going well, and I think having a routine and making it a little bit mundane when things do go wrong can be helpful as well. So as a leader, you can really set the tone for how those meetings and incidents run. Was it a catastrophe or was it just one of those things that happens? And it's like, "Are we going to seek blame or are we going to seek learning and improvement in those contexts?"

And I think absolutely managers have a huge role to play in setting the tone around incidents and when bad things happen and that can help the team navigate them with more clarity actually. If you're feeling threatened, you're not going to have clarity of thoughts. If you are feeling panicked, you need to be calm, you need to feel supported and it needs to feel like routine. Another engineering practice would be, I've said SLIs, SLOs, but actually doing fire drills to get the muscle memory there across everybody in the team who's holding the pager, just do fire drills so you know exactly what you're going to do.

If the pager goes off at 3 A.M. in the morning and you're not at your best, that's a really powerful way to make those things that can feel catastrophic, feel kind of mundane and normal.

Shane Hastie: Something that happens a lot and seems to be happening quite a lot at the moment is organizations merging mergers and acquisitions and inevitably there is a, I want to say a culture clash or a slight disconnect or maybe not a slight disconnect. With organizations who are going through that, how do managers, leaders in the technical space and team members, individual contributors, how do they make sense of this new world?

The impact of mergers and culture clashes [15:44]

Hannah Foxwell: It is something that I've been through and it's incredibly hard. I was part of a company called Pivotal that was acquired by VMware. VMware, we're a much, much larger company. I'm not going to pass judgment. I'm not going to say which culture was right or wrong. They were just very different. They were just very different. And some of the anxiety I felt from my team was, "Oh, we're going to lose the thing that makes our team special. It's going to be lost in this transition. We're going to be forced to do things a different way."

The thought process I went through and I shared it with everyone is, my knee-jerk reaction is that this is going to be hard and I feel like I'm losing something special and I want to run away. And I said, "The only way to guarantee that we all lose this team spirit and this culture that we've built is if we all run away and we all go and do something else because of this fear we have that we will lose the thing that we value about it. And so I said that the only way that I'm guaranteed to lose this is if I walk away. If we stay, nobody can change our culture. No one.

If you're looking at an organization of over forty thousand people that VMware was at that time, it's not a homogenous culture. It's not one culture across the whole entire place. Teams have their own culture and even when we were part of a smaller company and we were part of Pivotal, we had our own unique culture. And I think when people go through that transition, there is a lot of anxiety and there is a lot of fear and there will be change. No doubt there will be change. But the thing that is culture that is about the humans and that is about the humans making a choice about what elements of their culture they value and that they want to maintain and sustain.

And so that's what we did. And even after I moved on from VMware, I truly think the team culture is still the same in that amazing little team I used to manage.

Shane Hastie: Managing teams, a fair number of our audience are new to this. They were great individual contributors. We've taken the best technical practitioner and put them into a leadership position. That's quite a big transition.

The hard transition from individual contributor to team leadership [18:15]

Hannah Foxwell: Yes, it is. It's one of the hardest in my opinion. I really enjoy mentoring people who are making that transition. One of the reasons that I feel like I really focus on it, and one of the reasons that I want to create those opportunities for people to step into management is because I love being a manager for one. But I also found those first couple of years of management really, really hard, really hard. So the temptation, because you are usually managing people who are doing the job that you used to do, is to impose your way of doing the job on them.

Another temptation is to focus solely on your team or on delivery and to see that as your absolute top priority, when actually the job of a manager is much, much broader than that. And so I think it's very easy during those first few years of management to overemphasize the delivery management aspect of your role and to really focus on the work that you yourself understand because you were doing it before. Now when I'm mentoring a new manager, I try to break it down in terms of your three teams, your three teams that you are now a part of. That is the team that you manage and that's sort of easy in a way.

That's where you are familiar. That's where you're in your comfort zone. So the team that you manage, you need to look after them and their well-being. But you've taken a step-up in terms of the hierarchy and of a leadership team that you are a part of and you have influence in that leadership team. And you have peers who are likely going through similar challenges to you. You have a manager of managers who's got an even broader perspective and might be looking for your insights to help drive a strategy. And these are all new things to you. But what you need to do, first of all, is you need to establish that leadership team.

You need to make sure that it's a collaborative and supportive leadership team because you will be better together. When communication flows simply up and down the hierarchy and we don't have that peer-to-peer communication, we miss out on so many huge opportunities to be better together. And so I say that's your second team. Think about your leadership team. How do you build cohesion there? How do you collaborate more closely with those peers? And then your third team, again, it's a peer group, but it's a peer group outside of your domain. So if you're an engineering manager, these are your peers in product, these are your peers in design.

Maybe you have peers in infrastructure. And even if you reach out into the organization, you can be looking at having peers in marketing or in customer support, people really, really close to the customers and the strategy. And how do you bring those people together to solve the bigger problems, the problems that extend well beyond engineering? And I think that third team is the one that people find most difficult to navigate when they step up from an individual contributor. They probably have a strong network in engineering. They probably don't have a strong network across the business.

And so cultivating that network is so important because we don't deliver value to our customers in isolation. It's a cross-functional team. And so by thinking of it in those three and investing time appropriately, instead of just going laser focused on your team, the people who report to you. How can you broaden your horizons to deliver more impact with the team that you have? So that's one piece of advice I always give to new managers. There's a hell of a lot more to management than just that as well. I could talk all day. What should we talk about next?

Shane Hastie: Let's dig into inclusion, equity, equality, belonging. As an industry, we don't have a great reputation. We haven't done a particularly good job. How do we get better?

Getting better at inclusion, equity, equality, belonging [22:27]

Hannah Foxwell: It is something I care very deeply about. And again, for a new manager, I think it's easy to believe that fairness and equal opportunities come as a result of treating everybody the same. And that's simply not true. Equality is treating everybody the same, and equity is making sure everyone has an equal chance of success. And success, again, it means a different thing to every person. The ambitions of one person, their strengths, their weaknesses, they're all going to be different. So understanding where your team is at at the moment, each member of them individually and where they want to get to and building a shared plan to get them there is so important.

I believe that everyone in my team should have an equal opportunity to succeed, but I also know that that is going to look very different for each individual. And so it does take time to build that trust, build that relationship, and really hone in on what that next move is to get that person on the path they want to be on. And so I think there are elements in that where you do come across systemic inequality and biases. I read a report, I think it was it in the Harvard Business Review, which said that one of the reasons that women don't rise through the organizations as quickly as men was a lack of sponsorship.

And so with new managers, I always tried to help them understand that the way you manage someone is going to be different at different points in their career. Sometimes someone will need a mentor, they will need you to teach them the things that you know. And other times they will need a coach and they will need to think for themselves and they will need to find solutions on their own. But then third, and really importantly, sponsorship, which is to create opportunities for someone by allowing them to do something they haven't done before. By taking a risk and supporting them to do something new, to take on a leadership role or to take the lead on a particularly complex project or to step up into their first management role.

Sponsorship is what happens when you say, "I believe this person can do it even though they haven't done it before." And there was one study that indicated that one of the reasons that women don't progress is through a lack of sponsorship. Women are over-mentored. Their managers were sharing their wealth of knowledge with these people, but they weren't creating opportunities to demonstrate that they had these skills and they had this ability. And that's something to be really mindful of as a manager, because hopefully you have a diverse team. Members of your team may have encountered these biases and may not have had equal opportunities with previous teams and previous employers.

And it's a challenge for you then to figure out what that right next step is for that person, given that context. Perhaps this person is absolutely already ready for a step-up and you mentoring them or coaching them is absolutely the wrong thing to do. Maybe their previous manager wasn't sponsoring them the way that they should have done, and maybe that's what you need to do next. So that's advice that I would give to new managers, which is it's not always about sharing what you know.

It can just be about taking a leap of faith and trusting that a person can do a thing that they've never done before because that's how careers are built. It's built through sponsorship, it's built through someone taking that step and taking that risk.

Shane Hastie: A lot of solid advice, good content, lots to think about here, Hannah. If people want to continue the conversation, where do they find you?

Hannah Foxwell: Luckily, I have quite an unusual name. So if you Google Hannah Foxwell, you will find me on the internet. So you'll find me very easily on LinkedIn, Twitter. I do have a Mastodon account, but I have to admit, I do not monitor that one. And I have tried to engage on BlueSky, but that one isn't really working out either. So honestly, my primary places on the internet at the moment remain, I said Twitter, it's not even called that anymore, is it? It's called X. I will never reprogram my brain to call it X.

I'm sorry. So those are the places you can find me. You can also have a look on YouTube because I've delivered many talks in a similar vein over the years. What are the practices that make life better for the humans in software development? So there's plenty more of my ramblings on the internet.

Shane Hastie: And QCon as well.

Hannah Foxwell: And absolutely, absolutely. My talk at QCon, I'm really looking forward to when that's shared more broadly and publicly. I think at the moment you can only access it if you've got a virtual password of the conference.

Shane Hastie: Hannah, thank you so much for taking the time to talk to us today.

Hannah Foxwell: This was fun. Thank you for having me.

Mentioned:

• Hannah Foxwell on LinkedIn
• QCon London