In this podcast Shane Hastie, Lead Editor for Culture & Methods, spoke to Sarah Wells about the value and importance of improving the Developer Experience, the upcoming QCon London conference and avoiding yak shaving.
Key Takeaways
- Optimizing for speed and flow and removing friction in the developer experience can become a competitive advantage
- Developer enablement does not mean enforcing a toolset or specific approach – it is about building as little as possible of a layer between the vendor and the people using it so that you don't abstract away useful stuff, but you just make it really easy for people to do the things they need
- There will inevitably be some standard approaches and choices (in areas such as security) which will be imposed, but you want to keep these to as few as possible
- As environments have got more complicated and varied the need for Developer Experience has become more and more important
- Paying attention to DevEx is one way of reducing the Yak Shaving work in software development
Subscribe on:
Transcript
Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today, I have the privilege of sitting down across many miles with Sarah Wells. Sarah, welcome, you're a frequent visitor to the InfoQ family. Good to see you again.
Sarah Wells: Lovely to see you too.
Shane Hastie: Now, there's probably some of our audience who are not familiar with you, so let's get the 30 seconds of who's Sarah?
Introductions [01:01]
Sarah Wells: Well, up until the end of last year, I was Tech Director at the Financial Times. I've been at the Financial Times for 11 years and I, first of all, was Tech Director for Operations and Reliability and then from April last year for Engineering Enablement. So that was every team of engineers at the Financial Times who were supporting other engineers to help them do product development.
Shane Hastie: Maybe let's delve into that first, but also we'll talk a little bit about how that's playing out at the upcoming QCon Conference.
Tracks at QCon London [03:09]
Sarah Wells: Yes, we've got a couple of tracks at QCon London that relate to things I've been doing recently that I'm finding really interesting. There's one track hosted by Matthew Skelton who wrote the Team Topologies book with Manual Pais, which is called Optimizing for Speed and Flow, the track title might change, but that's basically what it's about. And it's really looking at the things you can do to try and make sure that you deliver value as quickly as possible. So building the right things and building the right things fast, I think we'll touch on things like topologies, Wardley Mapping, and several other sorts of techniques. It's a people and process-related track. I think it's going to be very interesting. There's another track that I'm not actually involved with particularly, but it's about Developer Experience, which I think is going to be really interesting because it's something that I've seen becoming obvious that you need to have people focused on building, tooling, and providing that kind of paved road for the engineering teams.
I think that over the last five years, the way we build software has changed massively and it lets us do things much more quickly. There's much more flexibility, but it tends to mean you have a much more diverse set of things and you have more of them. And the idea that in microservices, there's a lot of flexibility to have different data stores and use different languages in different parts of your stack, and it's great until you have to maintain and support it and you really need that tooling and that automation and the experts. Another book I really love is Accelerate, which is Nicole Forsgren, Jez Humble, and Gene Kim, it does research into high-performing software development organizations and one of the things I find really interesting is it's about decoupled architectures and fully functional teams. But I don't think you can have every team in an organization understand everything they need to understand.
The need to focus on Developer Experience [03:09]
Sarah Wells: You can't be one team that understands the platform, DNS, coding, accessibility, testing. I think that's a lot to ask. You have to have expert teams that you can go to, to get help with anything. You'll know the basics, you'll know the stuff that normally comes up, but when you hit something that's a little bit different, you want to be able to go to another team that can say, I know how you can do this particular thing. And that's where developer experience comes in. So I'm very excited for that track as well. I think this is definitely an area that I see lots of people starting to write tools for it as well, which I can see that companies are starting to put things together.
Shane Hastie: The Technical Director for Engineering Enablement sounds very much like building that paved road.
Building the golden path for others to follow [03:48]
Sarah Wells: It is actually, yes, totally. So in the first quarter of 2021, at the financial times, we moved some teams around to make sure that every team that built things for engineers all were part of my group. So that meant that things like cybersecurity engineering and cloud platforms that I had not had in my area when I was tech director for operations before reliability moved in. What that allowed us to do is to look at what is that paved road? We talk about it as the golden path, a lot at the Financial Times, but the idea is that you have a path that people don't have to follow, but they should want to follow it because it's easier than hacking through the jungle. You make it something that's really attractive and easy to use and you look where people go off that path because if lots of people are going off that path, maybe actually you haven't made it as easy as you thought.
Not everything is optional [04:39]
Sarah Wells: So there's obviously something things that you can't let people go and just choose their own approach. There are always some considerations around security or cost or just complexity or operational concerns. So for example, I wouldn't really like to have people using different log aggregation tools in the same company, because I'd like to be able to follow a request across all teams and not have to jump out into different tooling. There might be cases where that's really good, but maybe for a local improvement, you are losing something globally. So I think engineering enablement is about, can you pave that road? And can you also try and argue the case for something should be a little more standardized? That's what I've been doing for this last year. And there's a lot to it. There are a lot of things there that are really interesting because you're always trying to balance what you're trying to achieve. Product development teams want to build that product.
Shane Hastie: Let's explore some of those balances. And when it starts to go even outside of the development teams, the development environment, there are business trade-offs that we have to make. How do you pave that road?
Balancing multiple factors [05:41]
Sarah Wells: Yeah. So one thing that was interesting to me as I started to get more involved with the relationships with our cloud vendors, with the cloud platform, is the realization that people tend to assume that a team in charge of that is mostly focused on costs and cost control. But I think there's always a balance. There's always a balance between do we want to spend more money on our platform because we are building more stuff and we are selling more stuff? So an example for this would be when the pandemic started, the Financial Times made a lot of their coronavirus coverage free, and it is great coverage. I mean the infographics on this are exceptional and we got a lot of people to come to read it. So that means there's a lot more traffic to the website. And the interesting conversation that led to was well, we end up paying more for our CDN.
We end up paying more for our log aggregation. We have lots of things where there is a cost to serve a customer. So you have to have a conversation with parts of the business outside of a product and technology. Say the finance department, you need to explain how cloud platforms work I think. IT is quite often looked at as a cost center. You have to make the case that it's not a cost center. You want to spend more money if you're building products that people will then want to subscribe to. So I think there are some interesting things there. It's a credit, I think to our engineering team, that our editorial team didn't really worry about making lots of things free to you. When I first joined the FT, I think we talked a lot about, can the site take this traffic if a big thing happens? We've got so much more flexibility and scalability. The editorial doesn't really think about that in that way anymore, but there are always consequences of your editorial decisions.
Shane Hastie: Yeah. And just on a personal note, that FT content was so valuable particularly in the early days of the pandemic.
Sarah Wells: Yeah. I think the graph-type visualization really tells a story incredibly effectively.
Shane Hastie: It really does. What does it take to bring the,I'm going to be a little bit stereotypical, but the loose cannon developers who really are incredibly productive and creative and they really want to do great work. How do you do that balance and sell this golden pathway and this paved road?
Pioneers, settlers and town planners (reference Simon Wardley) [07:52]
Sarah Wells: I think it's interesting. You have some developers who want to do everything themselves. They're very able to turn the house into a lot of things. I've heard Simon Wardley talk about pioneers, settlers, and town planners, the pioneers, the people are out there building something new and they might be quite hacked together and held together with sticking tape. And then the settlers come in and they kind of make it a bit more solid and town planners is like you're continuing to do work within something that's very regimented. I think that probably five years ago, we had a lot of people doing pioneering work at the FT, and as people came in to make it more solid, the settlers came in, actually, there were lots of things they didn't want to do. They would that you find the things that actually aren't interesting and valuable. Lots of developers want to focus their efforts on stuff that solves problems for the business.
So they probably don't want to have to maintain their own cloud platform or ship logs somewhere different because that's a local movement that you've done. So I think you find the things that are pretty common and that not everybody wants to know about, but you have to make the case that there are some things where you lose out if it's a completely free choice for everything. And so what we talked about a lot at the FT is building a golden path or a paved road. So the idea is you pick a set of tools and approaches and you make them easy to use. And that's typically building as little as possible of a layer between the vendor and people using it so that you don't abstract away useful stuff, but you just make it really easy for people to do the things they need.
A lot of the FT is built on AWS. You put some tooling around some of the AWS stuff so that people can easily deploy resources. So you pave out the road, you document it really well because you want your product development engineers to be able to do 80% of standard stuff without talking to you, they're just going to read the document, use your tools, do it. They can move fast and they're not held up. And then you make sure that if they get stuck on something, you are very responsive and you are solving their problems for them. And if it starts to come up a lot, you start adding that to the paper. And the idea is that people can go off-road, but it should be much easier to go on the road than to hack through the jungle. If people are starting to hack through the jungle, then maybe you haven't solved a problem the way that they need it so you need to be responsive to your customers.
You are building a product and your customers are engineers [10:02]
Sarah Wells: So it's kind of engineering enablement. You have to understand that you are also building a product and your customers are engineers. So you use the same tools to talk to your customers and work out where their problems are and how you might solve them. And you have the problems of people telling you this would solve their problem, but actually you need them to step back and tell you what the actual problem is. So I think there's a lot of persuasions. You want to try and avoid too much mandating because engineers don't tend to respond well to being told that you must do this, but you can obviously explain to people that it's too costly for us to have multiple of these things in play.
Shane Hastie: How does this grow out to that broader developer experience that we touched on earlier?
Sarah Wells: I think if you want engineers to use the things you build, you need to be thinking about how you make those into an enjoyable experience. You want people to come to you and say, this thing is so useful I can't imagine not having it. When I saw that happening at the Financial Times, there's a really lovely culture there where people will say, I love this tool and it's so great. And if you're in a slack channel and someone says, this is useful to me multiple times a week, then you go, hey, we've hit something there. And then there's the flip side of this is annoying. You need to be responsive and say, okay, why is it annoying? But I do think that developer experience is really important because the key to actually building good stuff is that your developers are able to focus on the business problem and not be held up by everything else.
There's something really frustrating about I just want to make this one change and I've got to do all these other things to get there. That yak shaving thing, like if you Google yak shaving, it's great. It's like the idea that you need to do this thing, but to do that, you do something else and you need to have something else. And eventually, you find yourself shaving a yak to get the fur stuff or cushions. I recognize it. And if you can take away some of those steps and allow people to keep the focus on the thing they're actually trying to achieve, you are going to make their life easier, happier, and more productive.
Shane Hastie: I want to be skeptical and say, but if this is common sense, shouldn't we have done this for years. Why is DevEx suddenly a thing? If we look at it on InfoQ, it is something that we are saying is on the early adopter. And it's over on the left of our technology adoption curve that we see. Why has it taken us so long?
Why DevEx is important now[12:21]
Sarah Wells: It's a good question. I don't know whether it's a recognition of the benefits of actually having your engineers focused on the valuable stuff. Maybe it's really also to all the other things we've introduced. So when you introduce microservices and you have DevOps, things get more complicated. So when I was a developer, I was writing Java and I was committing it on source control and then it was being built into an artifact and someone else was deploying it. So I didn't have to know a huge amount. And we had DBAs and testers and things. And so by the time I left the FT, mostly there aren't DBAs, the developers are finding the right data stores and you're probably using managed databases from somewhere, but you're having to do lots of different things that you didn't have to do. You're setting up the infrastructure, your writing, maybe deployment pipelines.
So things are more complicated. I think maybe the other side of it is like the idea that DevOps made it less of a confrontation between developers and operations. The idea is that operations want to keep things up. So in their interest, never to change anything in the production environment and developers want to release code and they don't really feel the pain if it goes wrong. I think it's the same thing potentially with platform and tooling teams, you've got your aims and maybe you are a security team and you're thinking we'll be the ones that have to explain if something goes wrong in terms of security. But if you get to the point where you think you build it, you run it, and you are making decisions, you are therefore responsible for those decisions, then your role in that team becomes different.
The role in a security team becomes we make sure that you understand the context of your decisions and we guide you into the right paths, make it hard for you to do something that's risky, it doesn't necessarily stop you, but we should be able to identify it if it happens and we should be able to help you avoid it. So I think some of that is that we expect far more of our team. So, therefore, in the developer experience, there's a gap there to make that easier for people.
Shane Hastie: For the technical leader who's listening to this and saying, yeah, this is all great and FT is huge. You've got lots of money, lots of people to do this with, but I'm a much smaller organization or I'm a bank or I'm in one of these heavily regulated tightly controlled environments. How do I do this?
Advice for team members [14:35]
Sarah Wells: I think the FT is pretty small, it's not a big development organization. It's a couple of hundred engineers. So banks are much bigger. I mean, there's always a difference between different industries and domains. So we had a conversation at our internal tech conference a few years ago at the FT where we said we're not a hospital or a power station, no one dies if we release the front page and part of the story is blocked by a box or something. So there are different levels of concern, but if you take the approach of, we're trying to build things so that they're easy to use and they guide people in the right way, that guidance might be a little stronger in some industries, but you want it so that it's self-service so that people aren't held up so that it's automated so that you can build automated pipelines that have the necessary auditing in place so that you can do continuous delivery in banks. That's been proven. So basically, you just consider what's the important stuff to ask and how can we make it easy for people to comply and thrive in those environments?
Shane Hastie: We're at the beginning of 2022, let's look forward, where's this going? What's next?
Sarah Wells: I wonder whether we're in a kind of Cambrian explosion. It's like everything's got very complicated. It feels like there's a scope for things to get simpler again. One of the reasons I think that Kubernetes has done well is that actually, you get the weight of people behind something, and there start to be lots of things built on top of it. It's easier, you can like hire people who've got experience, you can use tools that fit with that. I think Kubernetes is quite a complicated thing to run. I think it's great for certain size organizations with a certain complexity. Maybe we'll get a little bit to a point where some of the cloud platform offerings are just very straightforward and I think there's lots of stuff there already. So I feel like there's still always going to be a role within an organization, a group of people who just oil the wheels.
So making sure that they have a relationship with the vendors, they're aware of what's new and what's available and they're just packaging it in a way that suits that organization. So I think there's that maybe it will get a little bit less complicated, but I think that the developer experience thing is still really going to be important. And I think I can see tools that were open-source alternatives of starting to see vendors talking about some areas in that space. So it feels like it's becoming a more mainstream thing. It's moving up that technology adoption curve. So lots of things that at the FT we ended up building.
So when we started doing microservices, we discovered it was really difficult to keep track of all the services that we had, because we had hundreds and hundreds of services. So we built a thing that we call Biz Ops that lets us track all of those services and which teams own them and what do they do and how do we troubleshoot them and where are they monitored and things like that. And you look at some things that companies are starting to build that they're in that area. So we had to kind of build it and I think now you would probably buy if you could.
Shane Hastie: And advice for leaders, people, maybe the newly minted leader, you've been through that and got to that position of technical director. But somebody earlier in their career who are moving perhaps into a team leadership or that sort of a role, some sage advice.
Advice for technical leaders [17:55]
Sarah Wells: One of the first things that I say to anyone that is becoming a tech lead for the first time is your job is not to code anymore. Because often you are a really good developer and then you become a tech lead and you don't want to give up the coding and you shouldn't give up the coding, but you need to make sure that you are not on the critical path for anything because you will get pulled aside. You'll be dealing with something, you'll have a meeting, there'll be an emergency, something like that, and if you are the person who is building the core thing that you are delivering this week, that is a problem. So what do you end up doing? Well, I ended up doing when I first was a tech lead was you find the little annoying things around the edges. So you end up building team tooling and things that there's not a deadline, but it's quite nice to tidy up and improve.
So that's a good place to keep your tech skills there, but you also just need to be aware that's not your job anymore. Your job is to coach and mentor other people and to have a view of what your tech estate is and what makes sense to do in that. And I think the second thing I would say is your life is better if you spend time communicating to people outside of engineering about what you're trying to do and think about why you are doing it, not just for engineers. So for example, you've always got a number of things you do for what people generally say, tech debt, there's always some level of tech debt, but you have to kind of explain it in terms of why do we need to actually do this? So it could be, we need to upgrade the database because you need to upgrade databases.
And if we don't do it, there's a security risk or we are getting really slow at delivering value because this area is very complicated and we didn't get it right. So helping to explain to the people around you that it is natural to go back and redo things, but having an ability to recognize that the business doesn't really care what database you're using, probably. So they just hear you saying names and words and so basically try and have empathy with the people that you work with to explain what your team is doing.
Because I think your job as a leader is to make it easy for your team to be able to do the right things and some of that is about selling your team and selling what they're doing so that you can keep going. So I think there's a natural distrust of having to sell things as an engineer. I certainly had it, but if you ask people they're generally proud about the stuff that they do. So you have to speak up about it because that is what gives you the space to do the next thing you want to do.
Shane Hastie: One of the things that I've heard more than once is that when you move out of the individual contributor role into those leadership roles, and in my own experience, this has definitely been the case, there is value in actually starting to share your knowledge. What do you feel about that?
Sarah Wells: Absolutely. I think that it's important in general, it's important for your team that you can share what you know. It's also something where if you can start to write about the things that you know or speak about them, you really help put all of that together into a coherent train of thought. I started speaking when I was a tech lead and it's a lot of work to put together a talk, but after you've done it, you kind of have something to draw on that's your opinion on this subject. And I think the ability to put together a story or an argument about something is a really good skill. It's a great skill as a leader, the ability to tell a story that can persuade people into a particular direction. So I honestly think that a leader's job is generally to say, have you spoken to so and so about this?
And just keep saying that repeatedly because talking to other people is a good thing, but also just can you put together one page of why this is important? And you quite often get pushback from people, but I think if you're really keen on doing something and the idea of writing one page to explain it is putting you off, I'm not convinced that you've really thought it through. We used to do 10% time when I was in the content platform at the Financial Times. So one day on a fortnight, you could if you wanted to go and spend a day doing whatever you liked that would help your team or the FT and explore something in technology. So the only things that we said were, you need to tell us in advance what you're going to do and you need to present back the day afterward, just for five minutes. And the reason we did that was that I think it can be really easy for people to just start putting on a thread and just not quite do anything concrete.
It's much better if you can say, well, I am going to try and install this software and it was absolutely fine for the next day to say, I just didn't get anywhere because it was actually really difficult to use. And there were a couple of things that were really great about this. One was that people got used to just presenting back the things that they'd been doing. Another was that if you got into a discussion with people about a particular thing in technology in your team, and they were something like I think we should try this thing.
You could say, well, do you want to try it out in 10% time? And sometimes they'd try it out and it would be great and you'd adopt it and often they'd try it out and they would say after a day of investigating, yeah, it's not as good as I thought it was. And both those ways, you've got a lot of value and you haven't spent weeks arguing about it. But so I think there's something really good about trying to put together a case, a very lightweight case for doing something. It makes you put your thoughts in order.
Shane Hastie: And spreading even further, you are a pretty regular speaker now, what does it take to get there?
Becoming a conference speaker [23:21]
Sarah Wells: Oh, well, I think in the first case, my boss suggested I put in a proposal for a conference and I thought, well, I'll put in a proposal, I'm not expecting to be accepted. And I've encouraged a couple of people that work with me to do something similar. If you want to just start speaking, setting yourself the target of I am going to submit to a conference, doesn't feel as scary as I am going to talk. And if you get accepted, then you worry about the next thing. Then after that, I said yes a lot initially. So I said yes to repeating my talk that I think I submitted the same talk to a second conference very soon after because I thought if the first one goes really badly, I'm going to have to do it a second time. So there's a chance that I'll fix that up.
And there was so much rewarding about it. I felt like I really understood the topic after I'd written a talk about it. And you find people will come up and talk to you about it at the conference after you've spoken and you learn a huge amount from that. And then you meet other speakers and find a lot from them and the network of people that I had met for example, when I started being a tech director, I could often find someone that I knew in another company and say, you used this technology, how does that work for you? And people are quite willing generally to say, this is our experience and to share stuff. So I found that network incredibly valuable in helping me to make progress more quickly than if we tried to solve everything from scratch.
A trivial example would be, it's not trivial, but one example we were thinking about trying to manage our operational incidents in Slack. And we were speaking about, oh, maybe we should hire a Slack bot for that. And I knew that Monzo had a Slack bot and they'd open source it. And we just decided we would just use this and we spoke to the people at Monzo a little bit about it and then we just installed it and we figured we could fork it and make change as we discovered things that were different for us, it just got us going really quickly and we knew it had worked for them. And we knew about it because we had that connection with people at that company.
And they've since spun that up into an instant.io, but we're still using the fork of their open-source version. Yeah. It's really interesting that network. And I think the confidence level as well, my confidence increased a lot after I started speaking. I used to often feel like I was talking about things and I didn't feel I really knew about them. But once you kind of stood on stage and spoke about microservices or talked about DevOps, you have to admit to yourself that you know something about it.
Shane Hastie: Thank you so much. So if people would like to continue the conversation, where do they find you?
Sarah Wells: They find me on Twitter. Sarahjwells on Twitter is probably the best thing. I mean, I am on LinkedIn, but I really don't check it very often. And I'm taking a few months off at the moment so I may not be as responsive as the times. I'm just taking a bit of a break, but yeah, Twitter is the best way.
Shane Hastie: Sarah, thank you so much for taking the time to talk to us today.
Sarah Wells: Brilliant. Thank you very much.
