Transcript
Shoup: Maybe we can briefly introduce everybody, and what you do.
Narayanan: I'm Sangeeta Narayanan. A director of engineering at Netflix. I lead an org focused on infrastructure and productivity and enablement tooling, targeted towards content creation use cases. I've been at Netflix for over 11 years now. Prior to that, been at a few startups.
Wells: I'm Sarah Wells. I work at the Financial Times. I am a technical director for an organization called Engineering Enablement. We've just set that up. It is things like platform teams, enabling teams, operational concerns, basically.
Caldwell: I'm Nick Caldwell, VP of Engineering for the consumer team at Twitter. The consumer team is all the fun parts of Twitter, so on timelines, the content creation stuff, and all the health systems. Also, on the board of a really great company called HubSpot. I've been on the board since January. Previously was VP of Engineering at Reddit, Chief Product Officer at Looker, and at Microsoft for a long time.
Helping Platform Teams Foster Customer Centricity, and Empathy
Shoup: The first one is, how to help platform teams foster a sense of customer centricity, customer focus and empathy for the teams. The particular thing we noted as we were preparing is, a lot of times the team that build a particular piece of platform-y software, don't use it as much as you might imagine.
Narayanan: This one's something we run into all the time. I look at it in a couple different ways. One is, as you pointed out, do you use what you build? The other is, how connected are you to the business and the end use case? If you think of a product, you have product managers who play that role. How do you do that in a platform or with smaller companies who likely don't? When I think of tactics, a couple that I feel have worked well, that I've seen work well is embedding in this virtual squad model, go work with the product teams, see what it's like, and that really helps build empathy. Our compute team does a great job of this, they have a liaison model. As an engineer, you have different customers, and one person is dedicated to understanding the use cases and business requirements for a particular group, for instance. That way, you have the continuity, and you have an internal evangelist in your infra team. A couple things that I've seen work.
Wells: All of those are great things. The things we've done additionally to that is, we actually look to recruit from our product teams into some of our teams that are building tools, because they've got that customer focus, and they know some of the problems. Additional to that, when I set up the precursor to my current group, doing reliability engineering, we agreed with our product teams that they would send people on secondment for three months at a time, we will get two people from a particular group. That was fantastic, because first of all, we can check things with them as we're building them. They're also fantastic evangelists for the stuff that we've built. We find that they continue to submit PRs for our code. They find ways to use our tools that we didn't think about, but because they were embedded they've learned that and that's fantastic.
Caldwell: Sarah took the best answer, which is having a thoughtful way to rotate people in and out. I've also used some other approaches maybe more directly connecting engineers in the platform. This works for any place in your org, but connecting folks with what's going on in the design and research department, so they can directly hear output from research studies. Then, if not, having them sit directly in those sessions, then taking the output of those sessions and presenting them in all-hands, and forums like that. The biggest risk as a company scales is, particularly your platform team, like you're going to be focusing too much downward instead of externally. You have to find ways to keep people connected.
The other thing is, we also have used at various places I've worked, NSAT reviews, or some way to basically have customers come in and get in your face and tell you what the problems are that can either be driven by PM or sales. The key thing is to have engineers hear that feedback, even if it's raw and visceral, and they'll take it away, and at least know why they're doing what they're doing. I don't know if they'll adjust their behavior, but help them stay connected to the customer, which was the thrust of this question.
Shoup: I often do another kind of embedding, which is have the platform team person go and sit with the product engineering team. I think you were talking about rotating the product engineering person into the platform team. To be super clear, both those models work. It also has second-order effects in terms of camaraderie and cross team collaboration, and stuff.
Narayanan: Something Sarah said, I thought this might be helpful sharing, because camaraderie embedding. I was talking about embedding platform folks in product. I think when Sarah said, that helps these people build a lasting relationship to where they're contributing through PRs. One thing on the mental model that we use is, in one sense, you want to look at product teams as your customers, but you also want them to feel like they're partners. It's not just a customer provider SLA based relationship. It's, we're collectively trying to make things better. This mental shift of making them feel like partners, I think also helps.
How to Foster Collaboration & Knowledge Sharing Between Product Teams
Shoup: How do you foster collaboration and knowledge sharing between product teams, without it interfering with the flow of those respective teams?
Wells: I think it comes from your culture. If you've got a culture of people like to share what they've learned, and they like to talk about the cool stuff, then they'll naturally do it. You need to try and default to things being open. If we've got fortnightly showcases for a particular group, we want anyone to be able to come along and see it too. If you could persuade people to write blog posts, I had a great comment from my colleague, Anna Shipman, that actually your blog post is as good for internal communication as it is for external. You have to write it really clearly because someone is going to be reading it without context. It's a fantastic way to document stuff. You have to make time for it, but it pays off. For example, ft.com team, there was a blog post written about two years ago, and the architecture has changed a bit, but it's still a really good explanation of why they made certain choices. They use it in their interview process. People always refer to it.
Caldwell: This only works for companies of a certain size. If your company is a certain size, you should encourage particularly tech leads or folks who are passionate in a particular area to form guilds, where they can meet and discuss topics. We have had guilds for ML, backend engineering, frontend engineering. Then the key thing is to not assume that those are unpaid positions, so reward them in different ways. The places I think that do the best job of this structure not only allow guilds to be formed, but provide them with funding, and then also to some extent, incentivize this behavior through job ladders in the promo process as well. The idea of contributing back to share knowledge base gets formally recognized not just in a fluffy way, but people tangibly benefit from it. That also works for mentoring and other techniques. Always back up the behaviors you want with incentives. Your culture over time is just the accumulation of your incentives and disincentives. Don't just say you want something, come up with a system to actually reward it, or discourage the behavior you don't want.
Narayanan: Huge plus one to incentives and culture. I think, tactically, maybe one thing I'll throw out there is for enablement teams to also think, as part of their charter of being a builder of communities, we can create communities of practice. Maybe they're the ones bringing all their users together, and they could play a role there.
Shoup: I love the point of connecting up with just performance management, career ladder or career growth. We all know, almost by definition, whatever you call it, like staff, principal, when you're at the higher levels of individual contributorship, that's a first order thing. That's also a deal breaker, if it's not there, like, "Randy seems to be fine at the typing that he does, but he's not helping anybody else." That's a real thing.
Sharing Code and Components between Product Teams
How do we share code and components between product teams?
Wells: I feel you shouldn't. I feel like you should share services, but as soon as you share code, you run into problems. Maybe libraries, but it has to be something that doesn't change very often.
Caldwell: Very carefully. It's something I always hear, because in an ideal world, this would be super easy. Particularly as your code base gets more mature and more complicated, it becomes harder for people to contribute. I think culturally, always aspire toward if some team is wanting to contribute into a code base that's not in their local domain that we culturally encourage stewardship, and the ability for people to cross boundaries. What you don't want to have happen is, if you don't allow this, you create an unsustainable network of interdependencies that will prevent you from being able to ship with high velocity. You have to have some facility for teams to unblock themselves, rather than everyone just taking dependencies on each other. It's easier said than done. Generally speaking, the larger the code base is, the bigger the company is, the less likely it is that team A is going to be able to effectively work in team B's code base. Allowing internal mobility in the company and encouraging it is the most effective way I've seen this solve problems at scale. Rather than taking a cross team dependency, saying, can person on team A come and help out, even if temporarily, with team B's code base. Not only unblocks the inter-team dependency, but it also shares knowledge.
Shoup: The idea of stewardship, like contributing in an internal open source model between the teams. That's not exactly sharing components, that's scratching my own itch, but it's not the priorities of the team. When I ran App Engine, like that worked really well. Tying back, Nick, to one of your earlier points like, it was a line item on their promo packet that they contributed to this other thing. Just like an external open source project, you have to be super clear on, here are the contribution guidelines. We'll read your PR, but we won't necessarily merge it, and we expect the tests. There's a bunch of good stuff there.
Dealing with High Management Resistance
Any hints on how to deal with high management resistance to change even if the organization in general is willing to change?
Wells: I think change works best if it does come down from the top. We used to do tech talks, and we used to invite people in to come and talk, and it would generally be connections we had. It is amazing how someone externally to the company coming in and saying the same thing you've been saying internally, can have an impact. Charity Majors came and gave a talk about observability, and literally for the next six months, engineers all over the place would say, Charity said we should do this. It's like, this is great. Actually demonstrating that other people think this is an approach so that people feel, yes, actually, this is a standard thing. That can help.
Narayanan: I think the question maybe was also how to overcome management resistance. There's definitely hearing from industry experts. I think this touches upon a topic that we were discussing in the prep, which was how to quantify the business impact of enablement teams. I think that's where you start getting into management, because this is an indirect impact. I think Nick touched upon it in terms of NSAT scores and things like that. That's a whole other area. There's quantitative metrics and qualitative feedback. That's what enablement team is looking at in general, and that they can't make the case.
Caldwell: How you solve this depends where you sit in the org hierarchy, in terms of what levers you can pull. As a company gets larger, it will incur innovator's dilemma that hits foundational teams, as well as the end user product services as well. You always have to reserve some capacity to allow people to innovate. My default assumption is that it will not be 100% of the people in my management chain will want to just be hyper change tolerant, because as a company gets bigger, you're naturally disincentivized to do that. Where you go really wrong is if you don't have any capacity to reserve for innovation at all.
From where I sit in the org, there's two things I do. I always allow teams to spike ahead and experiment with new technologies or new experiences or new ways of working, and then bring that knowledge back to the core teams, and we can expand out. That way, you're not relying on massive culture change from all of your managers. If you're at a mid to large-size company, and you try and get all of your managers to be change tolerant at the same time, it will probably cause more trouble than it's worth. There's a reason it's called change management, you have to manage it. That's one thing.
The other thing is there are times when you cannot afford debate on change. If you have a burning platform problem, or if you have a competitor who is outpacing you, and if you have people in your team, who really just fundamentally want to do things the old way, that becomes personnel issue. You sit down with them and say, for this part of where we're at in the business cycle, these are the behaviors that we need. Then I'll let your imagination run to where that takes you. Sometimes you just have to make the call about what you need for a given moment in the business, and decide how patient you're going to be to bring systems and people along with you.
Shoup: I'd love just to reinforce that, I think all of us have happening. If you've worked in a company that's been around for a while, there's definitely behaviors that were adaptive when we were small, and they're not adaptive when they're larger. Really, anybody that's worked in an organization that's grown substantially has had this situation. This doesn't come from like, I want to fire this person. It comes from, let's be all open and honest and transparent with each other about, everything you were doing was super awesome five years ago, and it's not working now. We got a couple of options. A lot of times when you have that conversation very transparently with somebody, they're like, I see that and I'm on. Or, I see that and, no, I want to go do another startup. We all know people that are the early stage seed startup people and they're like, 25 people is too big. Then there are the people that are like, 10,000 people is too small.
Wells: For the Financial Times, it wasn't, we're growing massively, it was, we're changing our culture. We're adapting, and we're going from 12 releases to the website a year to 30,000 changes last year. We're releasing changes all the time. Moving to microservices, you need a different type of developer, both generally and on platform teams. Part of what you do is you look to recruit those when you have natural turnover, and some people who are in the team will make the transition because they see this and they say, "This is really cool. I want to learn it." Some people will decide, it's not for me, and they'll leave. Then you have a few people who it's not for them, and they don't leave. That's the more difficult thing. That's where the incentives come in, because that's the only way you can show this is the behavior we want. We did do a pretty big change and went from people who probably spent 80% of their time writing code. Then they committed it, and someone else packaged it up and deployed it. To people that do a lot of different things. That's a massive change for people's day to day work.
How Not To Structure or Organize Product Teams
Shoup: Anything you would highlight on how not to structure or organize product teams. Maybe you've encountered some team structures, which really don't work, maybe related to team size, ownerships, lack of ownership, or team member to product ratios.
Caldwell: There's an easy one for not to do, like the holacracy movement, which was popular a while ago. Anyone who's been a manager just looked at that and was like, "This is bare idiocy. What are we talking about here?" I think it works first. There's maybe one or two companies on the planet where that thing works. In general, as the company scales, you will need different organizational structures. An analog you might use is a cell splitting. In the smallest stage, it's just one cell, everyone's doing everything. Then it starts to split, and the two cells each have parallel functions, and so on. I generally think that there was periods and times when not having managers or having overly flat hierarchies was popular. We are, thankfully, out of that phase of whatever management philosophy, to a place where I think we're approaching more reasonable, we call it at Twitter, full stack teams. Given a project, make sure that it has all of the resources, people, and personnel necessary to carry on to whatever that mission is. Then you can then stack related teams into aggregate groups who are all marching toward the same direction, and then upward into higher level products, and so on. If you think about it this way, you end up with a nice hierarchy of organization.
Then you can also think about across all of those product type teams, what shared capabilities you might need. Those tend to end up in your platform teams. Then you also might think about, what teams do you need to better understand your users? Those tend to be data science, or growth teams, or things like that that sit across all of your product teams. That standard framing is what I typically recommend to people. Then the main thing is, like org design also depends on the type of people that you've hired and the culture that you've built. How much freedom or independence versus top-down hierarchy you're going to give people. All of this influences your org. Also, the product you're building, your product organization is usually mapped in some way to how your org and your product is designed. All of this stuff is a mix. Hopefully, that's good general guidance.
Narayanan: I think also, it's important to be intentional about how we structure our teams. Sometimes if you're not careful, you put something in place and you're just drifting. That's one thing. Then talking about incentivizing, let's be clear, our product teams incentivize for velocity, nimbleness, and all of that. The platform team, what do you incentivize for long term leverage? Are they inherently at odds? How do we bridge them? I think I've seen challenges when we're not clear about that. I think finding that balance and make it very clear. That's the people part. If you have more infrastructure-y people in your product org, there's tension there. Those are some things I would watch out for.
Wells: I think the key thing is, you don't want teams that have to wait on other teams to make progress. There's the whole thing in Accelerate, it's like if you want to be loosely coupled. You want the team to have ownership of something, and to be able to make progress. You don't want them to be sharing a test environment. You don't want them to be going to a change board. You don't want them to be waiting for someone else to approve a request for something to be provisioned. That structure means that you're looking at the places where things get held up and trying to automate it, trying to simplify it. If you have some part of your estate where you're not clear who owns it, that is going to cause you trouble, and you need to work out how that goes. That can be difficult because you will have some places where people want to make changes. If you're at the FT, people probably want to put things on the website for various product reasons.
Application SRE Embedded in Agile or Product Teams vs. Platform SRE
Shoup: What's the panel's point of view on application SRE that is embedded in the agile teams or product teams versus platform SRE to provide cross cutting concerns as a service? Is this model viable? Choose a model or a third model?
Caldwell: It depends on your org. My preference is embedded, but that assumes that you have enough resourcing and that your central SRE org has enough shared tooling and capability to support embedded engineers. You may or may not be in that state. At Twitter, right now, all SREs are coming from a central model, we're not embedded. I've worked at other places where we've managed to fully embed. One way to think about org structure and org design is like removing dependencies as a lever to very quickly improve your velocity. Embedding not just SREs but many functions is a way to accomplish that. To get to a point where you can embed, it does require your organization has reached certain levels of maturity.
Narayanan: I'd say that more strongly, totally in favor of embedding. I do feel like, to the point of maturity, yes. I do think that the enablement function does because it really rises in importance. I do feel like the industry is at a point to where there's enough out there that you don't have to build it all. You can get leverage from a relatively modest investment. I'm definitely a big fan of operate what you build.
Wells: I totally think you build it, you run it. We don't really have an SRE model. Our website is running on Heroku. We're outsourcing some of the work that you would do to create that platform for you. It's quite conscious. It gives that team a lot of freedom. Making sure you have the skills in the team, whichever way around you want to do it. Either because you have someone embedded who can do the work, or you've got something where you can get it from a third party, or where your platform or enabling teams have made something that's very simple, well documented, self-service, so that actually you don't need someone to be there embedded because it's really straightforward to do what you need. That's the Holy Grail.
Showing the Value of Platform and Enabling Teams to the Overall Business
Shoup: How to show the value of platform teams and enabling teams to the overall business.
Narayanan: Think of your platform as a product. How would you, I'm not saying sell it, but what's the value you're bringing? That's something you think about if you create a business and a product, so that mindset, I think you start with that. Then the tactics are, what are your KPIs? Often, you just gradually drift into building these things, "That'll make this easier." Then at some point, it's like, ok, shaving the five minutes off of this, is that the place you'd invest? Having some KPIs. In all our businesses, you have all kinds of metrics and dashboards that execs are looking at, how many of us have the same level of rigor when it comes to our internal enablement team? Trying to do that after the fact gets harder. Having some sense of that, I think, is it.
Then we touched upon the other aspects, the qualitative feedback surveys, and interviews. I think UX, somebody brought that up, that's the thing I personally have learned over the years. My customers are engineers, I don't have to worry about this stuff. No DX, DevEx, so that matters. I cannot understate the value of culture, because we touched upon this with the management. At some point, you can make the case. At the end of the day, we're able to say, it's about freeing up capacity so our engineers can focus on business. There's this indirect benefit as well, and have senior leadership buying into the culture. At the end of the day, we can have all the metrics, but I think the culture and the environment you create is huge.
Wells: I agree completely, the sales thing. I think, particularly infrastructure engineers don't tend to write something that says, here's this great thing we built, and this is what it does for you. Doing that, and thinking about how you'd sell that to someone who's not another infrastructure engineer, I think is really valuable. In Randy's talk, he talked about the fact you're effectively competing with vendors. If you have a culture where someone could go out and get this from someone else, the fact that they vote for you, proves that you're providing value. It's a bit scary the thought of offering that. You should be able to meet the needs of the engineers in your organization better than anyone else can, because you have them right there that you can talk to and find out exactly what they need.
This is tangential, but we run an internal engineering conference once a year. A couple of years ago, we did a panel on how standardized should we be. If you talk to technical leadership, they were very much on the, we should all be free to make our own choices. It turned out that a lot of our engineers were like, please, could we just have some standard things we can use? Having that panel where there was a vote and hands went up, it had an amazing impact on what people thought. Feedback is great, but asking the question in public and having people see what other people think, is also good.
Caldwell: I think you have to be, as an engineering leader, a strong advocate for your platform teams, because they're at a disadvantage compared to your user facing teams. Your user facing teams, when they ship something, it's going to get multiplied by the external environment. Your customers are going to be high fiving the PMs, and that's going to go all the way up to your C level. The odd thing is, if you think about the impact that platform teams have, it's actually a greater impact. It touches every engineer in the company. It's just, you don't have that magnification layer. The way I think about this is if you treat your engineering organization as the customer, and then try and find ways to get those folks to celebrate your wins in the same way that an external customer might, then you can get the same celebratory feel when you ship a change. Then some of those changes are really dramatic. Right now at Twitter we're moving a lot of our internal data tooling to GCP from some on-premise systems. We're seeing order of magnitude improvement in our ability to understand data and our pipeline speeds, and so on. This stuff is just as valuable as the fact that we shipped Spaces last week. We just have to always keep in mind like, impact, who's the customer? Then formalize getting quotes, or insert, or however you want to do it. Focus on making sure that that customer voice when they're satisfied, is heard, and that gets broadcast throughout the entire company.
Never lose sight of incentives. It's great to have high fives from customers, but you have to back it up with actual tangential rewards in your performance and review system. Just make sure that when you're sitting in the promo committee, or however your company does it, that it is just as valuable to have change improvements on internal infrastructure as it is to ship customer facing features. That has to come from engineering leadership.
Shoup: I'll just restate the problem. How do we prove to executives that there's value to doing this platform investment. To everybody's point, like, so much leverage. You add a marginal person to the platform team that should accelerate lots of other people's work, how do we measure that? One thing to do is you can use the four key Accelerate metrics around deployment frequency, and lead time, and change failure rate, and MTTR, which I am actually doing my personal version of this quest. The other thing is, value it in terms of the stuff that the business cares about, which is people, money, and time. If one, hypothetically, would imagine that XYZ product team wants to achieve a certain goal, and ok, product team says this is going to take us 50 person months to get from here to there. Then you can have the conversation, let's try to figure out what that 50 is, not because I want to argue down and beat you down. I trust your 50, but I bet I can make that 50 40, or 30, or 20, if we apply some engineering discipline to it. That's the unlock. You do that a couple of times or all the time, and then it's like, shouldn't we be investing more in this leverage? Again, remember what your business cares about and your customers.
Failure Scenarios due to Company Changes
Do you have failure cases to talk about, maybe situations where people left because they didn't follow the company change, or companies where changes were not easy?
Wells: Yes, absolutely. Any change that you do, you will have people who are up for it, you'll have people who will come along the way once they see where it's going, and you will have people who do not want to do it. You need to be very clear about what this change is and what it means, so that people can decide, this is not what I want to do. I've spoken to people in the past that I managed and had that conversation about, I think you would be happier in an organization that works like this. Obviously, eventually people can dig themselves in, but really, people who are motivated generally will go and find the thing that works for them.
Caldwell: You have to tell people the change you want and find people who are going to be supporters, and double down on them. You have to find people who are going to be detractors, and you have to ask them to do something else. Has this 100% worked in my career? No. I'll give a very specific example. I used to work at Microsoft, and I once got marching orders from my upper management that we were going to take a bet on the Windows Mobile platform. I'll just leave that dangling thread. I did all the above, found change agents. We took a bet on Windows Mobile, and then the whole platform dropped out from under us. People weren't happy with it, so it led to another round of change management. As senior leaders, you're constantly managing this change. It's par for the course.
Narayanan: I think speaking of change, it's probably worth understanding why someone is resistant to change. People fall in different cans. There's some great literature out there. Sometimes we think about it's not, are you in or are you out? It's, how do you tackle this cohort versus this cohort? That's something I would consider adding.
See more presentations with transcripts
