Transcript
Avery: Welcome to Scaling the North Star of Developer Experience. We're going to be talking about what it would take to grow a business through the different stages of business growth, and how we can change ways of looking at developer experience during those different growth stages.
Who am I? My name is Phillipa Avery. I'm an engineering manager of the Java platform here at Netflix. I've been at Netflix for about eight years now. Prior to Netflix, I was in academia. I did work for the government, as well as doing some small work in startups and medium-size companies. I'm taking a lot of the experience that I have gained from those companies and applying that to this presentation to say what it is that we should apply in terms of tradeoffs, and how we should measure those tradeoffs for different aspects of developer experience as a business grows. I'll be going through the different stages of the business growth that I'll be focusing, and what those specific tradeoffs are that we're going to be looking at. As well as going through some examples of how those tradeoffs have been applied with different processes for developer experience.
Stages of Business Growth
Let's start with talking about what the different stages of business growth are. We have a focus of hydras or different heads representing the different types of business growth that we have. When we start, we have a startup, which is focusing on a single thing, we're trying to get something out the door quickly. We have a baby that's coming out to try and really achieve one single thing, and achieve it well for the audience that we're trying to target. Then as we move into the growth stage, where we actually take what we have that we've learned from the startup, and we start to apply it to a bigger audience and we get more learnings. We might start branching out into different aspects of the business domain that we're focusing on.
Once we've gone past the growth stage, we really start moving into maturity, where we actually know what we've done well. We know what we're trying to achieve. We know the target audience that we're trying to actually work on and make sure that they are getting the best product that we can achieve for them. We're going to have different conflicting business prioritizations and different business needs. That is really represented in this maturity phase where we actually have to look really closely at what the different tradeoffs are, and make very key decisions. Which leads into the final stage, which is renewal or decline, which is more of an optional stage. You can actually continue to grow at maturity, with some pros and cons. With the renewal or decline phase, you really need to make decisions as to whether you're going to put all your bets in one particular aspect, whether you're going to kill off one aspect of the business. Maybe grow in a different aspect of the business and start branching out and start seeing different changes and complexities grow as you're starting to branch out into different aspects.
Tradeoff Focus
With those business growth stages, the next thing we're going to look at is the tradeoff. There are many tradeoffs which you need to look at when you're talking about business growth. Specifically for developer experience, we're going to be focusing on the tradeoffs of velocity versus stability. This is generally a scenario where if you need velocity, you will have to lose out in stability, and vice versa. The velocity means that you are moving fast and you're more prone to make mistakes, hence the loss of stability. If you are more stable and having to take things slower or making sure that you are making less mistakes, you would generally lose on being able to move quickly and highly constructively.
Startup
How do these apply to the different business stages that we have? Let's look at our baby hydra here with a higher weight on velocity. When we're doing a startup, we're really focused on getting some quick reads from the audience and getting a fast turnover to make decisions as to where we're going to focus on our product. We really need to have a very high velocity to get our product out the door with changes as quickly as possible. We will lose out on stability for that. That's ok, because we don't have a large target audience base at this point. We're really trying to focus on experimentation and product development, rather than we're focusing on making sure that the product that we have is stable for the user base.
Growth
After the startup phase, where we're really iterating quickly and trying to do more of a proof of concept rather than a stable product, then we start moving into our growth phase. This is where we actually start having different aspects of the business start to compete as we're looking at, where we're going to put our money, where we're going to put our favoritism of what we're focusing on. We might see a change in velocity and stability as a result. We have a larger user base now that we need to make sure we're maintaining experience for the users and creating a more stable product. Given that, we will probably see a drop in velocity to some degree, as we're starting to focus more on our stability and our resilience story around the growth of the business that we are currently seeing move out of a startup and baby phase, into a teenage phase.
Maturity
Once we reach maturity, this is where we have some really interesting questions on tradeoffs, where we have a number of branching areas that we could focus on and areas that we could experiment with. We might actually even start branching out into different business domains. We really need to find a great balance between velocity and stability. If we focus too much on stability at this point, we lose our ability to make choices and move and continue to adapt to the market that's out there. We really need to make sure we're finding ways to maintain the velocity as well. This is where a lot of the developer experience will come into play. If we are bogging down our developers with their stability requirements and stating that they don't have to push or do any changes due to the fact that it will break the service, then we're actually reducing our ability to innovate and create velocity around that. We really need to make sure that we are focusing on finding that balance, both in terms of the business requirements, but also in terms of the technologies that we're providing to our developers, to allow them to continue to keep developing with this balance happening.
Renewal or Decline
Finally, we move into renewal or decline. It's hard to represent using an icon. Essentially, we are looking at more of an up and down here where you might see different areas of the business experiencing a high degree of velocity, and others are going more into maintenance mode entirely, where they will completely focus on stability. They don't want to make any changes to that particular type of the business. It might be fading out, or it might just be perfectly good and we're not looking to innovate in that area anymore. It's just an existing customer base that will continue. We want to make sure it has the least amount of changes needing to happen. You might focus mainly on security. Whereas other areas of the business, you really need to keep cutting edge. You need to make sure that you're providing a developer experience which is moving forward quickly.
Choice of IDE
Let's go through some of the examples of how we apply this at Netflix and the choices that we've made over the years. We're going to start with a really simple use case, which is choice of IDE. When I first started at Netflix, everyone could have whatever IDE they wanted. We had a mix of Vim versus Emacs, which of course Vim wins. Then we had Eclipse and IntelliJ. You name it, we had almost every IDE that you have available, including Visual Basic at some point as well. I have seen that. What we found was that this actually really worked well for our startup scenario when we had a very small amount of engineers coding their code base without having to make changes to other people's code bases as much. There was no real need for consistency of IDE at that point in time. As we grew, we found that having a recommended version of IDE was the right balance. By recommended version, we actually chose IntelliJ to say, this is the one which we will support. This is the one we'll provide licensing for. This is the one which going forward if we're going to create things like plugins, there'll be plugins for this particular IDE. At some point, we probably will need to make the choice of actually saying, and this is now the only IDE which you should be able to use if we start bringing other aspects in that we don't currently have. These are the sorts of things which we're looking at when we decide the tradeoffs again, velocity versus stability for our IDE choice.
Monorepo vs. Multi-repo
Next we're going to talk about monorepo versus multi-repo. Netflix has a multi-repo, we use library version management very extensively. We made the decision early on not to do monorepo, and it has a lot of tradeoffs in terms of velocity versus stability. This is actually one of the tradeoffs where we're continuing to lean on velocity and having a higher propagation speed, rather than leaning on stability, where we're going to have less breaking changes, potentially blocking different services from being able to release their application. When we have a release that happens from the libraries owner's perspective, they're able to release their library and it will go out into the environment, and through our CI/CD process, it'll slowly get picked up by the different consuming services. Those consuming services, 90% of them are probably going to pick up that library without any issue.
We have a bunch of version management and dependency graph management that happens from the customer perspective, in order to make sure that we're locking different dependencies to reduce the variables of what they're picking up. It still has a large degree of complexity and a large degree of failure. For those 10%, for example, of services that have trouble when they're picking it up due to conflicts in dependency management, particularly when we have transitive dependencies potentially locking some things, they are pretty gnarly to try and fix. We spend a lot of our support time looking at that, which we wouldn't have to if we had a monorepo. The downside to that is it could actually block being able to release if you're not able to get through those 10% of services quickly enough to release. By having our multi-repo or having our dependency publication, we can then also publish it for the services that need it quickly and make them focus on plugging it in really quickly. It gives us more flexibility of choice.
Of course, with any service like this, you all have to have a high degree of CI/CD. Specifically with monorepo, you really need to invest in your integration testing story. That is something that we have not done particularly well and we really need to focus on getting that done right, either for monorepo or for multi-repo. With monorepo, it is particularly important to have that story correct. The other approach, which we actually do have some degree of is hybrid approach, where you have different aspects of the business, which will have a monorepo versus other aspects of the business which will continue to have a multi-repo. We are focusing on this. It can be hard to find aspects which are self-contained enough to be able to create a monorepo on their own, but they do exist. There might be a very well informed reason as to why you have it for one aspect and not have it for others. This aspect might be contained enough that the velocity is not actually significantly impacted, versus a very large monorepo for an entire organization where you do have impact from velocity versus having a more stable release cycle.
Build vs. Buy
Next, we're going to talk about build versus buy. When we have the aspect of the choice of whether we're going to create something in-house, or whether we're going to purchase or use open source products that are available externally, a lot of the time, what we'll focus on is, can this actually achieve what I need it to do? Probably, if you've actually been trying to make this choice, there is going to be some small things which won't match up to what it is that you're looking at, that is available to purchase, or utilize in open source. Will it give me some degree of customization or tailoring? You have to make that choice as to whether it's worth doing that customization or tailoring where even possible with the choices of buy versus building the entire thing in-house.
One of the mistakes that people often make is to say, it's free, essentially, as once I've purchased it, that's the cost, or once I've used open source, that's the cost. Then I continue to keep rolling it in. In order to use open source, you've got to remember that you're actually maintaining and you've got to have a great relationship with the open source community, or else you're going to end up branching, and you're going to end up having to maintain your own product anyway. Which means you've got to be highly active and engaged with all of the requests that come in for different needs and requirements with different business use cases from other people who are also using the same product. You need to try and make sure that you're influencing as well as contributing back to the open source repository. Same with if you're buying something, you need to talk to your vendor, and make sure that the needs of the company are being met with the product and the future direction of the product. Which is going to cost money either in terms of asking the vendor to do something, or in terms of your engineering time to make sure you're on top of the open source community.
That in itself is going to outweigh the support costs of choosing build versus buy. The real choice comes down to the scope of the problem. If it's something which is going to be a quick maintain, like build scenario, then you might choose to build and maintain that in-house. I think the learning curve of anyone who comes on board is pretty easy, because it's a small scoped product that you've created. If it gets to a larger scope with branching complexity. At that point, you might be better off leaning into saying, there is going to be complexity that we have to manage anyway. We might as well take advantage of the open source or buy alternative, in order to make sure that we are on top of what the current community outside of our company is, so that we are keeping up to date in cutting edge. Additionally, it actually helps expand your knowledge base as you start bringing in people outside of your company that are really pushing on what you assume to be true, given what you are experiencing inside of your company. Having the scope of the problem increase, actually, you might lean more on wanting to buy at that point, far more than build.
The Blooming Hydra
Finally, we're going to talk about the blooming hydra area. Abstract, but once you start growing your business, and you start growing into different areas of the business, now again, you're going to have different needs and requirements. You might have completely different aspects of the business that have different needs. For example, when talking about Netflix, we talk about what we refer to as our streaming need, which is our product for streaming. We might have the actual content, which is one different business aspect, versus the UI, which is another one, versus the actual playback itself, and what it takes to stream the content. That is what we call our streaming side. Then we have our studio side, which is all the technology around what it takes to actually produce the content, and record, and have the whole studio production side of things. They have completely different needs. We have developers across the company which are working on these, usually in isolation, but there are some degrees of leverage that we could apply from the lower services of developer experience here.
The key thing comes down to whether we're looking at generalization of a particular aspect of developer experience, and how much complexity you're going to introduce by needing to make sure a single product is going to encompass the needs of all of these use cases. Here, we're going to apply the principle of KISS, of Keep It Simple Stupid. Try and make sure that you're not over-generalizing, and that you are not making peace with a higher degree of complexity. If you start finding that you are needing to really slow down to the point of decision paralysis, because the amount of things that you will need to include and change will actually have repercussions across board. At that point, it might be better to actually separate these products even though you have redundancy, and even though there might be engineering redundancy or support. You're going to have a degree of velocity that will be able to be maintained specific for different use cases.
That's that tradeoff in terms of generalization or complexity. Do we want to have a single support system for the entire product across all of these different use cases, or do we want to have what we call redundant localized developer experience? Where we have a local team specifically supporting potentially the exact same technology but in a different way for this particular use case, and the same with this use case, and the same with this use case. In this case, redundancy might actually help in terms of velocity. Then you need to look at what the cost of what it is that you're saving for being able to quickly cycle and not have so much complexity, versus the redundant engineering headcount or areas that we might potentially be able to leverage if we put into a smaller centralized group.
Stop, Look, and Listen
Finally, stop, look and listen. In Australia we have this song, which is, "Stop, look, and listen, before you cross the road." It's been many years, and it's still stuck in my head. This is a reminder, this song stick in your head, stop, look, and listen before moving on down the road. Any time that you are making the decision to move down and to bring in a different type of developer experience, bring in a different type of process around developer experience, remember to apply these principles. Firstly, apply agile thinking. Where we're used to applying agile thinking to when we create software development, we also should be applying it to the decision making around developer experience. How we're going to say if we can quickly roll back this decision that we're making. Let's just move forward and try it. Let's take the learnings, let's change directions if need be. Don't get stuck on decision paralysis. Should we do this? What's the long-term repercussions? Take that into mind. Take it all in and then make a decision and try it in the smallest possible way. Take that agile thinking and apply it in every single time you make it, whether it's in a single domain, multiple domains. Try it, move forward. Take the learnings, change directions, or keep going if it's showing it's the right way to do it.
Part of this is also to avoid sunken fallacy cost. Doing these small iterations of agile decision making and agile development for a developer experience will lose money. It probably will, but in the long run, you are actually going to have a better experience and a better product. Same as what we do with agile thinking in software development. There is going to be a cost. Don't think of it as the sunken cost. That is perfectly ok to have some loss here. Keep moving, keep the momentum moving forward. Make sure that we are balancing again that stability versus velocity in our decision making process.
Make sure we are actually budgeting for that reevaluation improvement. Every time that we make that decision to say, ok, we're going to not do this anymore. That's actually something we need to budget for. It's not just something that we can just push off to the side and forget. Tech debt isn't an additional budget, it's something that we should include in our budget, when we're actually making decisions to say, we're not going to do this anymore. We are going to move down a different path. Next, make sure that you are building insight on developer experience from the get-go. This shouldn't be an afterthought. Every time that you create something which is for developers, create measurable, quantitative metrics inside a product, where you're measuring how they use it, how often they're using it, what their workflow is that they're using it for. Even if there are only a few uses built in, as they come, you will really want this metrics analysis in order to decide what the decision for how to move forward with this product should be.
Part of the pitfalls that you need to watch out for are, again, this growing fragmentation across this user base. If it's deliberate, make sure that you're accounting for that as to what it is that you will and won't support as part of that fragmentation. If it's not deliberate, then make sure that you are reaching out to people and showing them what the repercussions of having this fragmentation are. When you're making choices, make sure that you're again trying to use these measurement insights that you've built in, and that you are looking at the qualitative and quantitative balance, and not just listening to the loudest voice in the room. As a developer experience inside a company, it's really easy to be swayed by loudest voice in the room, because people know you, they know where you sit. They can come talk to you. They can come ask for what they need from you. You will be very eager to try and solve their experience for them because they're right there and you can talk to them and get quick feedback. That might not be the best thing that you can actually focus on at this point in time, it might not be the highest leverage of what you can really achieve. By having that balance away, you can actually say, ok, this request is really important for this particular person. However, we have these quantitative metrics which show that this change will actually affect others adversely, or potentially isn't the best thing that we could actually focus on at this point. It will help you make better decisions going forward.
Hydras FTW (For the Win)
Hydras for the win. Really, we want to make sure that we are saying that there are different businesses at different stages, and will require different needs. If you take anything from this talk, please take the key thing that change is not a scary thing. If we make a bad decision, it can be costly, but it usually means that we can recover from it in some degree by making a choice going forward. If we continue to keep pushing down the same path, then we can have issues as well. Make sure that we are actually saying that we are going to change with the idea of velocity versus stability. We might say a particular aspect of the business might not change, and we're ok with letting that go into maintenance mode. Overall, we need to make sure that we are creating a velocity aspect in some area of the business, that we are continuing to improve and that we are continuing to keep challenging ourselves as to how we really provide a developer experience that enables them to have a high degree of velocity in different aspects. We're not trying to solve for world peace, although sometimes it may seem like we are in terms of the complexities that we're dealing with. Essentially, this is a solvable problem. This is something that we can get through. Make sure that we are looking at the metrics. Make sure that we are looking at the tradeoffs. Ultimately, make the choice, be agile, move through with it, and continue to evaluate at regular points in time.
Questions and Answers
Kerr: You talked a lot about the most important point being you can change your mind. What's something that you all implemented for developer experience that you changed your mind on?
Avery: We're constantly changing our minds. Actually, let's talk about something which we haven't changed our mind on yet but we're constantly reevaluating. If we go to the IDE choice as an example, we were talking about how we have this non-single choice. We haven't actually said we're going to dictate that this is the IDE that you have to use. At some point, and I missed this out in here, we might have to say, we're going to do a specific type of IDE which a lot of companies you'll find use what we call remote IDE versus local IDE. Remote IDE is where you essentially will have a centralized website, which you do all of your development on. You don't do any of your development by pulling down your code locally onto your laptop, it's all done remotely. We've made the choice to do local development, but we are actually continuing to evaluate whether we're going to do remote development, because it does have a lot of benefits that you could get from having that centralized control, as well as having the exact same hardware that you would have when you do your deployment. We are constantly evaluating, we're currently doing it at the moment, what the pros and cons are, and whether the business value for implementing that infrastructure is actually worth making the changes to migrate everyone over from having the local development to the remote development.
Kerr: Do you have some people doing remote development now?
Avery: We have people that experiment with it. Ultimately, though, that is an investment in infrastructure, more so, especially when we're talking about trying to get the networking as close as possible to developers. Having essentially the distributed system which we have all of our production services, need to be as close to the developer as possible, literally, as in regional, close to the developer. We, at the moment, have three main regions that we work on, and trying to get it closer so that the actual interaction cost over the wire is as quick as possible, is the main thing that we would need.
Kerr: Getting the responsiveness that we get naturally from local development?
Avery: Exactly.
Kerr: Some decisions don't feel like ones you'd want to change your mind about, like we have a monorepo and it's not going well, but the pain of splitting back out again, we feel stuck. Have you experienced anything similar like that?
Avery: Yes, 100%. I think the monorepo is one of the trickier ones, because as you grow, rolling that back actually gets harder. What you can do is you can start breaking it up. That hybrid approach is actually an interesting one, which you can take and start saying, ok, we're going to say that this is a monorepo, and we're going to have the right degree of CI/CD around this monorepo. Now we're actually going to start pulling in more use cases into that monorepo. You can do a split approach. Think of it as Roman riding almost, where you basically have the old and the new at the same time. The Roman riding is where you have two horses, one is a fresh horse, one is the horse that you've been riding, and you switch the horses. You essentially will have the old business value, which is not in the monorepo, but you will have a new business value, which is in its monorepo. Then you can actually start pulling use cases, and repositories start getting rewritten. I'm a big fan of constantly rewriting code, if it's still being used, and constantly making sure that you are evaluating the way that you are doing business requirements in your code. Part of that is rewriting it and seeing if we can actually create a new microservice of it. As you're doing that and you're creating a new repository, you then start putting that into the monorepo, if that's the choice that you've decided to go forward with.
Kerr: Bit by bit. It seems like it would be easier to go into a monorepo than out, but still you can focus on the areas that you're working on. Domain driven design has some good pieces about that.
Do you have a single mandated issue tracker?
Avery: We do not. We majority use Jira. There are a bunch of other use cases.
Kerr: You use the same Jira, or do people like flex it because I know Jira is not just one thing?
Avery: Individual teams will have their own Jira projects, and each team chooses how to use it, and they choose their workflow. My team in particular has projects and boards for each product that we own. Therefore, if you want to have a ticket for a particular product, you will go and make a ticket inside that particular project.
Kerr: How do you source requirements from your engineering teams for the developer platform, if that's a thing you do?
Avery: Yes, we do. I like to have what I call both inreach and outreach. Inreach is really easy. You're sitting there, that's that loudest voice in the room. People will come to you and they'll say, this is what I want. It's easy to react to. It's easy to obtain, because you just sit back and let it come to you. Especially when you're really busy and under water, it potentially is the easiest way to get customer feedback. Doing the outreach balances that off. By outreach, I'm talking about going out and speaking to users, planned. Saying, we're going to target each aspect of the business and we're going to talk to them independently. We're going to make sure that we're getting surveys from our user base. Try to get a good representation, not just the people that are the ones you talk to the most. You want to make sure that you're doing outreach to the people that you actually don't talk to the most.
You want to make sure that you are setting up different ways of achieving this. Some of them is through having communications, we use Slack, to having the Slack support. We have office hours as well, which is that inreach then again. We have constant, we call them roadshows, where we go to the different teams, and we actually say, these are what we give to you. This is the how you use it. We have education where we actually will go out and seek to make sure that there is best practices being applied to the products which we are providing to people. As well as that survey. The surveys can be hit or miss. You really need to try and make sure that they are as small as possible, and that you're not doing them too much, because then people will start getting burnt out by them. Once a quarter, we have one big one that will go out to different targeted user bases and get that information.
Kerr: It's almost like you have a little miniature company with marketing and customer success and documentation.
Avery: Yes, 100%. I think that as you scale, that's really important because, otherwise, the fragmentation I was talking about in terms of how different people are using not just the different technologies, but also your technology in different ways. You might provide them a product and you don't anticipate the way that people will use that product all the time. Suddenly, they'll come back with a support request, and you're like, I had no idea that that was how you were using that. Then you have to try and figure out that tradeoff. If you can reduce that fragmentation simply through education and understanding prior to that happening, how people are using it, you can get on top of it and try and have a more consistent experience overall.
Kerr: How does Netflix's culture affect developer experience?
Avery: A lot of people have heard the freedom responsibility aspect of Netflix. Freedom responsibility is essentially trying to say that people should have the freedom to make the choices that they want to make specific to their business needs. As long as they are responsible for those choices, in terms of the cost of those responsibilities and how that will affect the company's bottom line. Especially as a centralized platform team, this can hit pretty hard because that fragmentation of what people are using and then wanting to come back and get support for it can actually increase if there is no dictating to say we only support this one specific thing. IDE choice is an example of this, where you might have people that are using Eclipse or are using VB and come in and say, I need support on this. We're like, we don't have the capacity. We don't have the bandwidth, and we don't have the knowledge of how to support that particular type of product.
It has pros and cons. What we try to do as a centralized business is be very clear on the responsibility aspect. We don't tell people, you're not allowed to do this. We say, if you choose to do this, here is what you are responsible for, and here's what the repercussions of making that choice could be going down the road. We have what we call paved path and off paved path. Paved path is our well supported products. We ask people to stay on paved path, because it will help us to actually support them going forward. If you choose to go off paved path, here's what the actual repercussions are, especially if you move to a different team in the company or you move on.
Kerr: What's an example of the paved path?
Avery: One of my teams is the framework team at Netflix for backend microservices. We use Spring Boot for that. It is a Java based product, and we support Java for our backend services, a majority. If someone decides to go and create a product in Go, for example, because there is a perfectly good business use case for creating it in Go versus Java, we will educate them and talk through the repercussions of that in terms of support down the road. If they make the decision, ultimately, that this is the best thing for that particular use case, that is up to them.
Kerr: What are the Java teams getting that the people who write it in Go are responsible for themselves?
Avery: What is the support?
Kerr: Yes.
Avery: The Netflix ecosystem, for example, we have a bunch of metrics integrations which are in-house. We have some really good metrics tools to be able to have operations of your service in production, traffic insights, tracing, real time metrics, and that's written in Java. The integration is written in Java, the API is in Java.
Kerr: You get this insight into what's happening in production just naturally if you use Java and integrate with the other tools.
Avery: We have a bunch of different ecosystem, what we call ecosystem integrations, metrics is one of them. There are a lot of others. Even our security is written primarily for Java. We do have polyglot solutions now, so that security does need the ability to be able to have it regardless if it's in Java or not, but it's not as paved path supported and easily integratable.
Kerr: Developers in Go would need to find their own way to output metrics, and get those into whatever frontend.
Avery: Potentially even rewrite the libraries that will be interacting with the ways of doing that.
Kerr: Because they do need to mesh with the Netflix ecosystem, but that's not going to be done. It's not going to be trivial.
Avery: Then that rewritten component that they've had to rewrite in Go, they need to maintain that and support that going forward. If they move on and go to a different team or a different company, someone else has to be trained in how that was rewritten. As opposed to a centralized team owning and maintaining it, and having that constant iteration of new learning curves, being up to date with the current status quo.
See more presentations with transcripts
