Transcript
Scanlon: I'm going to start with a quick confession, which is something that I need to get off my chest when I'm talking about high-performance discovery teams. The first thing is that I was a software developer, and I was a, very bad software developer. I'm currently a product manager. As I was a bad software developer, I've also been a part of some teams that I would describe as fairly low-performing. I've seen some things out there, I've seen deployment processes that take teams of people eight hours to get something out to production. I've seen entire codebases that don't have any automated testing at all, teams that don't track their work, all different sorts of things. As I transitioned my career from engineer to product, I wanted to make sure that I was a product manager that engineers were going to enjoy working for. I wanted to have a high-performing team. I wanted to make sure that people were comfortable with the work. I think that over the course of my career, I can say that I've done that a little bit.
I want to show a chart right here to give a quick definition of what I might consider a high-performing team. This shows a series of iterations that we've performed. We're tracking the standard velocity here, both in tickets and points. I think this says that, as you see towards the end, we're getting very consistent in how we're delivering things. We've almost doubled the output of the team from when we started to where we are now over 20 or so different iterations. You see there's a metric at the top that says greater than eight. We measure happiness and engagement, our scores are consistently very high. I feel fairly vindicated that we have a very high-performing delivery team right here. Pretty proud of this.
However, you're probably guessing that there's one caveat to this, and that's that this team was mostly in charge of innovating. We were supposed to be building new products. What you see here, what these normal metrics show you is that we're delivering working software at the end of every iteration. It's working very well, but we're not necessarily having that innovation. We're not delivering a lot of value to our business. My talk is going to explore a little bit about why that is.
I'm going to give you the key learnings upfront, and then keep coming back to them, layer them in as we go through. The first is that there are fundamentally two different types of work and two different ways of thinking. I'll break those down as basically the delivery work that we were talking about, shipping working software, and then discovery, which is basically understanding things about our users, learning different things as we go, validating hypotheses, that thing.
As we think about what makes a team successful in discovery, there's four areas that are a little bit different than a team that's delivering working software. The first is maximize learning. Then the tactics on the right here side that we use to do that are accelerate delivery and use MVPs. The other areas of focus are going to be better ideas, alignment metrics. Again, we'll talk about all these in more detail. The topics are on the left, the tactics are going to be on the right, and we'll walk through those.
Our Team
I want to start with a bit of context and a bit of introspection. I don't know what you all know. This is the area that I come from right now, we're working on an atomic product team, which means that we have everything we need to do our job successfully. There's a product manager, typically a designer maybe half time, but at least some portion of a designer is working with the team in an embedded capacity, and then engineers. We have typically three to six in the environments that I've worked in. There are also some other functions that aren't represented here. Where I am now at Flatiron Health, we have quantitative scientists, which are basically data scientists. They might have a typically a different background, but if someone is supposed to be building a product, they're on that team. We normally use something like Scrum, Lean, or Kanban, but it's typically up to the team to choose the tools that they need to actually get things done.
The example that I'm going to walk through to illustrate the points I have is from a company called Handshake, which does B2B e-commerce. I think this might be fairly relatable for a lot of the people in the room. Handshake was recently acquired by Shopify. This is background for what these examples are going to be. The context that I want to walk through here is that this team was tasked, after having that good delivery output, with building a B2B payment system. We already have a record of this team delivering software very well over a course of many different iterations. Now we're expanding to a completely new part of B2B e-commerce that Handshake has no prior knowledge in, the B2B payment system.
As we know, we've been delivering this software fairly consistently, however, it hasn't always been delivering value to the customers. I want to take a step back from a 30,000-foot view talk about why that is. Basically, I to visualize work in this way. This is an arrow pointing to the right, pretty straightforward. This is our delivery track. I mentioned we were using Scrum. We have these little iterations here, this represents how we do our work, it's usually two to four weeks at a time. We start with a sprint planning, we put working software out at the end. We have daily scrums, we inspect and adapt every so often. That's all gravy.
However, there's a different concept and area that we were completely ignoring in the process frameworks, which was the discovery side of things. This is something that Jeff Patton and Marty Cagan talk about a lot. They do a lot of product owner trainings and product management trainings. Essentially, "Discovery" are interactions that you have that help you learn something either about your users or about your products. It's typically the domain of the designer, the product manager, and some members of the engineering team, who are out there learning and validating different things and figuring out what it is that they're doing. At some point, the learnings that you have in the discovery track are things that flow back down into the delivery track so that you can do things reprioritize your backlog, do backlog grooming, those sorts of things.
It's important to mention here that I think discovery is going to happen, whether or not you're planning to do it or not. It happens anytime a user interacts with your product or a bug report comes in, incident, anything like that. Discovery is always happening, even if you're going to be thoughtful about actually going out there and doing it.
One of the first things is that there's fundamentally two different types of work, as well as two different ways that this can be structured within team mandates. If you think of a perfectly balanced team who has a mandate from the company to do 50% of delivery work versus maybe 50% innovation, you may have teams that are very committed to roadmaps and projects, where I would say maybe up to 80% of their time is already built out in a structured roadmap. Maybe you have a day reserved for personal projects or innovation or something like that, but you could go up to 80%.
As a product team, a lot of what we're doing, and especially in a new product space for something like building out a whole new part of an e-commerce system, would be really much more focused on discovery. We might have something that's 20% delivery, 80% discovery. This is how we would try to think about balancing our time. What this says is that this team is supposed to be innovating and finding new spaces to work, and not necessarily focused on delivering software. You can see the first problem here is that we're almost entirely describing our process in terms of the software we're delivering, when it's not necessarily the key output of the product team right now.
The other learning about this is that the context shifts from the team from iteration to iteration. Even though we have a team that's focused primarily on discovery, as you see in the early iterations here, at some point, we're going to circle around a solution. We're going to converge on what actually needs to get built, and then we might shift into more delivery mode. Maybe two months in, we've done a couple customer development interviews, we feel pretty good about what it is that we actually need to build, we're going to start shifting from thinking about the discovery side of things to actually building the software and getting it out the door in the typical Scrum, Agile, Lean processes that we have.
At some point, that's probably going to shift away, maybe actually release a feature. We'll probably still monitor it to make sure it's working properly. Maybe something else comes up on our roadmap, and then we have to start the process all over again. It's a continuous cycle that we have to go through evaluating which context it is that we're working in. The first key learning here is that there's fundamentally two different types of work, whether we're planning for them or not.
Cynefin
Next, I want to touch about on Cynefin here. I think this is interesting, every talk I've been in so far has mentioned Cynefin. Before today, has anyone heard of Cynefin? A few. How many people after today have heard of Cynefin or seen some introduction? A lot more. A quick background, I'm not going to spend too much time - Cynefin is a Welsh word that means habitat or place of abode. It's a framework that says there are four different types of problem areas, four different ways that problems can react and systems can react. The little gray box in the middle here is that we don't know which domain that we're in right now. I'm going to talk through a couple of different examples here.
The way to read through this framework is thinking about a problem in terms of cause and effect. In the obvious quadrant in the bottom right, we might have a very straightforward relationship between cause and effect. This might be something in terms of engineering terms where we have a very simple checklist for something that has to go out, or a playbook that gets written where someone who doesn't necessarily have domain knowledge can come in, pick it up, understand what to do and solve the problem that's at hand.
The next level is "Complicated." This is something that has a lot of different inputs, a lot of different variables that go into it, but it's something that requires domain knowledge and can be solved algorithmically or heuristically. A complicated problem, we'll walk into an example, it's something where you can understand the current space that you're in, all the different inputs and outputs that you have, and figure out the best path forward.
A complex problem is something where that path forward isn't apparent until you actually encounter the problem until you do something to change what happened and see how the system reacts to what it is that you put out there. If you think about complicated, it's very straightforward for me to have the domain knowledge I need. I can debug, I can do something, I can fix the problem. For a complex problem, though, I'm not going to be able to do that until there's some outside system force that gives me more information. I have to get through this process very quickly. Then chaotic is something that we don't necessarily have to go into but means there's no apparent relationship between cause and effect.
For some concrete examples of what a complicated problem might look like, this is one of our failing builds right now. I hope it's not still failing. I can see here that the build failed, I can see the error message. I get a stack trace in our logs. If I am someone who has the requisite domain knowledge, I can look at this, I can walk through what actually happened when this build failed. Then I can use my expert knowledge to figure out how to proceed from there. If it's the test case needs to be deleted because it's obsolete, if it needs to be updated to accommodate a new use case, I should have the context to actually go through and figure this out start to finish without actually having to encounter the problem any further.
This is something that we use a lot, called a story map. I think this represents what a complicated domain looks like. The way to read a story map, if you're not familiar, is that we have broad buckets of user actions that are at the top. We have different paths that they can take through the system at the bottom. We like to slice these things horizontally so we can figure out what should be in an actual release. This is essentially something that we use when we're encountering a new problem space or trying to build out a new feature that will tell us what it is that the user cares about and maybe where we should be focusing our time.
This is actually one that we did when we were building out the new B2B payment system to figure out what our different options are, what users are going to care about and figure out how to structure those releases going forward. That's part of the problem. We did this in a vacuum. We tried to take this complex problem of understanding what it is that a user would want to do in a brand new feature in a brand new area of our product, and we tried to distill it down into a complicated problem, where we said, "These are the features that are out there. This is the release plan that we've identified, and we're going to go and plow through and build all these things. Maybe we'll put some logging and monitoring in to make sure that people are using it. Broadly, we know best, we're going to go ahead and do this." Where we wanted to get to was more on the complex side. We needed to build in that uncertainty that said, "We don't know how the users are going to react to this, so how can we get those learnings as quickly as possible?"
This is a good reference if you're interested in going into more detail about complexity and complication in business. We already talked about the first step, which is recognizing the type of system that you're dealing with. You also have managing and not solving. We're not intending to solve user problems, we're intending to manage the system and set ourselves up for success. Try, learn, adapt whatever it is your chosen OODA loop of choice. Just making sure that we're constantly ad ing how it is that we're thinking. Then the key point I have here at the bottom is developing a complexity mindset. All this means that we want to be cognizant that there are areas where we don't know best, where we have to get things in front of users as quickly as possible so that we can get these learnings and figure out what it is that we actually need to build.
Basically, these are the two key learnings of what we needed to change. There's discovery work, there's delivery work. We were treating both of these as the same thing, but we needed to split these out into different areas.
Our first thing that we tried as were doing this is that we wanted to maximize how much learning we were having. If we understand that we're operating in this complex environment, we want to make sure that we are encountering this problem as quickly as possible so that we're getting a lot of feedback faster and can act on that feedback later on. This is the model that I showed before. We had infrequent integrations from the discovery track down into the delivery track. What we wanted to do is make sure that we at least doubled this. We wanted to learn as fast as we could and make sure that we were actually getting that information back to where it needed to go.
The other thing here is that now since we're learning so quickly, it doesn't necessarily match up very well with our process model from before. Learning is oftentimes very messy. It might not be something that fits into the Scrum process, and we had to adapt as we went to make sure that it still worked for us.
MVPs
The tactic that we used to accelerate that discovery was the concept of MVP, a minimal viable product. The goal of the MVP is to maximize learning while minimizing risk and investment. When we were thinking about this from the discovery standpoint, all of our MVPs were going to be single-use case MVPs. We thought that paring down the functionality enough would give us enough leeway to put it out there in front of users and get that feedback. However, there's several other different layers of things that we could have been doing to make sure we got that feedback a lot faster. Recognizing that we wanted to do that, we had to try some different things here.
I have an example up here of a paper prototype that our designers put together. This is something that we can easily draw out in an hour, and then we can validate the mental models we have of the problem. We can put components on the screen, present it to a user and see if how we're thinking about the problem makes sense. If we need something more high fidelity, we can move into an Invisionapp thing that has nice high fidelity wireframes and put that in front of them and see if that's how they were thinking about it so or not. The point of all these, to have these different types, is to make sure we're validating the next thing that we need to do as we go through it. At the lowest level, which is validating the problem space, do we understand what it is that a user wants to do? Are we thinking about something that may be fairly foreign to us, as a B2B payment system? Are we thinking about that in the same way that the user is? Just making sure that we're all aligned on that.
The next stage might be validating usability. Can we put something together that they'll actually be able to use? Then at maybe the higher stage where you have a more high fidelity HTML mockup or something, we'll validate the interactions that we have as they go. There are really a couple different things we can learn as we go, and we need to adjust the tools that we use accordingly.
"Lean Customer Development" is really good for figuring out how you can structure some of these MVPs. I put some other types up here because I think these are interesting if you start to see them out in the wild. I actually got MVP the other day by Warby Parker. They have a new app on their website. It says, "Enter your phone number and it will text you a link," but they did not actually text me a link. I'm guessing that's an audience-building MVP that they have out there to see if there's enough demand for that new app.
That's the first area we focused on. We were trying to maximize the learning, so we accelerated the discovery cycles. Then, to do that, we had to use MVPs to actually get them done.
Idea Flow
Now that we're having these learnings, we want to make sure that we're not only having them faster, but we're having higher quality ones as well. This is a representation of the team again. We wanted to make sure that the ideas were bubbling up from within each of the team members and that they're getting the most out of all the learnings that are going on. The key thing for us to do this, and I don't think this is very new, is making sure that we have a comfortable level of psychological safety. Project Aristotle is a study that said there's a couple key factors in what determines how well a team is going to perform. At the highest level, it's psychological safety. This is going to be the best predictor of how your team is going to do.
From within our team, we were able to focus on making sure team members felt safe to take risks. They could be more vulnerable in front of each other. We could have healthy, productive conflicts and still resolve them and still work together on the same things. This is something that was going to give us a higher quality level of idea as we were having these learnings.
However, this is really good for focusing on our individual product team. What we found is that we are also trying to validate these ideas with users, who could be people who are well outside of our company, if we're building something that an admin is not going to use, or maybe an internal team, customer support or success or product operations, something like that. Psychological safety really took care of making sure the ideas were flowing within our team. Between those teams we didn't have a good way to actually do that. We needed to make sure we were tying the dots together a little bit better than we were doing at the time. This says that even teams outside of your walls are things that you're going to make sure you need to have communication pass for.
Collective Intelligence
What we did next was focus on our collective intelligence as a useful framework for figuring out if we're going to have better idea flow, and if we're going to have more high-quality ideas. Collective intelligence is basically defined as your ability to perform a wide range of tasks as a team, and how well you do on that. It's very highly correlated with performance on complex tasks. "Team of Teams" is useful reference if you're interested in building out your team's collective intelligence, especially across an organization, super helpful as you scale.
Collective intelligence is the bar in the red. This says that it's going to predict how we do on complex tasks up to five times better than either the maximum individual intelligence of a team member or the average team member intelligence as well. When we're thinking about this, it's much more important to have a team that feels safe and comfortable than it is to have A players on our team. That's how we're going to generate better ideas, if they're more comfortable in taking those risks and having interpersonal conflicts when they know they're not going to be judged personally for those.
There are three things that we did here to help raise our collective intelligence as a team. The first was removing friction in the feedback process. We did this both with external customers and internally. The next was having informal internal teams who are really focused on collaboration and not status updates, as well as having more frequent broadcasting of updates to whoever was interested in getting that sort of information.
The first one looks like this. If we're actually broadcasting our updates, we wanted to focus on pushing status updates out, whether it's to our customers in the form of release notes or more relevant communications, emails, marketing, anything that, as well as to internal teams as well. Whatever you use at your company is probably fine, but if it's status reports, if it's something called snippets, whatever you want to do to get your information out there.
The other thing that we started doing is really having closer relationships with our specific customers. As we're thinking about this learning process, it's very helpful for us to be able to show someone a paper prototype, and then a mocked-up wireframe, and then something that might be slightly more interactive to gauge their reaction at each different level. We wanted to make sure we had very close relationships with these people. We would run a lot of programs to get them to sign on for maybe three or four weeks at a time. Where we have a weekly meeting, we can show them something a little different every time and see how they're reacting to it.
The other area is focusing more on our internal teams as well, especially when there's dependencies on the software that you're building. If you have to depend on an infrastructure team to maintain the code, if you have to rely on a separate team who's going to be an internal user of it or security or something like that, we wanted to make sure that we were having the ideas flow earlier in the process compared to things that happened after the fact. That was really where we saw we ended up getting into trouble.
We start to form these internal teams, we've called them working groups, that are really focused on a specific problem and making sure that all the voices are represented so that we don't have any surprises at the end. Similarly, how you want to integrate early and often, we were actually building functioning software. We want to make sure we're getting those voices together as early and often as possible too so that later on when we try to actually integrate the working software, we don't have those surprises.
This was the second area we focused on. Now we know that there's two different ways of thinking about the work that we're delivering: the discovery work of innovation and building new products, as well as actually building the software. In order to work in the discovery area, we're accelerating our discovery, and we're building out lots of different types of MVPs to raise the quality of those MVPs. We're making sure that our teams feel safe in taking risks, as well as building out collective intelligence with our customers and across the organization.
OKRs
The next thing is that as we were building out all of these different MVPs and as we're talking to customers and generating these ideas, we want to make sure that the team feels empowered to act on them, as well as making sure that they're all working towards the same goal. We have a lot of different ideas that are flowing right now as we have these very quick learning cycles and we're validating different hypotheses. We might end up with a team that ends up with ideas that go in different directions than what the ultimate goal of the product is. What we want to do is make sure that everyone has ideas that are very tightly aligned to what it is that we want to build in the long-term, and that they understand the why of what it is that what we're doing. There are three things that are fairly useful for this. We use OKRs at the organizational level. Below that we use product briefs to talk about the why of what it is that we're doing. Then we shifted how we do roadmaps a little bit as well.
The first is OKRs. I'm at a company that was founded by two ex-Googlers right now. We think OKRs are fairly ingrained in us. OKRs stand for objective and key results. It's a way to iterate on your planning process, where you say, "This objective is where I want to go in the future, and these key results are how I measure if I'm actually getting there in the way that I want to." We do these on an annual level for our strategic planning as well as quarterly at the tactical team level. It really helps to say, "This is what it is that we're actually doing." OKRs are something that are very difficult to get right. This objective is supposed to be something that's aspirational. The key results are supposed to be metrics space, as well as maybe a couple of delivery things in there if you have hard deadlines. There's a lot of times when people try to do OKRs, especially on a quarterly cycle, and end up struggling for a little bit.
I'll give an example of what some of our objectives were as we were trying to build out this B2B payment system in first. The first thing is that this objective says we want to beat our competitors. Obviously, the point of building the software is that we want to make it so that our product is better than them. We're going to get more money for the owners. That's great. These key results are I want to integrate with Authorize.net. I want to build out this functionality, as well as different shipping charges. I have a metric in here because I think this is going to be something that actually reduces churn. Maybe I'll explore it integrating with Bitcoin and then PayPal as well. I know these things are things that I'm going to have to do at some point. I'm going to put them in my key results list.
I think all of these are probably wrong. The problem is that the objective isn't very aspirational. It doesn't give you a good understanding of why it is that we're doing something and who it is that we want to focus on with this. The key results aren't supposed to read a roadmap of things that you do. You don't want to take the current this delivery work that you've been working on and throw it into this format, because you're not going to get the same results. What we actually settled with was something more like this. The objective is aspirational. It's that I want to accelerate cash flow for small businesses. You can clearly understand here why it is that I'm building something, and even has the target persona in here as well, the small business owner, and the reason why they're going to want to use this.
One of my key results is actually delivery-focused, integrate with two payment processors by a certain deadline. I want to make sure that I have this out in time for a key busy season for us. I want to make sure that we give ourselves that leeway. I still have a delivery-focused key result here, as well as one that says, "I want to make sure my learnings are happening. I want to conduct at least 15 customer development interviews. I want to talk to 15 different users, understand the problems faced better than I do now, and make sure that I'm building something that is going to end up being very successful at the end."
Then this metric is actually one that represents pretty well. The one before this was reduce churn, which is fairly broad. It might not be something that this team can actually impact on their own. However, this one says, "I want to process $100,000 a month in gross merchandise value." I want at least $100,000 of transactions to flow through my system using this new payment processing stuff. This gives us the outcome of what it is that would happen here.
These are a lot more triangulated to what the actual goals are of what I need. Now I can have conversations across the organization to make sure there's alignment on what it is that I'm building, as well as report back up to senior leadership level to say, "This is how we've interpreted the vision of what you've put out," which was that build a broad payment system thing. The key thing here is, for the OKR perspective, it now it helps you align your organization around what it is that teams are trying to focus on versus what it is that they're actually building.
You see the objective here at a level higher than what this team is working on is establish ourselves as the premier B2B platform for small businesses. That's not something that came across where we said, "We need to build this payment system, and we have this list of features that it needs to support." This means that the team has more understanding now of the problem that it is that we're working on. We understand who our customer is. We understand a little bit better what it is that we need to actually do to solve the objective at the strategic level. Then the senior leadership level from their perspective can easily understand what it is the team's going to be focused on and more accurately incorporate that into their strategic planning and guidance later on.
What we do is, on a quarterly basis, we'll measure how we did against these key results. This is also something that comes from Google. There's a link here for a scorecard if you're interested in trying this out for your own goals. Essentially, we want to end up somewhere between 60% and 70% of our key results being graded out as green. You see at the top, I put the objective in here. We want to accelerate cash flow for small businesses. Since my key results are very measurable, I can actually go through this list now and figure out one by one how I did against them in the quarter. I have integrated with two payment processors by August 1st. Maybe I only got one of those done by August 1st. I got 50% on that, not the best. I did, however, conduct 15 customer development interviews, and I was able to process $80,000 in GMV, not $100,000, but it shows that we're moving in the right direction.
Overall for this team, 80% means that we probably did pretty well. We probably could have stretched ourselves a little bit further, maybe at aim for a higher goal of processing GMV per month or maybe doing more customer development interviews, because we could have stretched a little bit further.
That's really what OKRs give you here. The first is as you get to focus and commit to different priorities, you get to align and connect with your teammates to make sure that you don't have a lot of dependencies across teams that may be loosely coupled, as well as making sure that your team, your internal product team is very focused on what it is that you're building. It also helps you track for accountability by using something that scorecard that you had before, as well as the stretch goals that you can put in there. It easily lets you know if you're supposed to be aiming for 60% and 70%. Are you setting goals at the right level or do you need to raise or lower the bar that you're trying to attain here? "Measure What Matters" is a good reference for this if you aren't at a company that was founded by people who worked in Google.
Briefs
Next, we have these OKRs which are very useful with communicating with our leadership team and with setting the vision and the why we're doing these things. There's always going to be that lower level of things that we need to actually build. the epics, the features, whatever you want to call them. We're still actually producing software at the end of the day, and we want to make sure that we have a way that connects the high-level objectives and the mission and the vision, all the way down to the team who's doing the implementation so that they understand how that works.
This was the mental model that I think our team brought into this. When you think about alignment and autonomy, you probably think of them as two ends of a scale, where if you have a team that is very fully aligned, they're probably doing everything very in sync, but there might be someone who's very command and control who's forcing them to do those things. If you have a lot of autonomy, you might not have a lot of alignment. What people probably do when they think about this is try to pluck down this little slider at the appropriate level for the context that they have.
What we found, especially using "Art of Action" as a reference, is that this really isn't the case. Alignment and autonomy are two different scales. We can have a very highly aligned team that has a full amount of autonomy, and we can have a team with no autonomy that has no alignment either. Examples of that, if I have high alignment and no autonomy, this is probably something that you've experienced, where someone hands you a requirements document, says, "You need to go out and build this." What you're really missing here is the why. Why does this matter? Why do our customers care about it? What are these requirements here going to actually give us?
If you have a team with a lot of autonomy but not a lot of alignment, you might get a very vague metric that you're supposed to move or a vague direction to go, but you're not going to necessarily be doing it in the right way. Someone who says, "You want to increase the payment processing on our platform for small businesses," if you don't have the appropriate context, engineers might come back and say, "Great, I want to play around with Bitcoin. I want to integrate it, see if small business owners are going to do it." It's probably not going to be a successful future. You have this high degree of autonomy, not a lot of alignment in terms of what it is that you want to do.
What we found successful in doing this is saying, "Here's the metric, here's why this matters, and let me also provide more context so that you understand why it is you need to build the things that you need to build." Having that metrics focus, but also saying, "These are the reasons why," normally results in a better idea coming from the team after they have that understanding.
The tool that we found was most effective for doing this was these product briefs. Product briefs are a written document that we have. It's normally between two and three pages long, we try to keep them very short. It spends a lot of time building up the context and walking it all the way down from the strategic level to how it is that we got to the future that we're going to build. What's not on here is any requirements or any technical architecture doc. That's something that comes after the fact. It's really something that after the team is fully aligned on what the problem is and what the hypothesis is for this feature, we can actually figure out what it is that we want to build.
I talked about the problem space. This will include things like, "Here's all the research we've done, here's my supporting metrics that I'm going to use to justify this," as well as "Here's the context, here's the elevator pitch for what it is that I'm building." The product hypothesis is something that we phrase this, "We believe that doing this thing or building this for these people is going to have this tangible impact. If it works, we're going to do this, this rollout step or amplification plan, and if it doesn't, here's how we're going to roll it back." This helps us, one, bake in that complexity mindset that we talked about before. We're recognizing that there is not a lot of certainty in what it is that we're doing until we actually get further down the line, and saying that we believe, or we think, and giving us a little bit of leeway.
We talk about the metrics, but we do this in three different levels. I'll get into more detail later on. Basically the business metrics are top, bottom-line revenue, churn, customer satisfaction, things like that. We also have product-level metrics, which I think are typically outcome metrics. How do we know that someone who is using my product is benefiting in the ways that I've expected them to benefit? Then engagement is the lowest level of metrics. It's things that happen every day.
Then the key thing for these briefs here is going to be this concept of this back brief. This is the continual cycle of updating this document and having conversations that are really focused around it to show what it is that I've learned as I'm discovering new things about our users or as I'm trying to build something. Initially, it's going to give us a lot of alignment around the problem space and what it is that we need to do to actually build something. It's going to be useful ongoing as well as we move down into the actual iterations. Are we learning something new that invalidates the research that we've done before? Then it's easy to see how this changes over time, and easy to communicate up to leadership why it is that we're adjusting the strategy that we had before.
Roadmap
Now we have this concept of alignment around our visions and our goals using OKRs, the things that roll up into the KRs, the actual features. We have the context appropriately set using these product briefs. Now we need to put it in our roadmap, which, unfortunately, mostly still looks like a Gantt chart. This is something that we've typically used. I think I've used it at every place that I've worked. It says, I'm going to deliver this feature in this month, I'm going to move on to this one, it'll be done here, on and on. This is something that I think marketing and sales typically ask for. They need a level of certainty when they're actually out there talking to people. They want to make sure we're committing to dates. However, knowing that we don't fully understand the problem until we encounter it later on, we need to bake in a little bit of uncertainty with this.
The best balance we found is using something that uses broader buckets of time in more metrics space to communicate around what it is that we're doing. Using everything that we've pulled together before, I have my objective here. It ties to the top-line strategy. I have the key metric that it is that I'm trying to move with these different features. Then I can still say these are the features that I'm going to be doing right now, as well as links back to the product brief.
The next bucket of time is going to be maybe six months out. It probably depends on how frequently it is that you're actually delivering things. Six months might not be enough for you, but this, at least, allows you to say, "There's lower certainty in these groups of things, but we still think we're going to move on to these later on." You have the ability to back out if you learn something after the fact. Then later are things that are maybe good ideas that you want to explore, but they haven't been prioritized yet. You don't need to be a parking lot for ideas. It doesn't need to be your JIRA project, where you can throw anything in and then forget about it. All these things should still have product briefs. It might be something that's not a key for the business at that time.
We're generating these better ideas, and now we need to make sure that we have the alignment across the organization to do that. We use OKRs to communicate with leadership about what it is that we're doing, as well as focusing on the key results at the team level to give more of that context for the actual things that we're building. We use these product briefs to make sure that they're appropriately tied to the OKRs, as well as aligned with the team around the metrics that we're moving, and baking in the uncertainty that they need to really capitalize on ideas as they come up. Then we've shifted our roadmap a little bit so that we're talking about these things in a different way.
3 Levels
The next thing is talking about these metrics. This is the final area that we'll be spending time on. Essentially, we mentioned these three different levels of metrics before, engagement being the things that come up the most at the bottom. These are how do I know people are using my product? These are daily active users, monthly active users, different events that happen, different milestones, things like that. These are going to be the most frequent things that occur as we're measuring how people are using our product.
At the next level up, we had the actual outcome metrics. We call them product-level metrics. This is, "How do I know my product is benefiting my users?" This is normally the key one for a team to focus on. if I build something that is supposed to be a payment system, how do I know that it's having the intended effect of helping customers get their cash back faster? Then at the top level, it's tied to top or bottom-line revenue, average contract value, lifetime value.
Correct Category
I think typically where we've been when we talk about these is that for teams that are focused on delivery and for most teams in general, you have to have some tie back to revenue or long-term value or some cash benefit to what it is that you're doing. We found that this has actually been fairly restrictive in how it is that we do things. One thing that happens when we're only looking at revenue is that it's very hard to prioritize new ideas, new areas that a team should explore. If you're only tying to things that have an impact on the top or the bottom line, you're not really going to be able to tie that to a balance sheet maybe for the foreseeable future and maybe forever.
What's really freed up the teams is to focus on something customer satisfaction or NPS. This gives you the leeway to say, "These are the biggest customer problems that our product has right now. We should really have a team focused on moving these metrics as compared to the other ones." If we were focused on revenue, again, it's going to really restrict us in what it is that we can actually do. Whereas on the customer satisfaction side, we know that we can surface problems no matter where they are, as long as they're going to be important.
The final thing I want to tie in here is now that we have these metrics, is back to the high-performing delivery team from before. Still doing the same delivery work, still delivering consistently, delivering more and more value every time, engaged in what they're doing. The other thing that we need to reference now is, are we delivering that value that we said we were going to deliver? How we do that is layering in our OKRs here. You see this is basically what we had in the spreadsheet from before. We've delivered 65%, which is right in line with what we want to do. Now we can report on the value that we're delivering as a discovery team, as well as the actual delivery portion of the process as well. We know the software is going out, now we can measure if the value is actually there too.
To sum everything up here, we have the delivery, the discovery work. Discovery work is something that fundamentally is going to be uncertain until we actually get through it. We have to get through it very fast, we have to maximize our learning. We use MVPs to do that. As we're building these MVPs, we want to make sure that they're higher quality so we make sure that the team feels safe in taking risks, and that we're aligned across the organization and with the teamwork that we have.
For the alignment, we want to make sure that we have our OKRs set up to communicate the vision and the strategy appropriately. We use the product briefs to talk about the actual tactical level of the things that we're doing. Then we shift our roadmaps a little bit as well. Then for metrics, we recognize that there are three levels and that we're not going to be able to necessarily move the revenue side of things. However, we can choose a category that gives us a lot more leeway in how we innovate by focusing on customer satisfaction or NPS or effort score, something that's reflected in how our users are feeling about our product.
This is me. This is where I've worked previously, it was at the UN. I've done government contracting. I was at a startup called Handshake here in the city that was recently acquired by Shopify. I'm now at Flatiron Health. We're going to grow to 1,000 people by the end of this year. If you're interested in working at Flatiron, feel free to click that link or let me know. Then as promised, here are the references that have helped shape this. This really, I think, is a good overview of how you can go from something high-level strategy to actually implementing this at a team level and making sure that people feel engaged and have the proper constraints in the work that they're doing.
Questions and Answers
Participant 1: Let's say you are in a setting where you join a team, they have the roadmap, their OKRs, and they recently released something. There are constant fires. You realize that the product is unstable, and also the testing suite is incomplete. How would you go about adjusting the roadmap, the OKRs? Renegotiating basically what you have committed to and try to stabilize the team, the performance and make sure that releases happen in a better and more stable manner.
Scanlon: I think the key thing for that is if you're not delivering properly now, then that's probably the area you need to focus. Your primary measure of progress is going to be working software, so making sure that your engineering practices are squared away, is probably where you need to start. If you're already causing fires consistently, it does sound you need to renegotiate the roadmap. I would use that as your leverage for getting that renegotiation done. If you're wasting a lot of time doing things fixing bugs or you have a lot of technical debt that's never been prioritized, I would try to quantify that somehow and say "Look, 50% of our time is not spent on delivery or execution, it's spent on rework." We need to make sure that we can free up the team time later on so that we can start to do one of these other two valuable things."
See more presentations with transcripts
