Mastering Observability: Unlocking Customer Insights with Gojko Adzic

Transcript

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today I have the great privilege of sitting down across many miles with Gojko-Adzic. Gojko, wonderful to see you again. Thanks for taking the time to talk to us.

Gojko-Adzic: Thank you very much.

Shane Hastie: You and I know each other and you've been fairly regularly sighted on in InfoQ, but there's probably a portion of our audience who haven't come across you before. So a good starting point is who's Gojko?

Introductions [01:15]

Gojko-Adzic: So I'm a developer. I've been developing software for as long as I know. I started copying and pasting code from German magazines on Commodore 64 without understanding a word of German to make the computer go vroom and things like that. And I've been building software for myself for the last probably 12 years now, building my own products and operating them and doing everything from pre-sales to product management to support, to development, and even kind of making coffee in the office. I'm a decent barista now.

And in my past life, I used to do a lot of contract development and helping other people improve their process. I wrote a bunch of books mostly as a way of freeing up my short-term memory. I stole that one from Henrik Kniberg. He says, whenever his short-term memory fills up, he needs to write a book. Unfortunately, it's not so easy to upgrade RAM in my head. It's more difficult to do that than in an iPhone and slightly more expensive, I guess. So I do need to write books to empty out the stuff. And yes, that's kind of taken me to write nine books so far. And mostly, as I said these days, it is just development and operating on products.

Shane Hastie: I want to dig a little bit into the books. I want to go back to one that I refer to and refer people to over and over and over again. The specification by example, making that TDD BDD absolutely real. What was the driver behind it?

Specification by Example and TDD [02:53]

Gojko-Adzic: The driver behind it is pure anger. I was sitting in the audience at the conference in 2009, I think 2008, something like that. At that point, I think as a community we figured out how to do it. It wasn't a secret anymore, and people were doing it. People at conferences were already sharing fairly good practices about how to do that. And I was in the audience listening to this nutcase who was telling everybody that nobody actually does something like that in the real world and people should be more serious about proper serious requirements and not relying on this kind of let's get people in the room to talk to each other.

And I was really angry when he said in the real world, so I thought, "I'm going to prove that yes, in the real world, these things do exist". And that's how the book came to be. I interviewed lots of people who had a good process and I was very insistent on documenting their names, their companies, getting permission to write about them. And then the book actually showed that yes, there are people, quite a few people in the real world that actually do this and run fairly big and serious projects in investment banks, in telcos, from that to startups where these things are being used. And I think that was a book born out of pure anger.

Shane Hastie: What is the big message of that? Why don't organizations buy into TDD BDD as this is just the way to do things? Why are we still having that argument today?

Gojko-Adzic: I think there's a couple of reasons. Of course there's not a single reason. One is that for lots of people today, companies just have too much money and they can afford to do the same thing over and over and over and over and over and over again. The economic software delivery when you are a contract agency or feature factory or even inside larger organizations where one part of the organization delivers the software to another for some kind of virtual dollars or whatever they're using, the incentives are just wrong because the incentives are there to kind of get people to do things with more effort. And I think this obsession with effort is really wrong in measuring effort because if you look at what's the most common metrics people use today for measuring progress, they are all related to effort. And effectively what we're measuring is how much money is being burned, not what's being actually delivered as value to the market.

And this is where I think the whole product management community that's emerging over the past 10 years... And I know that the in InfoQ audience is more kind of technical and architects and developers, but there's been a huge movement outside of that where people are actually trying to figure out how do we measure value and deliver valuable stuff and prove that we're actually delivering value to the market and then drive our development efforts with that. So I think going back to your question from one perspective, organizations have too much money and they will pay lip service to the idea that you need to deliver high quality software, but actually software is not a bottleneck for them. And the other reason why people are still not doing this that much is I think pure politics and personal incentives and people not liking to talk to other people that much because lots of people got in our industry to talk to computers not to talk to people.

And talking to people is uncomfortable and difficult because people can disagree with you and a computer can only accept or reject stuff and mostly accepts even if you tell it to do something that's completely stupid. So that's I guess another effect. And I think the third one is that because of the focus on effort, I think lots of people are either trying to keep themselves busy or being kept busy by the organizations. So pausing for half a day or pausing for a day to actually coordinate and figure out what we want to do as a team is not necessarily seen as productive and other things can be seen as more productive. Although when you look at it from a perspective or whole system, investing more effort in things that should not be done in the first place is insane. But again, that's what people tend to do.

Shane Hastie: We go back to W. Edwards Deming, "Doing well, that which should not be done at all is the highest form of waste". So as a technologist, how do I ensure that what I'm doing is adding the most value for the organization?

As a technologist, how do I ensure that what I'm doing is adding the most value for the organization? [07:34]

Gojko-Adzic: So there's a couple of interesting approaches to think about that. And one thing is to start looking at outcomes, not that the effort that we are putting out and looking at the outcomes of our work in some shape or form. It's interesting there's a few models for that. One model that Melissa Perry presented in her book, Escaping the Build Trap talks about the value exchange system or value exchange loop where effectively we as an organization deliver some value to the market and then capture some value from that delivery from the users. But that's kind of in terms of money or reputation or market share or something like that. And the users themselves, they're getting some value from the product and then trying to track that becomes really interesting. And there's been some movement last 10 years or so in larger organizations that have better ways of doing this and the results are not that good.

Ron Kohavi published a book recently called Trustworthy Online Controlled Experiments. I strongly recommend everybody read that book. Ron is famous for actually starting to track value delivered from certain initiatives at Microsoft and publishing a paper in 2009 where he showed that only one third of ideas that they tested it Microsoft to actually improve the metrics they were supposed to improve from a business perspective. That was a big wake-up call for people. Ron became kind of famous and then other people started sending him information. And he published this book recently I mentioned where he has data from Slack, for example. And at Slack, only about 30% of experiments show positive results, which means that about 70% of what they do, they're throwing away. They have data from early stages of Netflix in the book where about 90% of what people were doing was wrong. And if you look at that, these are effectively leading companies of today in terms of software delivery and their success rate is somewhere between 30 and 10%.

And when you look at it from like a technical perspective as an individual contributor, as an engineer, as an architect, and you think that, "Well, about eight out of ten things we've done last year deliver no value or even deliver negative value". It's on one hand quite a depressive thing to think about. On the other hand, it's a massive opportunity to actually improve our process and improve our systems because lots of people in large organizations are suffering from slow integration testing for example. What if you threw 80% of your stuff away? There's no reason to keep testing it. We are suffering from long maintenance and fixing things in parts of the system that maybe should not have seen the light of production ever. And in particular, if we don't spot these things early on, then you end up in this situation where you're doing rework and rework and rework.

When I was working as a contract developer for other people doing the same thing five times over and over again just because it seemed like people didn't know what they wanted was depressing. But it's not necessarily that people didn't know what they wanted, it's that they couldn't express what they wanted, what they wanted was changing and things like that. And I think being more structured about measuring value, delivering value helps engineers avoid rework. It helps us develop better products. It helps us become much more effective than the competition. And these are really interesting things. So I said there's a couple of models for doing that. One model I particularly like to think about is if we do something and the behaviour of our users doesn't change in any way, then what we've done has no effect. Just wasted time. And if we do something and the behaviour of our users changes in a negative way, if they start doing things slower, less efficiently, then we're not delivering value to the market.

We're actually damaging the value we're delivering to the market. We're disrupting our users' flows. So a really good way to think about whether we are creating value or not is how is the behavior of our users, of our stakeholders or our customers changing as a result of us shipping things out? And this kind of behavior change is interesting because it is a measurement to whether we are solving the problem, not whether we're delivering a solution. And from that perspective, I can then offer five different solutions for the same behavior change. I can say, "Well, if that's the behavior change we want to create, this thing you proposed is going to cost five men millennia to make, but I can do it with a shell script and it's going to be done tomorrow. Or we can do it with an Excel export or we can do it with a PDF or we can do it through a mobile website not building a completely new app". And all of these things can address the same behavior change.

And the other thing why behavior changes are really interesting model for value delivered to the market is that we can typically observe and measure behavior changes very quickly after something is delivered. Things like profit, market share, revenue destroying competition and things like that, they tend to come on a very long delayed cycle where by the time we can measure these things, it's already too late to adjust course. It's already too late to say, "Well, six months ago we should have done something differently". But behavior changes in the terminology of what product people are talking about are leading indicators. They're not telling us whether we're winning. They're telling us if we're likely to win. And the nice thing about those is we can actually measure them quite quickly. And this is where again, technical people jump in because the way we can measure behavior changes is very similar to what we've learned to do for measuring the behavior of assistance.

Over the last 10 years, especially due to DevOps and things like that, observability became a big thing. Tracing distributed measurements, combining logs, error measurements and things like that, they've become the norm. That's kind of your entry level ticket to a good system now. And we can extend all these things to measure the behavior of our users. We can then figure out if what we've done is actually helping people do something better, faster, easier, sooner, or if we are actually making it slower, if what we are doing is causing confusion and delaying people from achieving what they want. And I'll give you a perfectly sane example from one of the products I was working on. I'm using an American payment provider for that. And being an American payment provider, they don't really understand European politics. Nobody understands European politics, even Europeans. But if you're selling stuff to individuals in the European Union, you have to charge them some tax based on what their country decided.

If you're selling to companies, you shouldn't charge them the tax, but you need to ask them for their tax ID. And if they enter a tax ID, then they're doing reverse charging and it's complicated, but... So there's a case where you need to ask people for a tax ID and being European Union, they have some bureaucracy over this. So most countries have a prefix with their tax ID, but if you're selling side the country, people don't put the prefix. If you're selling internationally, people put the prefix. Now is your buyer responsible for knowing all of that? Absolutely not. And the payment form I had lets people enter a tax ID, the payment provider validates it. Lots of people were getting stuck there. They were trying to enter a tax ID without a prefix, not knowing that they should enter a prefix. And then the form complains that it's an invalid number, then they think it's an invalid card number.

So they struggle to do that. Then they send me an email, then there's a non-trivial group of people who use weird countries just in order to enter their tax ID so they can pass through. And I thought, "Well, it's a problem. It's creating obstacles for people to pay me, which is not good, and I can't really do much about the validation on the form, but I can remove fields from the form". So I thought the smart thing to do was streamline the form, just remove any billing information from there, remove the tax ID. And once people pay, I can get them on my website and ask them, "Enter a tax ID, enter whatever you do, and I can do validation". So that to me seemed like it's a really good idea.

I released that with metrics and it turns out that I was causing an even bigger problem because now people were getting to the form expecting to be able to enter a tax ID and not being able to do that, they were scared that they're not going to get the kind of money process by the accounting. So I thought it was a good idea, but it wasn't. And by looking at measuring the behavior change, are more people completing the payment process more successfully, I was able to prove that my idea was bad and I did something else. And from that perspective, if I just said, "Oh, I need to remove this from the form and trusted my intuition, trusted my opinion", I would've probably lost a lot of money. But measuring the behavior changes at the end showed me that, "Well, yes, bad idea. Too bad. Let's do something else".

Shane Hastie: What's a lizard?

Lizard Optimization [17:05]

Gojko-Adzic: What's a lizard? Ooh. So I guess we're talking about my most recent book that's called Lizard Optimization. A lizard is somebody who does stuff that no rational human would do, and you cannot comprehend why they're doing that, but they're still doing it. And the metaphor comes from Scott Alexander who wrote this wonderful blog post about phantom lizard men where he looked at some demographic studies they were doing, they were combining demographic studies and psychological research. So the idea was that first couple of questions would classify you whether you're Australia and British, German, whatever, how old you are. And then they would ask you a bunch of psychological questions. And then he realized that is people... Like one guy selected American as gender, and yes, gender politics are difficult these days. It's all a minefield. But then he selected male as nationality and people were entering all sorts of weird stuff there.

And Scott said that about 4% of the people in the survey did things they just couldn't explain. And then he compared that with some other research that said that 4% of the people believe that lizard men are running the world. And he started calling these people lizard man. And the idea is basically that some percentage of your users will be confused, distracted. They will sit on the beach on the bright sunlight and not see the screen. They will work in the dark where the children are sleeping next to them and not be able to figure out what's going on. They will be visually impaired, they will be confused and a bunch of other things, and then they will do things that you can't place on a rational human to do when you look at the data.

For example, one of my products helps people convert PowerPoint to video. And some percentage of people was paying me to convert blank PowerPoints to videos, which makes absolutely no sense for a rational human to do. Why would anybody do that? So that's a lizard because we've just discovered something that makes no sense, but it might make sense if you try to understand the lizard logic. It's not human logic. And then I tried to figure out why are these people doing this and started interviewing them. And it turns out that by that point, the product was so nice in doing text to speech conversion that people were creating blank PowerPoints just to be able to use the text to speech function to extract an audio track. They didn't actually need the video, they just needed an audio track.

And then I realized there's a product opportunity here because making a video costs me operationally much more than making an audio file. It slows down the process for the user that wants to do that. So I created a very simple screen where people can just upload the Word document and get an audio track rather than having to create blank videos and things like that. And that actually increased the usage of the product quite significantly because when that was easy, lots of other people started doing that, and those lizards told their lizard friends and lizard friends started coming. So the whole idea is kind of by making the product better for people who are struggling in some strange way, we can actually improve it for everybody and we can systematically make our products better in that way. And that's kind of the idea of lizard optimization.

Shane Hastie: But to do that, and you were talking about it earlier, you've got to have the analytics built in, you've got to have the observability, you've got to be able to see what people are doing with your products.

Analytics and observability into user behaviour [20:39]

Gojko-Adzic: Absolutely. I think this is the massive opportunity for technical people in our industry. This is a huge opportunity for DevOps people, for the engineers, for developers, for architects, where we can extend the tools that we already have and then figure out what's the unexpected that the users are doing. If you look at, just like most of the analytics, we're talking about analytics, most of the analytics that product people are using now, they are looking for expected things. They're looking for clicks, they're looking for conversions, they're looking for purchases, they're looking for page views. That's all kind of the stuff that's expected. DevOps and observability, there's a lot of stuff there around exception tracking, looking at mismatches, looking at kind of deviations, looking at what's unexpected because we've learned if you have a hundred virtual machines running somewhere in the cloud and one of them starts coughing, we need to know about that.

And we can use these tools to then extend it for user behavior tracking and for tracking things the product people care about and spot all those kind of weird things that are happening and then effectively extract the signal from the noise. And I think that's a wonderful opportunity for people to do. One of the most trivial things everybody can do now is just logging whenever a user sees error screen. When a user does something that causes an unexpected path in the application, a good application is going to tell the user, "Well, sorry, you can't do this". But we have as an industry learned how to do crash analytics. If an app crashes, we send it back to ourselves. This is not the crash. It's not tragic. It's something that we kind of expected the user to do, but the user is doing it in a strange way.

Most of applications that people work on today ignore that level of logging because we've told the user, "Sorry". One example, again, going back to the screen where people can upload word documents. If somebody selects a file of an unknown type or a type I can't support, of course the software shows them an error saying, "Look, you tried to select the JPEG file. It doesn't contain text. I can't read what's in your holiday photo". But the software sends me the fact that somebody tried to convert a JPEG file so I can figure out what's going on. Now a significant number of people every day select APK files under each packages to load into a text-to-speech system, which is bizarre. Lizards. I have no way of explaining what happens there. My best guess is that somebody's trying to do that trick when they leave a USB stick outside of the building and figure out if somebody is going to plug it in.

A reasonable group of people every day try to upload MP3 files into a text-to-speech system, which is... I don't know, maybe independent singer-songwriters trying to get their music out there or something like that. But occasionally people do something that actually makes sense to support. So for example, some percentage of people a while ago were trying to upload subtitle files, and I was thinking like, "Well, a subtitle file is a text file. It's a closed captions file. We should be able to read this. Why not? It's a wrong extension, but why not read it?" And then I allowed that to happen, and then people started complaining that it's reading the timestamps from the file. And I replied, "Well, yes, the file has timestamps. You've uploaded the file with timestamps. It's reading the contents of the file. That's what a system does". They said, "No, but we don't want it to read the timestamps from the file".

So I said, "Okay, well, the meat of the file is in the text, not the timestamps". So I removed the timestamps and then people complained that, "Well, now it's speaking too slow or too fast". It's like, "Oh, so what they actually wanted to do was create an alternate audio track for a video they already had". And it took us a couple of cycles of feedback like this. But because again, it's quick feedback, we are talking about something that took a couple of days with a couple of people, not something that millions of users would experience as a problem. And as a result of that, I actually figured, "Well, why not build a synchronized audio track when somebody uploads a subtitle file?" 99% of the functionality was already there. This was just helping that particular group of lizards a bit more. That was the most profitable thing I've ever done in my life.

Because couple of weeks later, a massive, massive, massive enterprise software company started using the system to convert their instructional videos and they had hundreds of thousands of videos. They wanted to do it in dozens of languages. And the fact that my system could create an alternate audio track that synchronized was saving their content editors hours and hours and hours of time. And they were happy to pay some fraction of that in exchange to do this faster. And this directly came as a result of monitoring for user errors that the application wasn't crashing. It was behaving very decently, but it was also sending me information that I could use product management signals. And I think that's the opportunity for developers and operations people today is being able to spot and provide these signals so that we can make better products.

Shane Hastie: So digging into some of the concepts in the book, one of the things that struck me is your five stages of growth. Again, I want to relate this to me as a technologist. What do I need to be doing at each stage?

The five stages of growth [26:14]

Gojko-Adzic: The five stages of growth are a concept that was published by Alistair Croll and Benjamin Yoskovitz in Lean Analytics, which is a wonderful book about software analytics. I think people that want to look at measuring the business side of their software should read that book and they talk about how products typically evolve and what people should be focusing on, and the stages are effectively thresholds for what you need to prove to move on to the next one. The first stage is called empathy. And that stage is really about do we have a good problem that we're solving and looking at the different types of problems and trying to figure out are we solving this problem in a good way for somebody that's important? The next stage they call stickiness, and that's about proving that potentially there is a market for this product.

We have a product because we are solving a problem. Is there a market for this? And what they talk about there is that if we can get people to use this very frequently, if there's a lot of urgency usage, if people are staying, if the product is sticky, then potentially there is a market for it. If people just try the stuff we do and then leave immediately and never come back, then there's not much point developing it further. And usually in that stage, people iterate around different user groups. And one good example of that would be what I mentioned earlier where we have people building videos. They were building videos but not that many. But then when they were able to build audio files directly, the usage significantly increased and people are building a lot more audio files than video files. So we kind of repositioned the product really to look at the audio more, and that's kind of the part that is usually called the product market fit.

We found the market when people are using it a lot. The next stage they call virality, I like to call it growth because it's not just about virality and it's about, "Okay, now we have a market, we need to prove that we can actually reach that market because as a product we might not be able to go to our target customers and advertise to them or reach them by word of mouth or whatever the growth engine is". Then the next stage is they call it revenue where we need to prove that we can build a sustainable business model from that. Lots of people are okay maybe losing money early on to prove that they can build stuff, but then we need to optimize how we're making money. And then the last stage is scale. That's consolidating business operations, building optimizing profit rather than just revenue and figuring out how to capture more value from the market.

So in the value exchange loop, I mentioned where we have value delivered to the market and their value captured from the market, the first two stages are really value delivered to the market. The third stage is reaching that market, and then the last two are capturing the value from the market. And what we can do as engineers there is use that as a good argument when certain things should be done. As engineers, I think stuff in the last stage is usually under our control, and people tend to then focus on doing that too early. We tend to focus on optimizing for performance too early or figuring out how to scale things too early or doing things that would... Well, once the product is profitable and it's reached to market and it's growing and things like that, this is going to be amazing.

But doing that before the product is actually on the market and profitable and things like that is... Optimization is too early. Now I'm not suggesting we completely ignore it, but not focus too much on it, not spend most of our time doing that. As a CTO of a company about what, 16 years ago or something like that, I did not have that tool in my head, and we messed up really badly because of that. I wish I knew that because we've kind of done the first stage. We've proven that we have a product. We've not done the second stage, we've not done the sticky part of it, but the marketing people wanted to jump into marketing. Now we have a product. Now we've proven that it's a good product, but we've not optimized any of the stickiness. So I don't remember the numbers anymore. It's been a while ago, so I'll just invent some numbers now.

Bear with me. It's kind of the right order of magnitude. Let's say that out of every thousand people that came to the website, one or two actually came back the next day and we invested... Our entire marketing budget is a startup into doing this. But by bringing say a million people to the website, we got 2000 or 3000 of them to actually stay. Where they dropped off very quickly after that because the product wasn't sticky. Had we had done the stickiness phase first and then made the product sticky, made the website better, allowed people to start using the product with fewer rough edges and kept using it and then bring a million people to the product, then the outcome would be completely different. And as a result of doing this in the wrong order, we kind of spent our entire marketing budget and the company wasn't doing so well after that.

Although we did have quite decent technology. I think still to this day, that's probably the best technical team I've ever worked with. And I think that's the idea of the five stages of growth is to figure out when to focus on what and when to stop focusing on some things and move on to some other things and then use that as metrics around the product. I particularly like to use one stage to optimize and the other stage is to create guardrail metrics. So we might want to say, "Look, now we want to focus on growth, but we don't want our operational costs to grow out of proportion. So yes, we might need to optimize some databases or we might need to break out this monolith into microservices or something like that because the operational costs are going to be too much for growth, but we're not doing that to actually just optimize the operations costs. We're doing that just to support growth, just enough of scaling or just enough of this".

Or we can say, "Look, now in the scaling phase, we want to rip out this database, we want to replace it with some smaller databases. We want to do lots of other things, but we don't want to damage stickiness". If we do that in a way that people start dropping off, then we're doing something wrong. And that's a good way of actually figuring out the guardrail metrics, so what makes a viable solution for our things. And again, doing that from a perspective that's not necessarily measuring whether we are delivering a solution, but whether we're solving a problem. We're measuring, again, behavior changes, we're measuring value, not the product. I mean, if you look at this purely from a very, very old school gray beard technology perspective, Michael Jackson, not the singer of course, but the architect who wrote books in the nineties wrote a book about problem frames.

I remember reading about problem frames in the nineties. That was a massively influential book on me. And he thought about this as you have this kind of problem frame and you have a solution frame, and then there's kind of ways of information exchange between them and things like that. And we as software people, our world is in the solution frame, but the solution frame exists there to support the problem frame, not on its own. And by looking at what's happening in the problem frame, we can actually evaluate whether what we are doing in the solution frame is sensible or not.

Shane Hastie: Gojko, as always, really interesting conversations. Some fantastic ideas in there. If people want to get hold of you and continue the conversation, where do they find you?

Gojko-Adzic: I'm fairly easy to find because my name is unusual. Just kind of Google my name. What's going to come up first is this footballer who is sleeping with supermodels. That's not me. I'm the next one. So I mean, they can try contacting him as well. I doubt they'll receive a response. But yes, I have an archaic unusual name even for Serbia. So just Google my name and you'll find me easily.

Shane Hastie: Wonderful, as always, great to talk to you. Thank you so much for taking the time to talk to us today.

Gojko-Adzic: Cheers.

Mentioned:

• Gojko Adzic on LinkedIn
• Book: Trustworthy Online Controlled Experiments
• Book: Lean Analytics