Scott Ambler on Disciplined Agile and Agile beyond Software Teams

Show Notes

• 00:21 Shane:  Good day folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. I'm at Agile 2019 and sitting down with Scott Ambler.  Scott is the co-founder of Disciplined Agile, and has been doing this stuff for a fair while now,
• 00:36 Scott: almost 20 years. Yeah.
• 00:37 Shane: Scott, welcome. Thanks for taking the time to talk to us today.
• 00:40 Scott: Oh my pleasure. Thank you.
• 00:41 Shane: Some of our audience probably haven't come across Disciplined Agile. So do you want to give us a very quick overview and background? Why is it disciplined agile?
• 00:51 Scott: Very good question, which we get asked all the time.  Disciplined agile is a toolkit, and in the toolkit, we look at four levels.
• 00:58 We started originally with disciplined agile delivery, which people might've heard of. We were asked to answer the question, how do you do this agile software development stuff from beginning to end and all aspects of it, throwing in all this evil architecture and all this other stuff, too.
• 01:10 We then expanded over the years into DevOps, asking the question, and we're always in the environment of an enterprise with legacy systems and legacy people. And they've been around for decades, if not centuries. so it's not a fresh start that a lot of people like to talk about, and frankly, even in DAD, all the hooks for DevOps were there, but we expanded into what we call disciplined DevOps, where we answer the question. How do you do DevOps in an enterprise-class setting? Where you're bringing security into play, you're bringing the database stuff into play, you're bringing true release management to play. , I'm a firm believer in, you build it, you run it type philosophy.
• 01:45 But when we have a thousand development teams or hundreds of development teams in flight, which some of our customers do. And the last thing I want is hundreds of development teams just slamming stuff into production whenever they want to. There should be a little bit of coordination and hopefully automated as much as possible, but still, somebody's got to think this through.
• 02:01 And I certainly don't want hundreds of teams figuring out CI/CD for themselves. So, , there's a bunch of enterprise issues that come into play.
• 02:08 The next level, which the DevOps community is starting to focus on now, which is great, is disciplined agile IT, so how do we bring agile into the IT space in all aspects of IT?
• 02:18 How do you do enterprise architecture? How do you do portfolio management across all of it, not just the software development stuff, which we all like, but how do you actually do portfolio management when you've got existing systems and products, sometimes hundreds or thousands that you got to keep the lights on.
• 02:32 How do we do operations and infrastructure? When we have at least hundreds, if not thousands of systems? Because once again, I don't want each individual team to do that, it's phenomenally wasteful and dangerous.
• 02:42 And then finally, how do we do business agility? So how do we have an enterprise, a disciplined agile enterprise where we look at some of the  not so sexy aspects, certainly some of the interesting aspects of business agility, but also things like legal and procurement and HR,  well people, people management, like we don't like the word resource,  and all-important and all interesting in their own right?
• 03:01 But how do they all fit together as well? Which becomes an interesting question. , because your organization is a complex adaptive system, how do you deal with all those good issues?
• 03:08 The toolkit. It is all about, first of all, what's the full picture and how does it all fit together, but also what are your choices? What are your options? Because one size does not fit all.
• 03:17 So you might notice I've been using the word toolkit, not the word framework because even though we started calling ourselves a framework years ago, the challenge we found and framework is an okay word, I guess. But the challenge we found is disciplined agile is very different from all these other methods and frameworks,  where we are truly non-prescriptive, we are truly agnostic and pragmatic and where the other ones tend to have good marketing around those words when you only have one way of doing things, that's the epitome of prescription.
• 03:48 So we're at the other extreme we've multiple ways of doing everything and we give you choices and the toolkit helps you to make better choices and to actually do process improvement over time and more of a lean Kanban, Kaizen type of approach to process improvement.
• 04:01 Shane: 04:01 So that's the big picture. That's what disciplined agile is, and it sounds like you're now currently addressing that overall organizational stuff, but what's new and what's happening with discipline agile. How's it being received? What's the adoption like?
• 04:16 Scott: We're actually doing very well. We don't market at all for the most part, other than, you know, things like this.
• 04:21 So we've been out marketed by pretty much everybody else, which is fine, but we've had phenomenal grassroots adoption. According to Gartner. We're now the number two framework toolkit out there. We do have this problem that we are still being lumped into these frameworks and to be fair, they've got their quadrants and they got to do what they got to do, but we're still being lumped in with that.
• 04:39 According to the five-minute VersionOne survey, we're also number two, that's more of a popularity contest. I trust Gartner more than I trust the other stuff. But anyway, we're coming in number two and what's interesting is we have the highest, I can't remember the exact term for it, but it's basically, they've got a term for saying when people actually look at multiple options and think, and look for something that's fit for purpose, then we have the highest adoption rate amongst that category, which is good.
• 05:07 Because what happens is our message is fundamentally context counts and because everybody's in a different situation because your organization is very different than somebody else's organization and your team is very different, , forget about the organization,  your team is very different.
• 05:20 You need to have a way of working that is right for you. So how do you do that? The entire toolkit focuses on helping you to understand, instead of telling you what to do, we say, here's what you need to think about, here are your options, here are some good options.  I'm not saying we have all the options, but we're certainly saying we've got some pretty good ones and it's evolving all the time.
• 05:39 And here are the trade-offs for those options. So now you can start making intelligent decisions around does this practice, does this strategy make sense for the decision that we face and it based on our skillset, based on the culture, based on what we're allowed to do and based on what we're capable of doing right now, what's the best way.
• 05:55 Our message is one of start where you are,  we're not big fans of this burn the earth and start from scratch stuff because that's a very unpalatable and very risky thing to do, but instead, let's start where you are and then let's improve and let's get better. So do the best we can in the situation that you face and then improve from there.
• 06:13 Always try to learn, always try to get better.
• 06:15 Some of the new stuff that we've been doing of late in the DA space is something called guided continuous improvement, and this is what's captured in our new book Choose your WOW, where the basic message is around how do you do process improvement? A Kaizen manner in small changes.
• 06:30 And the message in the agile community for years has been teams and organizations should own their process.  You should own your own process and you should do whatever you need to do, and then improve and learn and all that sort of stuff and the smarter teams should be doing this inspect and adapt, PDCA, PDSA, whichever terminology you like:  we should try something, see if it works for us, adopt the stuff that works well, abandon the stuff that doesn't work so well, and if you're polite, you share with others because why wouldn't you do that? And then, , rinse and repeat.
• 06:57 Wonderful ideas. The problem, in reality, is that people don't process experts. So telling people to own your own process when you don't even know what's for sale, not very useful.
• 07:07 Earlier today I was with an organization giving the intro to agile stuff and it was wonderful and sort of sharing some basic concepts from the agile world that have been around for a long time and had some very weird conversations around, "people are actually doing this? This is so weird. How could this possibly work? And it sounds so much riskier. And what do you mean you were not planning for a year in advance? "and, classic conversations ...
• 07:30 Shane: We are in Washington, DC.
• 07:32 Scott: We are in Washington, DC so, it is what it is.
• 07:35 We're working through all this, and then I get into the PDSA stuff and the guided continuous improvement. And then finally, how do you actually improve this? So I share some stats from Don Riefer about, here's the actual effectiveness and practice of all these methods and frameworks and all these prescriptive things, and the reality and the marketing are very different.  Yes, people are getting some benefit from Scrum and yes, we're getting some benefit from SAFe, but nowhere near what gets promised and nowhere near what you would actually want. So we're still getting benefits from it. So that's okay. But you hit the limit as Ivar Jacobsen likes to talk about you're in method prison.
• 08:05 So you hit the limit of whatever problem Scrum solves, whatever problem SAFe solves for you. Once you solve it, you hit this limit and then suddenly you've got to figure it out. So how do you figure it on your own when you're not a process expert?
• 08:16 What we do in disciplined agile is we say, well, Hey, here are these other techniques to solve the same sort of problem, and then here's the tradeoffs associated with that. So why don't you pick one and experiment with and see if it actually works for you? If you read some of the case studies of like the Amazons, the Googles and the eBays of the world, they've been doing this for years, they've been doing this continual process improvement, Kaizen inspects and adapt stuff. Very smart, and this is why they're effectively the apex predators now in the business space is because they got really good and they're constantly getting better.
• 08:45 When I work with executives, I ask the question, who do you fear? What companies are you afraid of? when I work with banks, it's been several years now since an executive at a bank has told me that they're afraid of other banks, all of them tell me we're afraid of Apple pay, we're afraid of Google. We're afraid of Amazon. None of them are afraid of traditional competitors. And it's because they're apex predators. They know that they can't compete on the software side of things with these other companies. So my advice to companies is, well, why don't you look at what those companies are actually doing?
• 09:18 Not exactly what methods they're following right now, because more than likely you're not capable of doing that, but how did they get there? Do you honestly think any of them adopted SAFe? No, none of them did. And they would laugh at you. So instead, what did they do? Well, they did inspect and adapt, they did this PDCA PDSA thing, Kaizen So you should do that. But how the heck do you catch up? So the challenge that Amazon and eBay that they have they're leaders and they're at the leading edge. So they've got to really figure out what's the next step to do this improvement. But if your organization is 10, 15 years behind Amazon, which you more than likely are, if that, you don't need to reinvent the wheel, you don't need to do it as slowly as Amazon did or eBay, or whatever company you like, because thousands of companies have figured these problems out so we can leverage that.
• 10:02 And that's what the toolkit is all about, it's here are these great practices - you learn from that.
• 10:07 So now what the idea which we've been doing with customers is, we can increase the chance of you identifying a technique that will work for you and your situation. So the side effect of that is you have a percentage of successful experiments and as a result, you improve faster.  Will you ever catch up to Amazon? Probably not, but you can close the gap and you can close the gap enough to stay alive, basically.
• 10:31 So this has fundamentally been our focus lately in disciplined agile is how do we actually give companies, the toolkit that they need to actually do this process improvement stuff, but it still requires skill. It requires thinking and the willingness to experiment.  You'll still guess wrong, right - you'll guess wrong less often.
• 10:49 Shane: Sounds to me that what you're saying is at the team level, we're doing lots and lots of experimentation about the way we're working because we're trying to learn all the time. How do I get consistency?
• 11:00 Scott: Oh, well, we don't worry about consistency. Yeah. I know. Look of shock on your face.
• 11:03 That's a sucker's bet in my mind.  Because you won't. So let's backtrack that. So say we're in this fantasy world we're Harry Potter,  so we're magic and we've got our magic wand and I can wave my magic wand and all my teams are following the exact same process. we're in bureaucratic Fantasyland on this one.
• 11:18 So all the teams are following the same process. But they're agile. A fundamental of agile is you learn, you inspect, you adapt and you get better. So even if everybody starts with the exact same process, a few months later, you're starting to do different things. Certainly a year or two later, all these teams have diverged wildly, because of different people, different situations, different preferences, all good stuff.
• 11:38 So accept that, this is reality, I'm a firm believer in observation and accepting reality. Whether you like reality or not, that's what it is.
• 11:48 But now from an organizational point of view.; So now all the managers are freaking out, heads exploding. Agile is evil, right? From an organizational point of view, what we do in disciplined agile, we say, you know what, allow your teams to work the way they want to work, allow them to choose their own WOW. Their own way of working because they will anyway, regardless of what you tell them. And instead, govern them consistently and govern them well.
• 12:07 Most organizations really struggle with governance is all reviews, and documents, and all this other stuff, which doesn't work at all. So instead we focus on here's how you govern, even though these teams are working in very different ways, you can still have a little bit of consistent governance based on risk reduction, as opposed to , policing them and all that sort of stuff. Our philosophy around governance is good governance is all about motivation, enablement and monitoring and transparency, of course.
• 12:34 One of the things we do in disciplined agile is we extend the agile manifesto to address transparency, this is absolutely critical for your success, and frankly, transparency has been there in agile for years anyway, so let's just codify it. So it's this one of our values. .
• 12:48 We added a fifth value, transparency over false predictability, which Ooh, yes, . That gets a lot of attention from the traditional management crowd for some reason. And then it gets very interesting conversations going, which is critical. So we address that problem and you have to, because if you don't, it's all gonna fall apart on you. And I think this is where a lot of organizations struggle right now, certainly around governance because the old governance strategies just reflect traditional values to be fair, and we're not working in that way. So you want to be agile, you've gotta have lean, got to have agile governance.
• 13:16 Shane: What does agile governance look like for a bank? 
• 13:20 Scott: Yes. Banks are a perfect example. So banks have regulatory issues, financial regulatory, which is nowhere near as hard as life regulatory sets. Not that it really matters,  it's the same strategy.
• 13:29 we've got to respect whatever legal restrictions we've got. I'm also a firm believer in reading the regulations because, if you want to be regulatory compliant, well, you gotta understand the regulations are to begin with. So my advice is always to have practical people interpret the regulations as opposed to bureaucratic people.
• 13:47 Cause the bureaucrats are going to come up with a bureaucratic response to the regulations, which more than likely is what you currently have. So we work in financial institutions all the time because they've got money, , so that's a good customer and they've got a lot of problems to solve.
• 13:59 So we're constantly,  saying you know what, let's go back and let's actually read what the regulations actually say, not what you're currently doing, but let's read what the regulation actually says, because I bet you, it doesn't say what you're doing at all. And then, how can we automate? A lot of the regulations,  particularly in the financial world, that stuff can be automated.
• 14:15 what's the underlying message they're sending with the regulation. Well, you know what? We can implement that in a different way. Once again, it gets back to choice, right? If you only know about the bureaucratic onerous choices, Then that's what you're going to do. If you realize, for example, in the financial world, at least in the US there's the PCI DSS regulations for around financial transactions,  making them secure.  You can automate the living heck out of that.
• 14:37 So one aspect of PCI is the separation of concerns.   Your separation of duties,  so the people doing development are not allowed to make the decision to deploy to production for obvious fraud issues. But it doesn't say that the decision to deploy has to be done by another person, now that is almost always the way it gets interpreted, but you can actually automate that decision because it's been codified and logged and all that sort of stuff, it's compliant to have continuous delivery, or you can have continuous integration, continuous deployment, and automate the living heck out of all that.
• 15:09 And so you don't have to have this onerous process involving people to do reviews and make a decision to release into production that  might take days or weeks. You can actually now do this in minutes through better automation
• 15:19 Shane: And the automation would do the check.
• 15:21 Scott: basically the logic is, because you get the developer coding, doing whatever it is that they're doing, but the decision to deploy has been codified into scripts.
• 15:31 So it's effectively out of the hands of the developer and all this is being logged and tested and all sorts of stuff.  And the beautiful thing about all this is, this is more compliant than the physical manual stuff they were doing in the past.
• 15:44 And the auditors love the automated stuff because they can actually see it run, they know it's very difficult to scam, whereas humans in the decision loop. Yeah. Who knows what's really going on there?  It's not as trustworthy. So this is sort of thinking we've got to start having. , We do lot of work around governance, for the very reason why you asked this question is because the managers or the management crowd of the world get totally freaked out when you start talking about this and rightfully so.
• 16:08 Shane: Yes, they have fiduciary responsibilities and can be held to account.  earlier on you touched on the overall business and bringing agile there, we hear a lot, the buzzword at the moment is business agility. What does that actually mean to us?
• 16:25 Scott: That's a very good question. So, In my mind business agility is around the issue of is my business customer-focused. Are we taking the needs and the wants and the desires and the customer in mind? Are we actively trying to delight them? Are we able to react to the marketplace and react to opportunities, to threats as well? Are we working collaboratively? Are we working well with our suppliers?
• 16:47 At, DA we've been long believers in focusing on stakeholders, not just customers while clearly customers are important.  It's really all about the entire business being agile.
• 16:56 I think a lot of these agile transformations run aground because when you only focus on software development, it tends to fall apart on you.
• 17:04 Software development is obviously important, but the challenge is we have this myth., It's an absolute myth, around whole teams and it's just observable nonsense. I love it. I think it's a great idea. But then you start asking questions like, Oh your team's whole, so you have funding, you have funding mechanisms that you can decide what to fund on your team.
• 17:21 Oh no, no. We've got to go to the people with money over here to jump through hoops, to get funding. Okay. She no longer whole, cause you don't have the resources to do the job cause you don't have the money. Which is an absolutely critical thing. And what about regulatory guy? What about security? Oh, we've got to work with the security team to make sure everything's secure.  Oh, so you're no longer whole there. .
• 17:38 As you know, I do research and running a lot of surveys and stuff like that. And a few years ago, I ran a survey specifically around that, where I asked a series of questions and some of it was noise to sort of hiding what I was actually going for. But what I discovered was that 97% of agile teams were not whole. 97%. And yet we're constantly still talking about this and we will, decades from now, we'll still be talking about this,  cause it is a great idea. 
• 18:03 An implication of not being whole is that if I have to go to somebody with money, if whatever my finance mechanism is, if they're not working in an agile manner, well, that's like throwing an anchor around my neck. If I've got to go to the release folks to get something out the door and then it takes three weeks to release something, Oh, I'm no longer agile. 
• 18:19 So this is the challenge. So when we start doing these transformations, we want to focus on the software development teams, which is great, but any other team that my team interacts with, then they either have to start getting on the agile path themselves and I probably have to help them do that, which is great. Or I have to get an organizational pass from them and they have to be willing to say, you know what? It's not our time to learn this agile stuff, which is great, but we're going to get out of your way, you know, as long as you've got a story to tell where you're going to be responsible for doing this work, that we would have done, fine.
• 18:48 Now, the organizational pass is virtually impossible to get in most situations. So I would much rather help them become agile because that's really the long term solution anyway. And so I think what's happening in the industry is one of the reasons why we're focusing on business agility, well, some of it is to stay alive, your business better be agile or I think you're cooked, but also it's because even though we've started focusing on software development, we instantaneously, we run into these organizational challenges and years ago, our reaction was to blame the management, there's an easy target, which is completely not fair because the managers are smart. You might not agree with them, but they're smart people. They're doing the best that they can. And they've never been helped. If we don't reach out and help the managers to become more agile or the other non-managers as well, then we shouldn't beat them up for not being agile.
• 19:31 Let's get them to help they need and start addressing the problems. Like some people will say, well, this part of the organization is hosed and we can't help them because it's hard. It's hard, I'm lazy. I'm not going to do it well that doesn't get the job done. Cause, if they're still throwing an anchor on my neck, I'm in serious trouble here.
• 19:44 So we've got to step up and do this. And so I think some of the businesses utility movement is just this reality of the entire business really does need to become agile just for survival, but as well if we're trying to be agile in software development, the analogy that we've used for the last few years is actually around a racing car engine.  The original agile movement being around software development, we really got good at building racing car engines.  We're good at tuning these teams, getting better productivity out of them, a better quality of them. We really are good at that, but then we tend to take these great racing car engines and we plunked them into an organizational tractor, and it's a big surprise to us that we're not winning the race. Well, I've just got a tractor with a cool engine. It doesn't really pass the who cares test. 
• 20:22 if you want to win the race, you need a racing car. That's what DevOps is all about, but having a racing car doesn't get the job done either.
• 20:28 I need a driver, I need a pit crew and that's what disciplined agile IT is all about, but that still is not good enough.  That just means I've got a group of people with a cool car. I actually need a race, I need a Formula One or NASCAR or something. I need a race that I can take my really cool car and team and maybe make some money racing my car.
• 20:45 And this is what an agile business is all about. This is the Formula One organization, basically. So this I think is what we're getting at,  in some ways we're getting at business agility bottom-up, but also I think we're getting it top-down because executives are realizing, wow, you know, we've got to compete now against these apex predators and it's not going to go well for us.
• 21:03 Shane: Scott. Thanks very much for taking the time to talk to us today.
• 21:06 Scott: Oh my pleasure.
• 21:07 Shane: If people want to continue the conversation, where do they find you? 
• 21:09 Scott: A quick Google or Bing search will get the job done, but you go by https://www.pmi.org/disciplined-agile is probably a good source to start. You can reach out to me on Twitter, @ScottwAmbler, or on LinkedIn as well.
• 21:20 I'm easy to find. Just do a quick search.
• 21:22 Shane: Thanks very much.
• 21:23 Scott: My pleasure. Thank you. 

Mentioned

• Agile 2019
• Disciplined Agile 
• Book: Choose your WOW
• Ivar Jacobsen – Method Prisons
• Scott on LinkedIn and Twitter