1. Scott, let’s just start with the introductions; can you tell us who is Scott Ambler?
Yes; so I guess I’m a process improvement consultant; I’ve been helping organizations around the world to figure out, you know, how it is you do this software development stuff and both at the project level as well as at the enterprise level so; and the last 10-15 years I’ve been focused mostly on Agile but Agile is not the only game in town and you need to look at bigger picture because there are some good reasons to do more traditional stuff; there’s some very good reasons to do some Lean stuff so; I mean you need to look at the full range of options which I’m a firm believer in.
Yes; so as the chief methodologist for IT at IBM Rational, the Rational division at IBM and I did that for six years; I was in various roles over the years but focused mostly on process stuff over the years and helping organizations do all this good stuff; but yes, I recently left and now I’m starting up Scott W. Ambler & Associates and our focus is going to be on helping organizations understand and adopt Disciplined Agile Delivery both at the project level as well as at scale so I hope it will be a fun ride.
3. I’m sure it will be; so in June you published your new book; was that co-authored; I forget.
It was co-authored with Mark Lines of Above Measures at the present moment and a really good guy; I’ve known him for years; he used to be Rational employee long before I joined Rational and he basically left Rational shortly after I joined so it was a different curve paths for a while.
Michael: Passing ships.
Yes; yes, passing ships basically; so yes, like I said we’ve known each other for years and we work really well together and I guess two years ago he was looking at the work I’ve been doing this Agile delivery and he had been seeing it and applying it in practice at companies and he basically said, you know, we need to read a book and I said yes, didn’t really want to write yet another book but then he was right and so we yes spent a couple of years and wrote the book and it just came out in June.
4. And that’s called Disciplined Agile Delivery?
Yes.
Michael: That’s the title?
Yes.
Michael: Okay; and so which you acronymized I guess as DAD.
Yes.
Michael: So tell you us about DAD.
Yes; so there’s a few basic fundamental concepts there; so first of all it’s a people first process and just like all the other Agile methods, you know, not exactly news; but instead of being prescriptive, it’s goals based; so what we talk about is so, you know, at the beginning of the project you’ve got a goal of –you may have a goal of investing some time in understanding the scope and so our point is that there’s multiple ways that you can do this; so to understand the scope, you’ve got several issues you need to address; you know, are you going to write any sort of requirements, are you going to spend some time doing modeling; if so, who are you going to work with, if so what views are you going to address, if so how much detail are you going to capture, if so then what types of modeling, you know, how are you going to organize your modeling efforts; you can do formal modeling sessions, informal interviews, whatever.
So the point is that there’s many goals you need to achieve throughout the life of a project or a product and there’s different ways you can achieve those goals and each of the issues you need to address there’s different options and then there’s tradeoffs for all these options; so one process size does not fit all and that’s what we’re getting at; so we don’t prescribe, you know, we do say you should probably address these goals, but we don’t prescribe any sort of techniques and it’s interesting and most people seem to love it, every so often there’s a few people that don’t like it but for the most part some people want to be told what to do or I just want to be told to do use the stories and have a 15-minute standup meeting and blah, blah, blah; okay, if you need to have a prescribed process with Scrum then go ahead and do that that’s great, good for you but maybe there’s better ways of working.
We also have full delivery life cycle so from the point of view of a single release so how do you start a project, how do you do the construction stuff, how do you transition into production and then obviously you loop around and continue; another aspect is that it’s enterprise aware so we’ve made the not so radical observation that your team is not the only team in your company that there’s an existing infrastructure that you should probably leverage, there’s existing ways of working and there’s other teams that maybe in flight that you might have to integrate with that at least you shouldn’t blow out of the water when you go production; so you need to have this bigger awareness, this more of looking at the whole picture point of view like the Lean folks talk about; and you should be working with it, you know, building out the common infrastructure, following common conventions; so there’s a bunch of mature ways of working that can really make, you know, assuming that the enterprise folks who work in an Agile manner can really help your overall effort; so these are some of the more important aspects of Disciplined Agile Delivery.
I hope it’s refreshing for a lot of people because we are observing what we’ve seen and done in the real world, we are saying here you’ve got choices, it’s not quite the rhetoric the you hear from some people; we make it clear that there’s multiple ways of doing things; so maybe yes I’m a firm believer in test driven development but I also recognize that only a minority of the developers are doing that; you know, forget the rhetoric numbers show it’s something that’s being done by only a minority; I wish it was more but it’s not the case; so well if you’re not doing TDD well what are other ways of working that might not be as good or it might be better; there’s some better techniques so what tradeoffs are you making and we want to help you make the appropriate tradeoff for the situation that you’re in.
Because one of the things that we saw over the years so, you know, Mark and I have both done and I don’t want to make it sound as if it’s just coming from Mark and I; this is based on work that’s been done around the world by many, many people but Mark and I put the book together of course; so what happens is the – so Mark and I both had deep experience in the Unified Process and the Unified Process worked, did work very well and still does in many organizations; there’s been some challenges with some companies but we need to recognize that there have been a lot of successes.
And one of the things that we saw particularly with the Rational Unified process years ago was that it was trying to be everything for everybody; so it was all this here’s this wonderful knowledgebase of process ideas and they were all good, but the problem was they were all good; so a lot of people would look at it and go I need to do all this which is almost always the kiss of death; because the fundamental message was here’s all these good ideas because you’re a process expert you can tailor rough down to meet your needs; but software from a fundamental assumption of you’re a process expert and you’re not a process expert so what happened is organizations and teams and individuals will be making less than ideal decisions.
Now what’s happened over like the last ten years particularly with the Agile community is community is Scrum has gotten really important and the Scrum guys have come to us and said well here’s this small kernel of really good ideas and they’re all really good ideas in a given context and because you’re a process expert you can tailor up Scrum to meet your needs.
And so it’s the same fundamental assumptions been made also wrong; most people are not process experts, they really don’t know how to tailor and adopt processes; so as a result you see all this thrashing going on in companies as companies learn and figure things out; and there’s a lot of sub optimization going on and a lot of minds getting wasted and that was actually one of the motivations to develop DAD to begin with; because when I joined IBM I was pretty adamant, I was not going to develop another methodology; but then a couple of years in after helping organizations scale Agile, I kept seeing the same basic solution over and over and over again. They started adopting Scrum and then they would tailor it into something and that something would look basically the same all over the place, but they had a lot of learning experiences along the way; and they had a rough go of it, it was, in a short story a lot of money got wasted.
So we put DAD together to sort of give people some advice and it’s another foundation on which to scale, that’s another feature in DAD that I didn’t hit on; and the basic idea is that, so the fundamental assumption is that or rather fundamental idea is that you should start with something that’s end to end, that’s going to work for you end to end that gives you, you know, makes it clear here are the complexities of what you’re trying to achieve in software development, here’s the issues you need to think about, here’s your options, you make the right decisions for you; and a lot of the decision points for some techniques are based on scaling factors; so if you’re a larger team, well chances are you’re going to do a little more modeling; if you’re in a regulatory environment, chances are your testing approach is going to be more sophisticated than just doing TDD; so TDD might be a part of it but it’s not the full picture and then other reasons to change things; so this I think and you should learn as you go so we try to embed learning as much as we can into the process framework.
So this is what we’re trying to get at and I think we’re getting great feedback from people; the book has been selling really well so I’m hoping it gets picked up and I believe at least it really is the next generation of Agile method; and like a lot of the conversations we’re hearing at the conference this week, it’s all built into DAD, you know, DevOps stuff and it’s all good stuff; but …
You know, why figure it all – you know, there’s still this phenomenal amount of waste and you can see it at the conference; like all these organizations are struggling with the same basic problems and in many ways it’s because of the consultant approach that we’ve seen so far in the Agile community; you know, I think we can do better.
Michael: Interesting; so well just to kind of clarify all that.
Yes.
We’re not; we’re putting it together and we’re saying here; because all the messaging, you know, Scrum guys and rightfully so they say –you know, they’re looking at the leadership, how do you organize team but they don’t go in the technical practices isn’t it and they’re phenomenally clear about that, they’re very good about that; but then there’s this hand waving of you need to figure it out or there’ll be that, oh, yes just adopt XP as well, nobody really does that; and what happens is they adopt a couple of ideas from XP, they either adopt purposely a couple of things from Agile modeling or they reinvent a lot of stuff because they apparently can’t use any sort of a search engine and they adopt ideas from Lean and Agile data and stuff like that too; there’s a lot of reinvention stuff going on in the Unified Process; it’s amazing, you know, once again at this conference you’ll hear people all these wonderful techniques that they’ve stumbled into over the years and there are good techniques; and I’ll be listening and you know, that was in the rough in the late ‘90s, you know, what a shame we weren’t able to read and figure this out on your own and get a leg up when you’re figuring it out but it is what it is.
So yes, it’s a hybrid; we’re taking ideas from a bunch of good places and some non-Agile places as well because we need to observe; you know, rhetoric aside, we need to observe that a lot of very impressive systems have in fact been developed by waterfallish heavy processes and that’s still going on; like you know, just recently the Mars Curiosity rover landed that was not an Agile project team; that was a traditional heavy waterfall and they’re basically trying to hit moving planet hundreds of millions of miles away and they did it and they pulled it off phenomenally impressive; and there was no room for error on that one; so we need to give credit where credit is due sometimes; some of these older techniques do in fact apply in certain situations.
There is; so there’s some similarities and a lot of similarities; so first of all because we took the approach of let’s look at the best practices from any available source including Microsoft and other organizations, we went looking for what’s really working and what are seeing work in practice; so we talked about continuous delivery is among some of the options; so if you look at continuous delivery as a set of practices involving continuous integration, maybe continuous documentation, continuous deployment, good stuff like that, yes, we bring all that in.
Another interesting feature of Disciplined Agile Delivery is that we don’t even prescribe a life cycle; so we have two life cycles, you know, the basic life cycle what we call the basic life cycle based for the most part on Scrum but full delivery life cycle and then the advanced life cycle which is very Kanbanish; and what we do is we say more than likely most teams are going to start with the Scrum thing, okay fine but as you learn, you know, if you’re actually following the rhetoric of the Agile community and learning as you go and trying to do process improvement and holding retrospectives and acting on the results, all those good ideas then certainly the life cycle should evolve; so then it begs the question well how does it evolve; so based on our own experiences both customers and within the companies we worked for, we started seeing that over time they start to evolve towards something that’s more Lean and more continuous delivery type of thing.
So our suggestion is you probably want to start with this incremental iteration based or sprint based, whatever you want to call that type of approach, get good at that and then over time start taking away some of this, you know, start changing the – you know, start removing the common cadence from some of these practices and start moving towards more of a continuous delivery, also start typing up good transition efforts.
So one of the things that we talked about in the book and it was phenomenally hard to describe it was one of the things I worked on for a long time is that so we talk about how there’s three basic phases particularly when you’re doing just Agile of out the box; there’s an inception phase or project initiation phase where you get your act together; the Scrum guys will refer to this as sprint zero although it’s often multiple sprints so the terminology is a bit wonky; then you’ve got construction where you put most of your effort and finally you’ve got delivery, you’ve got to do transition; and one can call it release, one can call it delivery, whatever, I don’t care.
The average Agile team spends about a month in transition effort like release deployment efforts; in continuous delivery rightfully so they talk about, you know, if you’re releasing once a week or daily in a production, certainly your transition effort can’t be a month or so in length; you’ve got to basically turn, evolve from a phase which is where the vast majority of Agile teams are today; they have a phase or hardening sprints or whatever rhetoric they want to throw around it; they have this phase where they do whatever work they’re doing; you’ve got to squeeze that down into an activity of several minutes or several hours in length depending basically on how long your test suite is going to run right; and this is a very important observation.
So one of the process improvements over time because you’re not going to do this, you know, in a short period time is the squeezing out of the phases into evolving into a continuous delivery type of a thing; so yes, there is some interesting symmetry there I think and so ThoughtWorks is doing some great work; I ran in some ThoughtWorks people just before coming here and I’ve admired them for years and they’ve got some good people and some good staff so.
Michael: Yes; a lot of talent there.
A lot of talent there, a lot of smart people building real things so.
Yes; so we suggest that you manage risk or more importantly mitigate risk throughout; so we take what’s called a risk value life cycle which is straight out of the Unified Process; so where Scrum takes a value driven life cycle and that does in fact take down some risk; so in Scrum they talk about doing delivering potentiable software every weeks, every sprint, everything, phenomenally good idea; we talk slightly differently; first we look at the delivering potentially consumable solutions every few weeks because it’s more than just software; you’re delivering documentation; you’re running on hardware; you’re changing the business process, you may be changing the business process, you may be changing the organization structure and you darn well better be considering all of those things every sprint or every iteration not just software; so we need to up our game.
The second thing is yes, shippable is nice but any idiot can ship; it’s I’ve got to ship stuff that people want to use and are willing to use; so it’s got to be consumable and it’s actually one of the really nice things about this conference is there’s a full track on UX on usability so yes, and that’s like basically one aspect of consumability so that’s a very good thing; so we promote that certain idea.
9. So feedback would be super important in that?
Yes; okay so that was a value driven life cycle; I lost my train of thought there; so risk value life cycle extends the value cycle and what the observation is and like I said this is straight out of Unified Process is that there are certain risks that occur, certain common risks that occur on these IT projects; so at the beginning of the project you darn well better have some sort of reasonably consistent vision of what needs to be achieved and in what direction we currently think we’re going in; you should probably have a technical strategy and better yet prove it really early in the life cycle, make sure you’re going in the right direction; so at the end of it, you know, there’s these milestones you want to knock off pretty quickly and it changes the way you work so it’s not just a value; so in Scrum you talk about a product backlog where prototype is based on business value, a great idea but if there’s some technically nasty things that you need to do you darn well better knock them off quickly; I want to fail fast, right, or I want to succeed quickly, it’s probably a more positive way of saying that; so if something is going to bite me I want it to bite me right away and I want to get burned by it; because if there’s some certain technical things I just can’t accomplish well I’d be better just recognizing that right away and cancel the project.
And then I want to make sure that, you know, so the construction ends when we have sufficient functionality or sufficient business value has been produced to justify the cost of transitioning; you want to make sure you’re production ready; so this is actually an interesting thing from a risk management point of view was how do you describe when you actually complete your project; are you done when you’ve shipped, maybe or from a Lean point of view, yes you shipped that’s nice but you could still fail miserably in the marketplace or in production if you’re an internal organization; should you not have delighted stakeholders, so you know, should you not be in a position where people are using your solution, enjoy it or at least don’t hate it, better yet they enjoy it and are getting real value out of it; like that is really the mark of success.
So yes, and this is all about risk management, risk mitigation whatever you want to call that, some aspects of that and then making sure that you’re paying down your risk as soon as – you know, at the most appropriate time hopefully early.
Michael: Right, so getting back to this kind of delivery pipeline thing.
Yes.
10. Where does DevOps kind of fit into this?
My point of view on DevOps is that you want to get the development organization and the operations organization working together effectively and I’ve worked on both sides of that fence and they have different views of the universe; so the development people they want to get something out the door as quickly as possible, business value, yada, yada, yada, okay that’s nice and it’s great ideas; the operations people they want safety, they want to make sure everything is up and running, it all works together, you know, it’s easy to keep all those stuff running because they got to keep the lights on right and they don’t want to be building up technical debt, they want to minimize the number of platforms they’re dealing with, number of technologies they’re dealing with so they have some very, very important and very interesting issues which contradicts some of the development issues right; so there’s this tension, these tradeoffs that are being made in these IT organizations.
So DevOps is all about let’s get everybody working together effectively and so my point of view on this is that the only way this can happen is if you’ve baked from development process point of view that you’ve baked in operations concerns into the overall process; so we explicitly include operations and support people as stakeholders; we explicitly say you really better be involving these people at the very beginning of a project to make sure you understand what’s going on; you should be working with them all the way through; you should certainly be working with them before you get close to when you think you’re going to release into production; you should understand what their release windows are or what their release procedures are or what their concerns are because you don’t want to be going, you don’t want to be in a position where you think you’re done and then you talk to the operations folks and you find out oh there’s going to be another three months of grief because you’ve got some hoops to jump through or you’ve missed the release window; you know, for example in many retail organizations in north America at least they’ll have a blackout period of releasing in the operations around Christmastime; so maybe beginning of November all the way until after New Year’s though shalt not release anything in production because it’s just too risky because you don’t want to blow your stuff out of the water during your busiest time of the year.
So these things happen and if you don’t know that, you know, if I’m in a project and think I’m going to release in the middle of November and suddenly it’s the end of October and I find out oh, that’s not going to happen, awesome I’ll just add it through what’s under my project just because I didn’t have a conversation with somebody, yes, a few months earlier so; yes that’s just a few issues that – so anyway so this measured delivery we baked these DevOps concepts into the process.
One of the things I’m seeing right now in the industry is there’s still a lot of confusion around DevOps and rightfully so; there’s still a lot of voices and the voices haven’t coalesced yet but yes some of the vendors will be DevOps is my tool, my one or two tools will solve all DevOps issues and very clearly not the case; or that you get a few consultants that they’re focused on the dev stuff or they’re focused on the ops stuff, almost always in the dev stuff; so they’ve got a few really good ideas but they’re not looking at the big picture; so I think DAD provides a pretty good decent big picture at least from an Agile point of view of how this DevOps stuff fits all in so; and I think we’re that first real process framework that actually addresses DevOps so we’ll have to see how things go.
Michael: Great; and then finally this is an open question but too big movements in the enterprise right now cloud clearly and big data architectures.
Yes.
So I think that, you know, cloud first of all is an enabler, potential enabler of agility; particularly from a business agility point of view because I do want to be able to scale my – you know, so if I discover that I builds something and discover I need more cycles or whatever I want to be able to scale it easily; I don’t want to have to start building an infrastructure when I don’t, you know, unknowingly because it’s not as easy as some people would think sometimes to build up your infrastructure; so that’s good; the cloud can particularly be good for me internally for testing point of view like, you know, makes it easier to spin up testing environments and stuff like that; so there’s some real opportunity there.
But I can certainly be Agile and not be in a cloud environment, you know, clearly and many, many organizations have done that and I can certainly be non-Agile and be in a cloud as well; so they are in many ways orthogonal although given the opportunity I think I would rather be doing cloud stuff; but at the same time there’s still a lot of hype around cloud and I think if you look at the Gartner hype curve I think we’re still clawing our way up the curve and there’s going to be a reality check in a year or two because some of the stuff I hear about cloud is just what plan are --- you know, what are you smoking; but anyways.
And then on the big data’s point of view that’s also interesting; you know, we’re clawing our way up that hype curve as well; now there some very interesting things going on because analyzing all these various streams of data coming in at phenomenal rates is a very interesting and complex problem; so there’s some good stuff going on there; but there’s a challenge the data community has not adopted Agile as much as it should; they’re still at the beginning of the learning curve; now having said that there’s some good stuff going on in the data community; they’re finally getting there but they’re a good ten years behind and that’s a real sort of a shame there; so I think the most important message I can give is that the data community needs to take Agile seriously and they need to invest more time in it and we’re starting to see that happen; we’re seeing some good stuff in the data community but they’ve got a long road to go still so.