Sure. My name is Jez Humble, I am coauthor of Continuous Delivery and forthcoming Lean Enterprise, I’m a vice-president and I work at Chef.
Ralph: You mentioned Lean Enterprise; you wrote a book, I guess it’s not out right now, we can preorder it as far as I know.
That’s right, it’s been available for pre order for about a year now and actually just last night I did the final revisions to the last chapter, so it will be out by the end of the year.
2. Do you want to tell us a little bit about this idea of lean enterprise, what’s behind it?
Sure. I think there is a number of movements that have emerged over the last ten years, obviously agile was a big deal in 2000 and agile has its roots in lightweight methodologies from people like Tom Gilb and so forth, even in the late 1980s. But then what we’ve seen over the course of the last ten years or so is the emergence of things like lean startup, continuous delivery, DevOps, and all these movements around how to not only how to just build things correctly, but also around making sure we build the right thing. A lot of the features that we build deliver zero or negative value to our customers, so there is a big piece around how do we make sure we don’t build products that people don’t care about or features that people don’t care about.
So the Lean Enterprise takes all those movements and synthesizes them and presents a kind of overview of how you do product development in a lean way, in terms of everything from validating new business ideas to building large scale software projects in a lean way, all the way through things like financial management, governance, risk and compliance and how IT should work. And there is a particular focus on culture. So, I work with Gene Kim and the folks from PuppetLabs on the DevOps repport this year and we actually were able to measure culture and talk about the impact that culture has on creating high performance software development organizations. So all that stuff is in the book, it’s all basically designed to help leaders and managers in large enterprises to understand how to move much faster, adapt much more effectively to changing circumstances, taking the lead from organizations like Amazon and Google who have demonstrated how to build software based products at scale.
Ralph: So it’s kind of a logical next step of the introducing all the technologies that help us become faster or methods to become faster now to address the mindset of the management, because that’s how I feel that sometimes it’s just the mindsets that are in the way.
Yes, absolutely right, and that’s the focus of the book. I think the problem is not that people don’t want to do the right thing. Normally people do want to do the right thing at every level of the organization, the problem is the systems that we build in terms of the organizational systems, when you are an architect you are obviously building a system architecture for software is one thing, but we need to think really careful about system architecture for organizations and how we build adaptive organizations which are really good at adapting to changing circumstances, but also should be great places to work. There has been a ton of research over the last several decades which has been crystalized in books like the one by Dan Pink on how intrinsic motivation is important, the ability to be able to take pride in your work, to be able to have the opportunity to pursue mastery and purpose and have joy in what you are doing, and that’s a big part of creating enterprises. There is a great quote by W. Edwards Deming where he says that a bad system will kill a good person every time, and enterprises are very liable to be in that situation where there is a bunch of good people who want to do good things, but the system and the way we design organizations prevents people from being effective.
Ralph: That’s right, because in my impression when I look at agile methods or continuous delivery in big enterprises there is a clash because the business units are used to the fact that software has been tested for weeks and you have roll outs that take maybe four weeks or eight weeks, the process, so we have to change this at this point, I guess.
Yes, absolutely. And this is the thing, this continuous delivery book came out and from a technological point of view it’s just XP in many ways and there’s not really that much that’s new in it. It’s just showing you it works and you can stich it together and the deployment pipeline pattern was I think reasonably new, but the barrier has been these things that you talk about, the mindset, the way people interact with each other, how we design high performance culture, these have been the barriers and architecture as well, you have these monolithic coupled architectures in enterprise organizations that are very painful to do continuous delivery.
3. [...] Do we also have to change the developers so they recognize their responsibility?
Ralph's full question: Yes, I think so, yes. We mentioned to change the mindsets of the managers, but we also have developers in those big enterprises and I was talking to somebody and he told me that, it was a startup, in their startup developers are actually able to commit to production, so you must have rather high quality gates when working. Is this a problem, too? Do we also have to change the developers so they recognize their responsibility?
Yes, I think that is absolutely right. And one of the problems we have in enterprises is that the job of the developers is to get the code dev complete. Which is very different from getting code in a state where is releasable and operable in production and I’ve worked on projects personally where I finished working on a project and I moved on working on another project before the projects even went live. And so you never get that feedback loop, did the software that I build actually work in real life? Was it stable, scalable, secure and then even further, was it actually valuable to the people who use it? You never get that feedback so you can never learn how to get better at that. So, you are absolutely right, there is a key mind shift set for developers, just the general thing that you mention is the thing that I am building actually releasable, of high quality and usable? There is security and UX - developers have to consider these things. It’s not enough to say does the software pass the tests. It has to actually result in an improvement to the product for the users. So I think yes, developers have to care about these things, that’s absolutely right. If you look at companies like Etsy. Etsy, is an example of a company where developers literally push their changes into production. I am a big fan of John Allspaw, I try to catch up with him as often as I can and he told me a story about going past the developer, he’s walking down the corridor, and the developer says “John, this change, I am not sure about it, you think I should release it?”, and John Allspaw says to him “I don’t know, do you think you should release it?”. Developers are much more cautious in that environment and I think that’s a good thing. You should take responsibility for the impacts of what you are doing and take responsibility for your actions. I was talking to Katherine Kirk yesterday and one of the things she said is in a continuous environment you have to understand consequences. And actually suddenly you are presented with consequences for what you do and that can be difficult for people to get used to, so yes, that’s a big change.
Sometimes people want to know what is their optimal lead time to be able to get changes to production and the answer is if something goes wrong, how quickly do you want to be able to get a bug fix out to restore service because that’s what you really care about. Anytime your standard process is too slow to get a bug fix out that’s a problem because you would have to bypass parts of your process and those parts of your process is what’s giving you some assurance that you are not going to make things worse.
Ralph's full question: If you come to a large enterprise, for example governmental organization or something like that and your goal is to introduce these lean ideas, to introduce continuous delivery, what is harder? Is it the technical part because those enterprises have really much technical infrastructure that needs to be changed at this point or is it the mindsets?
The thing is that they are coupled. Conway’s law tells us that organizations produce architectures which represent their communication structures within the organization and so the production architecture in some ways reflects the communication structure and the history of the organization. So I don’t think you can treat them independently and this is a mistake people make. I’m sure you’ve experienced architectures where the person who designed it was doing in a vacuum. Considering what the ideal architecture would be without considering the structure of the organization, the capabilities of the people, the legacy systems. If you design in a vacuum you don’t take account of those kind of things, you just end up making things worse. So you need to do both.
Continuous delivery imposes architectural constraints, testability, deployability, ideally a service-oriented architecture where the independent services are independently deployable where you don’t have to release the whole world in order to get one single change out. So there are architectural things, but also the mindset things, particularly DevOps has drawn attention to this problem that developers will code something and then will throw over the wall to IT ops and that’s a big problem. That’s a huge barrier to improvement. And then also going back to the way we think about management, there is no effective training for IT managers and it’s considered to be something you can kind of pick up once you’ve been doing it for a certain amount of time and I think that is not true. There are key things we should be teaching people in management. Things like how to create teams which are autonomous and which are able to run experiments, how do we design experiments, both in terms of process improvement and in terms of feature development and products, validating products. These are key skills that aren’t really taught, they are not well understood and there is a huge amount of work that needs to be done there.
Ralph's full question: Yes, I believe this. So, you were mentioning architecture and software structures and monoliths. I think we heard a lot about microservices the last maybe a year, maybe one and a half, you can’t enter any conference without a microservice track. Is this new concept of microservices or this now popular concept of microservices, is it important for continuous delivery to really work, to be agile?
I don’t think you need microservices. I mean the two models that I see in the large scale web worlds is if you look at what Amazon and Netflix are doing, that’s very much a microservices model to me. Amazon has tens of thousands of services in production, many of them quite small. There are some bigger ones as well and there is a very deep stack of services. I heard there used to be an interview question on amazon.com where if you interviewed to work at IT ops they would ask you what happens when you type amazon.com into your browser and if it took you less than an hour to answer that question then you didn’t get a job, just because it’s just so complex. Just to render the front page of amazon.com a hundred different services have to interact, and that’s very effective and Netflix do the same thing. Amazon migrated to microservices architecture for four years, from 2001 to 2005 and Netflix looked at it and did the same thing in two years. But there is another perfectly good model which is if you look at what Google and Facebook do. It’s still a modular architecture but the independent modules are not independently deployable. They are all coupled together at build time and you end up sending out these enormous binaries with an entire service, a large service with a property like AdSense or Gmail it’s a big thing, the Facebook binary is this huge multi hundred megabyte thing that gets sent out using BitTorrent. And that’s perfectly viable, too. And then if you look at ETSY, it’s just a bunch of PHP files. It’s a monolithic architecture, it’s not service-oriented at all. So, I think all of these different models can be made to work, but certainly there are some nice properties of microservices that make them very interesting. But I don’t think it’s the only one way to skin the cat.
My personal definition of continuous delivery is making sure you behave as if you are going to release every change you make to the source code, and that doesn’t mean you have to release every change. One of my favorite examples of continuous delivery is the HP laser jet firmware team. People don’t update the firmware on their printers ten times a day, but they were still able to achieve enormous cost savings and quality improvements by implementing continuous delivery. They got rid of the whole stabilization and hardening phase, that just went away. And that had enormous impacts on the way they delivered software, but also on the relationship with the product people. Previously they had to do this very detailed planning effort where it took many months to nail down this detailed roadmap because the firmware people were always very late and you couldn’t change anything because you had to replan. Whereas once you got rid of that whole stabilization phase you could be a lot more agile about saying ok, we want to change our priorities and do this instead because no developer was more than one day away from what was in trunk, you can just say ok, we’re going to stop and change what we are doing.
The software is releasable, even a week or two before the new release they could put new features in if the wanted. So it changes the economics and it also changes the relationship between the products and the business and the engineers. And so continuous is really about that mindset of we are going to make sure that the software is always in a releasable state. We are not going to have these hardening phases. In a way the continuous bit is about feedback cycles, just making sure as a developer, as we talked about earlier, you take responsibility. It’s not someone else’s problem to make sure your code integrates, or that it’s well tested or secure, it’s your problem. So creating those continuous feedback loops where you actually have confidence that your changes are releasable, that for me is the big deal.
8. [...] Do we have improvements in the toolstack, also?
Ralph's full question: There was a lot of movement in the area of continuous delivery in the last one or two years, it got very popular also in traditional enterprises and stuff like that; so, what happened here? Is it just that this idea spread and it’s useful so it gained popularity or is it also around the tools that one can use right now? For example, Ansible got very popular in my opinion, within say the last 12 months. So, do we have improvements in the toolstack, also?
I work for Chef which sells tools, but even so I don’t think the tools are the critical thing. Tools advancement is a necessary, but not sufficient condition. People often ask me what tools I recommend and I don’t think the tool choice is that important, which is not to say that tool innovation is not important, because it is. But I think the reason people have started taking these things up is because it’s a better way to do things. So, Gene Kim at DevOps Enterprise Conference a couple of weeks ago in San Francisco, there was the CIO of the US Citizenship and Immigration Services which is part of the Department of Homeland Security, so a very risk averse, low trust culture, talking about how they were doing continuous delivery. He talked about moving from Silicon Valley to Washington to the government and about how he had this conversation where he was “we need to make a small change, how long is it going to take?”, and the engineers used to say “well, 12 months”, and the CIO says “what do you mean, I can make this change in seconds”, and then they were [grumping], and they would say “ok, eight months”, and he was “it’s ridiculous, let’s just do it”, and they said “well, we can’t, sir”, and he was “what do you mean I can’t, I’m ‘sir’, do you know who I am?”, they were “no, sir, unfortunately we can’t because of MD 102”, and MD 102 is a management directive and they can wave it around, it’s 100 pages of paper which sets out the software development life cycle which is this very waterfall thing. He basically created his own management directive to implement agile and they are actually going to put out their first release using a continuous delivery pipeline with Jenkins and Chef and some other bits and pieces. But I think the reason it’s popular is because why should it take eight months to put out a change to a website? It’s lunacy, and I think we’ve proven that this stuff works, it works at scale, it works in all sorts of different domains, and now it’s at a stage where you would be dumb not to be thinking about how you do that. It’s moving from being a competitive advantage to being just how you should be doing things. Tools are important for that, but I don’t think Ansible is suddenly going to make it an order of magnitude easier to do it, it’s an interesting approach and people will explore new approaches..
In researching my book I read a book by Douglas McGregor, The Human Side of Enterprise, which was written in the 1960s, he talks about theory x and theory y in management, and theory x managers believe that people are essentially lazy and that they don’t really want to work and that in order to make them work you have to use a carrot and stick approach in order to motivate them, and then theory y believes that people intrinsically want to do something valuable and they want to contribute and you just have to find ways to connect them to the purpose of the organization and they will do the right thing. And the problem is your attitude as a manager, whether you are a theory x or a theory y manager, decides how people will behave. So, if you are a theory x manager what you will find is that people will behave in the way you expect. Because if you give them a carrot and stick approach they won’t be motivated by their work and they will be lazy because they are in an environment where they are prevented from being able to enjoy what they do. Whereas theory y managers end up with theory y people. So this is like you work in enterprise environments, this is still a live issues today, so 40- 50 years old, I’ve got the 50th anniversary edition of this book, we are still having the same problem, so we still have some way to go on that front, on the one hand that’s great because it means I have a job for life, on the other it’s somewhat depressing because we still need to learn these lessons.
Raplh: So, thank you very much, it was very interesting, it was very much fun having you.
Thanks very much, thanks for having me.