2. Can we start by describing yourself a bit and what are your main focus areas?
Sure. So, I am CTO of a company called Evolve Beyond and we are focusing around Agile, Lean and continuous delivery sorts of activities. My background, I’ve been working in technology for a really long time, I would describe it as bringing technology into the business, so using it to really enable the business to do things that the business hasn’t been able to do in the past. So, some example is I worked on the first electronic crossing network for trading stock, so completely revolutionizing the way that people now trade shares and all of that and then went in and worked at one of the first software-as-a-service place and then we built cloud computing technologies and continuous delivery technologies. What I have found is I have now moved from there to… now that people utilize technology and having it be so key to the business, I actually spend more time getting people to work better together, to actually understand what’s going on and to adjust and start to look at ways that they can actually improve and utilize the technology that way.
If you think about the history of IT, IT was an ancillary service like HR or finance; it was something that people would use as a tool to enable things to go a bit faster. And when you have it like that you start to think about it as being like a building or something that’s a cost center. That’s changed, now that IT is so critical to business, in fact IT and IT services often times are the business and now that it actually is the business, I think that businesses need to change, they need to actually look at it as being the business and being how they make money. I remember when I was at a startup that was one of the first software-as-a-service place and I was at a board meeting and I remember one of the investors, we were talking about how we were utilizing our operational capabilities to be able to get us deals and he just went “I can’t believe that, you just somehow made operations into a profit center”. And I think that more and more people are starting to make that leap and I think that once people really understand what you can do, how you can enable the business that way, it will change, and I think that change will actually be good, I think that will allow for people to stop looking at the costs and be able to actually look at what can they do, what can they enable.
João's full question: Your talk was about DevOps and how it can help the business. There are some studies about DevOps adoption numbers and there are some differences between those studies, some of them say there is a majority of organizations that have adopted DevOps, other surveys came to the other conclusion, that it’s only a minority. Why do you think this happens and what are the most common challenges that have to be addressed for a successful DevOps adoption?
I think the big issue there is that the definition of what DevOps is, is fuzzy. So, you have some people think DevOps is: “I have some tools that I use to do some automated deployments” or “I utilize AWS and cloud technologies”. Then you have the other side of things which really is DevOps, which is how can I bring the operational and the development side, not only closer together themselves, but closer to the business, how can I actually enable things to happen, how can I have it so that everybody is on the same page. I think when you actually start to see it be successful within a business is when you can go to a business person and you can go to a developer and you can go to an operations person, if you happen to have that separation, and you can ask them a question about the service, whether it’s a business question or a technical question and they’ll all give you pretty much the same answer. And I think that most businesses that you go to and you do that you won’t see that, so I think that there is quite a ways to go.
I think a lot of that comes from some of the things that are out there where people say “oh, I am going to get rid of all the operations side all together, I am going to put in my continuous delivery tools and I am going to allow for everything to stream immediately from the fingertips of my developer straight out into the production environment”. It doesn’t need to be that way and in fact from a governance standpoint, it can improve governance. So, if you think about it, back in the past, a lot of the governance was to be able to go through and track manual changes that people were making to environments, so you would go through and have a document or several documents that would have a series of steps, type this, click that, put this and do that, call this number, those sorts of things.
Now we have automation tools, not only can automation actually do those things and do those things better because people don’t do the same thing exactly the same way all the time, whereas computers are very good at that, you can not only have those actions actually be automated, but you can actually have those tools create an audit trail themselves of exactly what they did, exactly what happened, even have acceptance tests that are associated with that actually check to see: are the results exactly what they should be? They will also allow you to be able to understand the difference between environments, what’s going on in my production environment and why does it work differently than my test environment and I think that greatly helps from a governance standpoint. Also, there is nothing that would stop even with a continuous delivery pipeline to be able to stop you from being able to have a button that the operations guys can push to do the last check. So, I absolutely think you can definitely have it, in fact I’ve seen lots of businesses that do have better governance and in fact have had their auditors go through it and their auditors are usually if not happy, if not completely tickled pink, they are at least happy with what they have.
That’s a good question, it’s one that I run into all the time and because I go into lots of big regulated environments, utilities, financial institutions, health care; you have to go back and think about the original intent of the segregation of duties. The segregation of duties was to be able to prevent any one person from being able to come up with an idea and put it out there and make it live. DevOps doesn’t prevent you from having different people doing different activities, however what DevOps can allow you to do and what a lot of people miss is to be able to tie threads throughout your lifecycle, to be able to say: “Billy had an idea, Billy’s idea was to do this with this kind of outcome, which Sally then worked on with her team and this is what she came up with as a result” and be able to tie those attributes together and then be able to still have the four eyes principle from either allow Sally or Billy to be able to do the launch, but you allow Joe, who happens to be the Ops guy, to push the button that says “I did this release this time and went to these following machines, I’m utilizing this deployment technology, and I can now see line of sight all the way back I was doing it because it was Billy’s idea and this is what the outcome should be and this is what Sally and her team actually created as far as attributes that were associated with that.” So, you can absolutely do it, in fact I would insist on anybody who is in a regulated environment to really look at how you can tie your attributes together, how can you go through and make sure people are working on the same page.
7. [...] How can we make business and operations be closer?
João's full question: DevOps famously emphasizes the visualization and the consideration of the whole valid stream and the collaboration of all the stakeholders from the business to the operations guys. Do you have any tips on how the operations and business sides can understand each other better, because it seems that in many organizations the developers are in the middle and they act like a bridge. So, how can we make business and operations be closer?
I think that one of the things that I see happens time and time again, and you can say it’s either the business side or the operations side, but I’ll start with the operations side; the operations side often doesn’t reach out to the business or even really understand what’s going on, often times they feel like they are under so much attack, so much pressure, getting accused of so many things that they pretty much gruel in their shell and they don’t actually look at the opportunity of what is the business actually looking for, what are they trying to do, what are the things that concern them. So, to give an example, almost all businesses and business people go: “I really wish I could deploy a lot more quickly, I don’t understand why it takes so long for this feature, or whatever, that I want or additional capability to be able to get out there”. And I think the operations people actually have the visibility of looking at what is the friction that’s involved, does it take a long time to install something and looking at why and looking at what are the things that can actually improve that.
Operations people all the time want more tools, better tools to be able to make their lives better, and they can go through and say: “this is causing us a problem, the packaging isn’t right or we have a really long and evolved process for being able to install things and if we make these improvements and these investments, we should see the needle move a bit”. And I think by reaching out to the business, actually making it be much more of a business value proposition as opposed to a techy tool type of conversation, I think we’d go a long way. And it’s the same with configuration management, I’m looking at what are the things that cause problems in a production environment, so: “do I have to do a lot of rework when I install something new”, “is there a component that is really slow or is vertically scaled?” and if we actually had a little more focus on changing that, improving that, it would not only allow the operations guys’ lives to be better, but allow the business to be able to be much more Agile and be able to offer better things to their customers and I think that is a big win, but it really takes making that step and I think the business should look at it from the same perspective. If you’re complaining it takes so long, don’t just talk to the developers, look at what’s actually going on, walk the lifecycle and see it.
I think this goes back to what I was saying earlier which is make it be something that is understandable within the context of the different groups. So, for instance, if you are on the technical side of the house and you want to be able to collaborate better and work better with the business, be able to actually have a conversation with the business, don’t say: “oh, I want to go and utilize Puppet”. You instead go: “I think that we can make big improvements in how we deploy our environments that will allow us to be faster, allow us to have fewer problems and be able to actually measure what’s going on currently” and even look at other places where you can do a little proof of concept to be able to show a pre-imposed side of things. I think that also making a lot of the metrics be much more visible I think is important. And this is one of the things I was talking about in my talk which is a lot of people will show aspects of load averages and those sorts of things. Those aren’t meaningful for a business guy, not unless the business guy happens to be an engineer hiding underneath the suit.
But if you are able to go through and say “we had these problems and affected our customers in this particular way”, that is also something that is more meaningful to business. I’ll give a third example, which I call the tyranny of the nines. So, this is around availability; I was at a very large company which will go unnamed, and I was away on holiday and I came back and found that the business was pushing really, really hard on my team to build and deliver a 99.99% available solution for a particular customer. So, I went to the business and I said I would like to understand more of the dynamics of what’s going on here, so what is it that the customer really wants and what’s the value of the deal if we are down, what is the actual cost to both us, both from a financial perspective as well as a reputation perspective, and what would be seen for the customer?
And I found something very, very interesting. I found that the customer really only cared about the particular service during business hours, so it was nine-to-five and it was Pacific Time US and they could care less if it was completely down for the rest of the time and I also found that the financial penalties for this was something like $120,000 if we were down. The solution that we were planning on putting in place was 8.4 million, it was a bit of a gap and as soon as I sat down with the business and said “look, we can go through and we can change the way that we do our deployments so that they are happening in off hours and if we make these kind of improvements will allow us to have a better understanding of what is going on and maybe be able to respond more quickly when we have a potential problem or reduce the problems that we might encounter during these particular times, but we don’t need to build all this crazy disaster recovery and all these active-active systems”. And when you put it that way to the business, they just went “you’re right, that is absolutely silly”, and I don’t think people do that enough and I think that would be really powerful.
The biggest strategy I follow is I go through and say “what is the business value of these operational things?” So, say you want to improve monitoring, well, why do you want to improve monitoring, you usually want to improve monitoring so you can understand what’s going on and be able to respond more quickly to stuff. Well, why do you want to respond more quickly and know what’s going on? Well, if there is a problem you want to be able to address it quickly. Well, why do you want to address it quickly? Well, because the customer might be impacted. A-ha! That’s something the business might be interested in and if you’re able to go through and say “look, today we missed things, it takes us longer to be able to respond to outages and problems that we have, it takes us longer to troubleshoot and if we get this we should see an improvement of x% or we should see our times improve in this following way.” That changes the game for the business and it suddenly becomes something that is actually business value. I always hate it when people call them non-functionals. It’s not a non-functional, it’s something that is providing a capability to the business or it’s reducing a risk that the business may have, maybe even a risk that they are not even aware of. So, I think having that conversation with the business, being able to articulate it that way and not come up with the answer of “well, we’re completely under fire or it would make our lives better”, that’s not terribly meaningful to business; if it is, it will improve, it will help our customers this way, it will help us be able to serve our customers in these following ways or it will enable us to deploy things more quickly or it will allow us to understand a bit more of what our customers are actually doing. That changes it a lot and I think we need to take more opportunities there.
I would actually disagree with that. I guess having spent so much time with operations in IT organizations and having to do the great battles of ITIL and COBIT and all those types of – you can call them methodologies much like you could call Scrum or Waterfall or those types of things, because it’s a framework of how to perform. So there has been a lot of stuff that’s been out there and I think that what you are seeing now is.. the development people haven’t been super exposed to that, I’ve already been exposed to all the problems part of it, I need to go through change advisory board, what the hell is a change advisory board, not knowing that, that happens to be tied to ITIL types of capabilities or being able to understand the risk profile of a change and making sure you have a governance that’s in place. But there has been more and more people looking at how to go through and change, how to utilize things such as Kanban, even doing some Scrum-like things, actually I think that is one interesting challenge area that I’ve seen, a lot of people trying to apply just development methodologies pure as the development methodologies are within operational context, not understanding that operations is indeed a different type of environment.
So, for instance, in an operational environment it’s incredible difficult to say what you are going to do for the next week or two weeks because my machine might start alerting suddenly that somebody is on fire. So I need to have much more flexibility to be able to do that, however there are aspects of things such as Scrum that are important, being able to have key sync points within a cycle, even being able to know how to work with a development team, if you again have separate operations and development people, how can you work with them and understand the methodologies that there are and allow those to melt. So there have been methodologies traditionally within operational in IT environments, I think that we are now starting to see even in the language of things such as ITIL are starting to encourage continuous improvement, looking at how to be able to be much more iterative, I think there is still quite a ways to go and there are a lot of people experimenting there, but I wouldn’t say it’s methodology free.
11. We’re coming to a close, do you have any final thoughts to our audience?
The big things that I would say are the key parts of DevOps aren’t just I am going to automate things, they are really reaching out across the business from the development, the operations side as well as the business itself and really finding ways to work together and collaborate, finding ways to, and this is uncomfortable for business people as much as technical people, reaching out to places where you might not be super confortable and having some conversations about stuff, looking at doing it in a blame-free sort of way, saying “look, I don’t want to say that it’s somebody’s fault, but I want to look at how we can improve” and look at incrementally improving, look at trying things, look at ways to go through and reach out and learn and there will be times that will stumble, there will be people that will push back about we need to protect the business or things on governance, but there are ways through that, there are ways to be able to bridge through those things; it’s just thinking about what is it that you are actually trying to accomplish and how might you be able to accomplish it better utilizing tools and technology and communication.
João: Thank you so much for your time, Robert.
Thank you.
João: It’s been a pleasure.
Thanks.