Sure. I'm the chief architect of DevOps and Continuous Delivery transformation at Intel. I haven’t always had this official role, I really just started getting excited about DevOps and Continuous Delivery. I was already an architect but I just started a community of practice and a year later we have an official program. So now we're officially tasked with the job of transformation of the IT organization and by extension Intel because IT is supporting Intel.
2. How and when did you first hear about DevOps?
The first time I had exposure to it I was in a team where we were just trying to solve one specific problem which is our path to production. We are hearing from our developer community that our path to production is painful. After they’ve done all the things they are supposed to do, because of all the various change control and the various permissions they need to get in order to release to production, it's adding eight weeks to the cycle. So we were tasked to find ways to reduce that to something less.
So that’s when I stumbled upon a book by Jez Humble, who wrote the book on continuous delivery, and I was already a proponent, I was already a practitioner for continuous integration and test-driven development. I felt that this was the natural extension, this is the next evolution and I was hooked, so I convinced the rest of my team to try this out in our project. Then a kind of snowball [happened] after that. We were so excited we started a community of practice, telling other people about this wonderful thing that we discovered and all the benefits: “It’s a game changer, everything you know about software development has changed really.” So that’s how we got started.
3. DevOps got started at Intel more like a grassroots movement kind of thing?
I would say it was both, it was a confluence of both. Every year Intel IT has an annual leaders summit, so that same year people came back from the summit and said “Hey, our CIO Kent Stevenson was asking about DevOps and continuous delivery. If we are doing it, what does it take for us to do it...”. So there was a follow up strategic discussion by our staff, and that resulted in officially launching the program. Since we had already started down on this path, the task came to us and it became my official role to make this work.
We have a central focus program on just the transformation effort, but we have a lot of distributed initiatives. The central program focuses on the methodology, how do we need to change the organization's behavior, and what do we need to do to help that along. We provide lots of coaching, the strategy and the roadmap, seeding the knowledge in other organizations.
We also try to partner with other organizations to bring the technology roadmap parts together, end-to-end and available, filling in the gaps. Things like monitoring, automated tests, continuous integration tools, those are owned by different parts of our organization. We're trying to partner with all of them to bring together a cohesive roadmap, with a cohesive user experience.
Then ongoing we have every team that starts on this journey implementing the project, and we try to seek more experts in other parts of the organization, so they in turn can bring others along.
Yes, definitely a lot of our work is getting the information out. We have a community of practice, we meet every two weeks, we deep dive into various topics, like our specific technologies, specific tools, or we share success stories, or war stories sometimes. We also brought in external speakers and experts. We brought in Roy Rapoport and Adrian Cockcroft from Netflix, and those were very popular. We record all our sessions so that we have a repository of information and people can watch it. People in different time zones can also participate in our community.
And then I’ve hosted a couple of hackatons. Those are really wonderful for getting people hands-on experience. We ask them to bring their project or help somebody else on a project and get them on continuous integration, pipeline and hook up test automation and work on the particular area they need. Those are very time consuming and require a lot of volunteers.
And then we have internal open source. We try to leverage that, so from all the output from our hackatons and from our project we try to share the code. We also have others who are contributing, so there are questions like “How do I write unit tests in this platform?”, there are code samples out there so they can see what others are doing. We are hoping to leverage this to also build out our stack and contribute automation scripts and really make this a shared effort for everybody to contribute, so that the burden is not in one team to spin up and adopt these practices.
Those are the different forms. I, of course, spend a lot of time talking to different groups, different organizations, a lot of power point myself to get the word out.
I came in with the expectation that those would be our road blocks, but surprisingly people have been very cooperative, and I think a lot of it is because we have already got the word out there. So when someone said “We need more scrutiny in this area and that area”, people spoke up and said “Hey, this doesn’t accommodate the beat rate that we want with DevOps”. So they are pushing back and asking for a solution that accommodates continuous delivery.
We are having conversations working through how can we make continuous delivery work with security compliance. A lot of it is going to be codified, a lot of it is going to be automated. So I personally feel that you get better security, better compliance, if you automate it rather than relying on change control boards and manual, human scrutiny.
7. Were there any other cultural challenges that you’ve seen in your organization?
Change is hard, there is no question about it.
Something Gary Gruver said really resonated with me, which is the emotional journey. As a leader you need to lead people through the emotional journey of this transformation and I definitely see that every day. I mean people have emotional attachment to their work, to their body of work and the new idea you are telling them is “Ok, the world is flat now, the world is not round”. I think for some people it takes some time for them to digest, before they come around.
So there is some patience needed with that but also approaching them with the sensitivity that this is different, that it is going to impact the work that they are doing. Just that sensitivity I found does help. Also I think the whole issue with trust, building trust - that’s also emotional. Being the trusted advisor takes time, to build and earn the trust. So just understanding that this is emotional for people, it’s not just about numbers and metrics and work, I think that we have to take that into account.
The greatest achievement – so we just recently looked at our user count in our Continuous Integration (CI) tool, and two years ago we had seventy plus users in IT, now we have over seven hundred. So a tenfold increase in two years of people who were not on this journey before. From a pure numbers perspective our penetration looks like we are on the right track.
We’ve had a couple of amazing success stories. One in IT where, after adopting continuous delivery, they went from releasing every eighteen months to cut it down to like six to eight weeks. For continuous delivery that’s not dramatic, but what they were able to do is the concurrent decrease in the number of defects as well. They went from triple digit defects, to single digit, to almost disappearing defects. They were very disciplined in tracking the metrics in their defect removal efficiency. So you were able to see that after they started adopting more automated tests, they were able to shift left, discovering more of the defects earlier. To the point that eventually all defects just disappeared from the environment. That painted a very powerful picture of transformation and what DevOps can do.
And on the product development side we have this large project initiative. I can’t take all the credit for it because they were already well on the road before we started this initiative. But they have over a few thousand developers across different sites, and they have a hundred percent code coverage as a pre-commit mandate. You can’t commit your code unless you have a hundred percent coverage. They are able to have releases to our key Intel customers like Samsung on a daily basis.
Manuel:
And any failures that you want to share? Those are part of it too.
I won’t call them failures, you only fail if you stop trying.
We’re constantly evolving, so some of the opportunities on our horizon where we want to put more focus on is the user experience. Because with continuous delivery, with DevOps, you now have to work with a lot more tools than you had before. Automated testing, unit testing, you have to deal with dependency injection, you want a dependency injection library and a mocking library, and so on and so forth. So up to now there is still a lot of stitching together these tools, and people focusing a lot of their time just working with the tools, getting them provisioned, rather than focusing on writing code and writing tests.
We want to make that part easier, just a one click provision to get all the tools they need integrated so they don't have to deal with the integration challenges. Focusing their time on learning the tools and writing better tests and writing better code. So that is on my to do list.
The other item on my to do list is analyzing our various DevOps practices adoption across the enterprise. We have pretty good adoption of continuous integration and continuous delivery but what we don’t have currently is infrastructure as code. Codifying our infrastructure, codifying our configuration management, that’s on my to do list to change that.
I think a lot of it is pain, people are sick of dealing with everything being so painful... I am trying to think of examples. We were touting a lot of the benefits. We have people working weekends, we still have big launches and people on launch calls, and coming in weekends because it requires downtime, you have to work after hours to minimize impact in the business. We are hoping that the benefit we're touting is the release can be a non event, can be less dramatic than what you are experiencing now. And most recently the pain of all the scrutiny, the change control.
I think we are offering an alternative that, if you have an automated quality gate setup, then you don’t have to talk to anybody. If you meet all the requirements and all the requirements are codified, then you can release as frequently as you need.
10. I'm not sure if you are familiar with the 2015 state of DevOps report?
Yes, read it cover to cover.
Yes, absolutely. The example I mentioned earlier, this one team in IT, they were able to reduce defects and at the same time going faster.
We also heard another testimonial where this team that was in a meeting with our end user and they requested some change and before the end of the meeting that change was already in production. And that’s not something we could remotely do before, so it is a real game changer. That’s a good story for us to tell others that “Hey, imagine you have somebody yelling at you, and you get in a meeting with them and before the end of the meeting you already solved their problem”. Think about how much better their experience could be, so we are definitely witnessing, we are hearing these success stories.
Yes, absolutely. Right now we have a couple of case studies and success stories that we try to manually collect a few metrics, usually around the three axes of velocity, quality and throughput.
With velocity we try to look for their frequency of release now and before, and also what is their cycle time before and after. On quality, usually that’s the easiest to measure, there’s the defect rate, how many defects, how many incidents.
Throughput is a little harder, it’s more subjective. How do you know that you’ve increased the throughput of the team? How do you know the team is more efficient? We ask questions like percent of time spent on value-added versus non-value-added activities, where value added would be new features and non-value-added would be time fixing defects or dealing with the process.
13. Do you consider working on the pipeline and the tools to be non-value or value activities?
I would consider that value added, because I think customers would pay for a higher quality product, with less defects and faster. We are trying to measure the percentage of value-added versus non-value-added and it’s subjective. It’s basically interviews. So those are the three primary things that we measure manually.
We are also working on an effort to gather metrics automatically so we don’t have this manual gathering process. I didn’t mention that we developed a maturity model. That’s how we measure where everybody’s journey is in their DevOps adoption and we try to correlate the maturity model with things like the defect rate and velocity. That’s another thing on our to do list for the future.
15. Can you give us a brief explanation on how is it based?
There isn’t one officially available in the industry so we decided to create one. When we started the effort we didn’t set out that as our intention. But what we found was teams were coming to us and say “How do I get started? This benefit is great, chaos monkey is great, but how do I get from where I am to there?”.
At the same time we also have other teams that say “We are already doing DevOps since our developers are carrying pagers. We're doing DevOps, we're fine, now go away”. Or “Hey, we are doing cloud. We’re in the cloud, cloud is DevOps, right? So go away” kind of.
So we realized that we have to have: one, something that helps people, guides people on how to get to their destination; two, kind of set everybody on the same context on what DevOps is. We spent a lot of time on defining what kind of practices we want to incorporate in our maturity model. We finally came up with four categories: if you say you are doing DevOps at Intel you have to be doing continuous delivery, continuous integration, Agile practices and some sort of continuous process improvement. From there we further break down to ten different categories. For each of those categories we have levels one to five of maturity.
How we use that is for teams that come to us and need help in this journey, we’ll do an assessment with them and help them set the next target condition based on the maturity level described and guide them accordingly.
Challenges and road blocks... I mentioned a couple of them, right now getting people on board requires integration of all these tools so that experience is not easy, so it’s definitely on our list to try to make it easier.
And again I think it’s just that change is hard. We just continue to educate people, to get the information out there and sometimes be patient with them, let them go through this emotional journey themselves, and come around.
Manuel: Thank you very much Sherry.
Thank you.