2. Thank you for accepting our invitation. Can you tell us a bit more about your current roles?
Evelijn Van Leeuwen: I'm a manager at infrastructure parts and before this I worked in the Internet applications development part. I was involved when we started with Agile development and Scrum five years ago and then went over to infrastructure to do real DevOps.
Kris Buytaert: I'm Kris. I work for Inuits, which is one of the larger open-source consultancy groups in Europe. I'm one of the co-founders and I'm basically one of the senior guys around there. Within my role, I do a lot of open-source consulting and helping people to adopt better DevOps practices.
3. I guess this question is more for Evelijn. How and when did you first hear about DevOps?
Evelijn Van Leeuwen: I was thinking about it, but I 'm not sure because we started with Agile and Scrum and then we were speeding up really fast. Then we found out really quickly that DevOps was needed. But I do not know when we DevOps was started. Do you know, Kris? Because you were there.
Kris Buytaert: The initial idea to start a conference was at Cloud Camp Antwerp in 2009 and Patrick [Debois] put down the name of the conference. He actually started it after he saw the talk at Velocity of John Allspaw. about ten deploys a day and Dev and Ops working together.
Evelijn Van Leeuwen: And I knew that presentation too, which is in the same layer because we had a problem bringing stuff to production. So that was why we needed it.
4. And how did DevOps get started at ING? What were the first steps that you took?
Evelijn Van Leeuwen: I think Kris was involved there.
Kris Buytaert: No.
Evelijn Van Leeuwen: No?
Kris Buytaert: No, because you guys were...
Evelijn Van Leeuwen: Oh, we were having the reorganization first. It was like a management insight. At the beginning I thought “Maybe this is not the way to do it”. You [Kris] spoke about it yesterday. I was thinking because of [DevOps being a] grassroots development, we really liked the way it went. But looking back, it was a really good decision because it made people think and it helped to break some traditions in our organization and processes that was necessary to break.
5. It was mostly a top down decision to go for DevOps?
Evelijn Van Leeuwen: To go for DevOps, yes. Because it was not really going fast enough.
Kris Buytaert: And then I got involved because a couple of teams were struggling with the idea and they were looking into how do we really do this. Ingrid Algra, who was also here [at DevOps Enterprise Summit 2015], contacted me and I started helping out her team and then I ran into different other teams and started helping them too.
6. What were the first things you helped them with?
Kris Buytaert: Basically DevOps 101 – What is version control? What is continuous delivery? How do you build a pipeline? Teaching them about infrastructure-as-code, teaching them about all those things. In a way, I was showing how I did stuff. I remember the very first day, this guy Marco was talking about “Yes, we want to do stuff with Jenkins, we want to start building a pipeline, we want to start testing it” and he was struggling with getting Jenkins up and running. I said “Let me show you. Here's a Vagrant box... vagrant up”. Five minutes later – five because of the proxy issues and the protection on the ING network – I had a fully operational Jenkins server. For him that was something which he had never seen and being able to show people how stuff can work, that this is not magic and people telling fairy tales, I think that was a big part in changing the minds of a lot of people.
Evelijn Van Leeuwen: I think what you also did is something we did not do enough at that moment – bringing people together. Because you spoke with a lot of different people in the organization and found out who was doing the same things and who was not.
Kris Buytaert: I think that the point where I, as an external consultant, knew more people within the organization that were doing the same things than the people actually working there, that was pretty scary.
Manuel: You helped connect the dots.
Kris Buytaert: I helped to connect the dots in a way that people knew that I was around and they came to me because they found out that I was one of the people who had the same interest, who could help them further, who could show them things and they came to me.
7. And right now, what are the DevOps initiatives that you are working on
Evelijn Van Leeuwen: The whole DevOps movement kept on growing in ING, also on the business side, so I think that we did a lot of stuff there and combining teams from the commercial side with development teams.
I think where we were waiting too long was really the Ops/Infrastructure side. We waited too long to integrate infrastructure. I went over last year and you [Kris] also came with us some days per month and helped us build our pipeline at the infrastructure part because we did not do that at all until that moment. We had our automation in place, but not the config automation we needed.
Evelijn Van Leeuwen: I think the Agile movement was really a bottom up movement. Few people came together and the CEO really liked it and supported it directly. So that was really nice. That made it work.
But DevOps was a top-down decision and that helped. I think that till today there are still organizational changes, there are initiatives, there are experiments. You can find it all in the organization, growing this whole movement, making delivery faster and with higher quality. That is what we are trying to do.
Manuel: Are you working on disseminating DevOps in the larger organization, across countries?
Evelijn Van Leeuwen: Yes, we do.
Manuel: How do you go about it in such a large organization? How do you even know where to start?
Evelijn Van Leeuwen: We learned that we have more communities and bringing people together and internal DevOps days.
Kris Buytaert: And internal open-source days.
Evelijn Van Leeuwen: Yes, internal open-source days and hackathons. This year there's an international one. So we really try to combine knowledge and information and bringing people together, working more on open-source, reusing the stuff we make. That is going further and further. We cannot stop that anymore I think, so that's good.
Evelijn Van Leeuwen: There was a lot of resistance from a lot of people at the beginning, also from people in the engineering parts. Not people who were great engineers, but people who were still learning a lot and thinking whether this is going fast enough for them. So there was also a lot of insecurity and people were not sure they could cope with the changes we were doing. But I think it worked out fine or better than I first thought.
Kris Buytaert: Change is always needed. When you want to change, cultural shocks are ways to change people. In the case I just told you about Marco, showing a “vagrant up” in five minutes, for him that was a culture shock. It was just impossible. A lot of what I did was just showing people “This is how it works in the outside world”. That alone is probably already a huge cultural shock.
The fact that I am constantly talking about using open-source tools rather than using proprietary tools is a huge culture shock for a company that has a hiring problem – the only place in the country with that [proprietary tool related] job description is us. If you want to move from that to a point where people need to be employable, that is already a huge culture shock.
Evelijn Van Leeuwen: It was a shock for the whole organization because we had to find out that IT was our primary process and before that it was not. But it became more important, especially in IT, making software as a primary process and it wasn't before. So we outsourced a lot [before] and we thought it was not important enough. I
In the last five years, we found out that if you really want to make nice things that customers like, we need to do a lot [of the work] ourselves and have a lot of experience about it. That is a shock in itself because we did not have a lot of engineers working anymore in ING so we really needed to change that.
10. What have been the greatest achievements and failures in your journey so far?
Evelijn Van Leeuwen: I really like that engineers are coming back to ING and people develop themselves as engineers. I am really proud of that, that we are managing that. We mentioned that we waited too long to get the infrastructure [team] involved. Kris, I think your story about how long it took to find them is really relevant.
Kris Buytaert: I think it took me about two years to find Brent.
Evelijn Van Leeuwen: I think we waited too long and it was not necessary.
Evelijn Van Leeuwen: I like that we have a consistent story, also from top-down, that we are creating an engineering culture. That is really a consistent story so it is no surprise for anybody. People can chose if they want to be part of it or if they think they cannot be part of it and I like the consistency of that story.
Evelijn Van Leeuwen: I do. I think you are getting faster, if you do this. That is true. But I also saw examples of companies here who started later and are speeding up much faster – that is an interesting story to understand. I think that quality is getting higher, there is more knowledge in the organization. I think that is really important.
Kris Buytaert: Because you are getting engineers back, you actually get people who can fix stuff and understand how stuff is working rather than people picking up the phone and calling a vendor. That culture change alone makes sure that the quality of your stack is becoming better, that you can deploy it faster and actually build better and stronger software.
13. You have seen faster business changes?
Evelijn Van Leeuwen: Lead time or incident meantime to resolve, stuff like that, yes.
14. What kind of metrics or feedback do you collect to validate your changes?
Evelijn Van Leeuwen: With stuff like this [lead time and MTTR]. In the beginning we also used function points to show that there is really a difference between ways of working. That was also good for us. It worked, so we could show that it was also cheaper to do it this way, that it was faster and delivering more value.
Kris Buytaert: It is not a real metric and it is really hard to measure, but the metric I see happening with them is “frustration level going down”. I think that at the beginning you called me much more than now.
Evelijn Van Leeuwen: True.
Kris Buytaert: And each time was like: “Aaaa!”.
Manuel: Is the metric how many times you call Kris?
Kris Buytaert: “I need somebody to talk to. We are doing this. How would you do this?”. And now that is at a much lower rate. I mean we meet frequently but I feel the frustration levels are much lower now than two or three years ago.
Evelijn Van Leeuwen: We have a higher maturity than we had years ago because we really had to learn a lot and now we know that you get frustrated in this process, but you can come through it. Just by trying stuff and bringing it further. I think that in the beginning we really needed people to show us how we should do that. But now we know that we can do it, if we try.
15. Final question: what are the challenges and maybe roadblocks that are ahead of you in this journey?
Evelijn Van Leeuwen: For me, the most important one is that there is a lot of frustration involved in this whole change and the frustration is not going away. So, there are always new problems to solve or a new thing that you do not know how to solve. I have a background in Lean and when you know a lot about Lean then you see more and more problems. You are in a store and you –go “you can improve this process”.
I told people when we started “If you start to see it, you see only more and more and more [problems]” and that is true. That is sometimes frustrating because you think “Oh, I thought this was really improving, but now I see many more problems than before”. But I think that is a good sign.
Kris Buytaert: Because you are digging deeper and not looking just at the surface anymore.
Evelijn Van Leeuwen: Yes, that is true. We understand it much better. I always say that if you do Agile and you “just like it”, you are not doing it. Because in a big organization you find a lot of problems so if you think it is just fun, you are not deep enough.
Manuel: Thank you very much.
Evelijn Van Leeuwen: Thank you.
Kris Buytaert: Thank you.