I am the lead of the Build and Release Team at E*TRADE. Everything that the organization produces, from the website to mobile clients, trading engine, settling systems and so on is built by my team and deployed by my team to different testing environments. We also produce other tools to help change propagation. We define the systems through which changes are going to flow through. We also produce tools to aid in maintenance of the systems such as deployment. We also work with regulators in case of any audit or anything else like that with respect to source code. So we have a lot of responsibilities.
2. How and when did you first hear about DevOps?
DevOps ... It’s a good question, probably about four years ago or so. I was first hearing about continuous delivery and continuous integration, probably continuous integration first. DevOps was probably four years ago.
3. How did DevOps get started in your organization? What were the first steps taken?
That question is like a chicken and egg question, right? We had efforts already to push incremental changes into testing environments. We always had a methodology for pushing incremental changes to the production environment when an emergency arises. So if there’s an emergency, there’s always a procedure to fix things quickly and reliably.
The question was: how can we apply the methodology to testing environments so we can push changes more often and get more value of the testing environments? As we were doing that manually, we realized we needed to automate that. It’s kind of a natural evolution of the need of having to change the environment more often as agile practices become more widely entrenched.
Well, first, it was a grassroots initiative. In fact, for one of the most valuable tools that we have, there wasn’t any project behind it or anything like that. I saw the need for what my team was doing. I wasn’t the tech lead at that point but I saw what my colleagues and I were doing day to day and I realized “we need to automate this”. We can’t be pushing this rock up the hill all the time.
So I created a tool for that and it has taken off from there, basically. However, after the value was exposed, we realized that we needed top-down support to really take this set of practices company-wide. It’s a two-pronged approach, basically, from the bottom and then from the top.
I guess I’ve been lucky enough to be in an organization whose current leadership has seen the need for organizational restructuring. That actually has put us in a better position to implement DevOps practices. It’s not like DevOps was the driver for any kind of organizational changes. I think the organizational changes have instead enabled us to better enable DevOps. Something that I said during my speaking engagement here was that even the communication and the structural organization is very important for DevOps.
The most important thing to understand in Conway's Law,as far as how the organization is shaped, is that it doesn’t only talk about the HR structure of the organization, the hierarchical structure you see when you log into the Intranet site, but also the communication channels that are opened between people.
So, even if certain people that I communicate to are not on the same team officially, I have made an effort to open up those channels and create those links. So, for all intents and purposes, we are working as one entity, regardless of what the organization, from an HR perspective, says. They can later on catch up with us.
The first way of disseminating it was to have a success story. We focused really intently with one group and we increased their productivity by orders of magnitude. We gave them tools so they can be self-sufficient, and then we started spreading the example around the organization.
What we’re doing now is we’re having a conference – actually, this week – that is very similar to this. It is kind of like a DevOps Summit where operations and development comes together, and we start figuring out where the friction points are, how do we need to solve things, and how can we get there. And bringing this success story to the table is very important to highlight the DevOps practices and communicate them across the table to development and operations.
Yes. There have obviously been culture shocks. What is more interesting though is that the teams that tend to have the worst culture shock are not those typical teams that you might think of, audit or compliance. In fact, if you’re able to successfully communicate to them what you’re doing, DevOps and all of the associated practices seem like common sense. People say, “Why weren’t we doing this before?”
The problem that I’ve seen with most cultural shocks is with very entrenched engineering practices and architectural practices from IT people themselves. They don’t want operations to be telling them how to do their job, and vice versa, right? So I think the biggest cultural shock comes from really long-time developers that might be uncomfortable with change. They’re all for progress but they don’t like change.
8. So far, what have been the greatest achievements and also failures in your DevOps journey?
Let me start with the failures. There must be something. There are certainly things that we have learned but it’s kind of hard because when you start a DevOps approach, there’s only one way to go: forward.
I think probably the training. In the very initial stages, we didn’t want to burn our credibility. So we knew we had to support the pipelines that we were building for different teams. And maybe initially, we didn’t put enough weight on it and they initially falter for a little while because people thought that this wasn’t going to scale or that it wasn’t going to be a workable model.
But once we realized that we needed that credibility and it was essential for us to move forward, we started to dedicate more time and resources of the company as well. I was lucky enough to have managers that are above me that directly saw this and helped invest in increasing the infrastructure for DevOps. We are now able to help more teams deliver faster.
The success story is one of our flagship products team, the Stock Plan Administration. We’ve been implementing continuous delivery practices for them, and they’ve been able to increase their productivity about 1200%, in real terms. And they were just selected the number one Stock Plan Administration application for the fourth year in a row. So I’m really proud of having been part of that story, basically.
Yes. One of them is creating ownership within individual teams. Usually, there are always development teams – when you start going around in the company – that are about four to five people that are working very closely together. When you get onto that level, you need to grab one of those people and make them your point of contact, your SME basically. As far as what a delivery pipeline looks like, how it works, and train them in whatever technology you are using, whether it’s Jenkins, ElectricFlow, or anything else that you’re using. They need to be experts at that, and you need to interact with them very closely when you’re building a pipeline for them. That way, whenever that team needs to make a change to that particular pipeline, they have an expert within feet of their desk to talk to them.
So it’s, again, the two-pronged approach. Executive support but also low level grassroots understanding and support and commitment as well. I’ve been very clear with all of the teams that I approached that this will involve a time commitment from these people. This is not something that you pay and get for free out of the box.
Of course I agree and I guess the example is one that I already mentioned. Our corporate services group online application that has been rated number one for several years in a row now.
Changes that are happening through E*TRADE that customers are coming to me and telling me, “I’ve noticed there are more changes flowing in” where we're redesigning a lot of our strategic pages to deliver better tools. That has been enabled in large part by the pipelines that we’re creating to empower teams basically.
I don’t want developers to have to interact with any “support organizations.” They need to be self-supporting. And in fact, in theory, the only thing they need to be supporting and reacting to is feedback on the code they’re building. Building features that are driven by business, and therefore, also by our own customers. The environment should be as frictionless as possible, and that’s what I think we're creating. Slowly but surely.
11. So you reduced cycle time, and therefore, business changes can be delivered faster?
That’s right. Not in all layers of the business, but we're building on that. We have reduced cycle time in a particular team. We’re building up on reducing cycle times in many other teams and we’re giving them the tools that go with that.
12. What kind of metrics or feedback do you collect to validate the value of this investment in DevOps?
The fact that we have heterogeneous environments right now where some of our applications are on the more traditional waterfall model – where we have more sparse deployments –, and we have DevOps teams, we are able to compare a couple of metrics.
One of them is leakage of defects into production that result in emergency changes. And also the amount of deployments that the teams can make. We have concluded that the team that is doing faster and more frequent deployments tends to have significantly less defect leakage into production. They have less emergency changes. Likewise, they’re able to obviously deliver faster because they are going through faster cycles.
13. My final question is about the challenges and maybe roadblocks that lie ahead in your journey?
I think there are two main challenges, which is obviously the common theme with highly regulated environments.
One of them is the cultural aspect, in particular, with fellow developers and fellow engineers to whom this practice of DevOps might come, as you mentioned, as a culture shock and they're maybe resistant at first. One of the things that I wish that could be implemented is mainline development only, ban all branches. That sounds a little extreme but Gary Gruver from HP was mentioning that and people thought he was crazy and then they loved it.I don’t have the power yet to implement that but I’m pushing for that. If not, I’ll settle for, at least, short lived branches.
Secondly is the regulatory aspect. Obviously, we need to partner with audit and compliance and external auditors to make sure that the right controls are around that and they understand how segregation of duty is implemented in an automated system. That is the second part. Communicate to them clearly the fact that these practices are not an anathema to risk management.
Manuel Pais: Thank you very much.
You’re welcome. Thank you for having me.