Sure. So I'm Application Delivery Lead at CSC. What I do there is, right now, I work with a major program with a federal customer. I have a team of 300 people, that includes 28 software development teams and I've got an operations team and a security team that all work for me. Then I also work for our sort of functional horizontal for applications. I work with a lot of other customers as well.
2. How and when did you first hear about DevOps?
Actually, I was in the software for a long time. In 2010, I took a role as IT Director of an about 500-person company at the time. When I got into that role, I was supporting development and I was trying to build out the development environment I would have liked if I was a developer, from an infrastructure standpoint.
It made me appreciate the actual Ops side of it because now I was doing that and I understood why all of the Ops teams that I ever worked with before didn't want me doing the stuff that I was doing to the servers.
At one point, one of the developers that I was supporting in Agile teams said, "There's this thing called DevOps, have you heard of it?" I was like, "No." He said, "You should. You should definitely look into this." It really just came out as a concept, this was 2010. It was really early, 2009,it just happened. I started researching and reading into it and having been someone that had crossed the chasm, I was like, "Oh, my gosh. This is it. This is that missing link I've been trying to figure out for forever." So that was a really great way into that field.
I feel like the first steps, it was like this evolution of a lot of things that we sort of did. In that particular role where I was IT Director, one of the things we did is we actually built out this like Dev Lab for development and test environments for our project teams.
When we were doing that, one of the things I did is I just got everybody in the room. So I had the people that were the build masters in the development team. I had our lead architect. I brought my Ops team and we designed what that needed to look like. We were both responsive to Devs team telling us, "Well, we want the ability to do dynamic provisioning stuff and do all these things” and our Ops team having to go “but we need it to run on some standard hardware because we don't have an army and we need you to help us."
Through that negotiation, that kind of helped us get in place. But actually, one of the most interesting things we ended up doing – I think at this conference, we're all friends with Gene Kim – is the head of Service Test came to me with the “Visible Ops” handbook. At that time, we had a lot of fragile animals, which is something from that book. We kind of took that methodology of "Okay. What's fragile in our environment that we need to stabilize? How do we stabilize it? How do we fix it?"
We implemented a collaborative change control process. It's sort of a change board but it was just us as a joint group, the developers and the Ops teams, meeting for a change discussion. So it wasn't like the Ops team is doing stuff and not talking to the developers. It was all of us in the room. That was kind of a great first step for me, transitioning into that mindset.
Well, I was IT Director, right? So I was in charge of all the IT of the entire company. That includes supporting. I had the executive support. I mean, I had the support of the leadership to do this, including the funding to do sort of this research project. I wouldn't say I was driving it but it was really both, right? I wanted to see this happen because I was committed, as somebody who'd been on development, to really making it work for developers.
But I also had a really great team of people who had been doing this grassroots stuff. So I kind of found the people that were already doing grassroots and I pulled them into my IT work. That's, I think, how I approached this. I found this sort of friendly developers who were going to help me in IT to do this transformation.
But a lot of it was us driving it from IT, which I think is unusual because often people talk about driving from the dev side. In that particular case, we were really driving it from IT. Sort of the other way around.
Absolutely. So I work at a federal agency and the civilian sector right now. We do sort of the systems service. For us, DevOps is a natural transition. We were doing Agile but obviously that wasn't enough and that was really locally optimizing. We're involved in the operation side as well.
Taking it to its natural conclusion, to really get better release management through DevOps. We've been really focused on, in that particular customer, the automation side. I will describe the cultural change as we've sort of already changed the organization. We made an agile migration to be more cross-functional.
We have teams, they have business people, they have developers, they have testers, they have Ops people all in one team. That kind of already happened.
One of my teams actually did a migration to Amazon web services, which was really successful for them. So now, they're 100% on Amazon. Their deployment is completely automated, completely infrastructure-as-code, it's phenomenal what they've done.
But getting other people in the organization to realize what that changed for how they work [was hard]. There was another team that wanted to test with them. They were like, "Hey, can you spin up an instance of your latest build so we can test against it?" The sort of government product manager went, "Oh, that will take two weeks."
Their mentality was of course this will take how long it probably used to take – but it probably didn't used to take two weeks. But the team was like, "In 30 seconds, it's done. Who should I give access to?" So the technology is always ahead of the people. We have the technology to do things well ahead of when we were mentally ready to go faster.
I think that's maybe the organizational change we're struggling with a little bit now. The automation is there, although we have more, we're constantly introducing it. It's changing how we work at a really local level at this point. It's really changing how teams work and how we collaborate and how we look at the customers even.
One of the other successes we had recently was convincing a customer, and the union. You don't just deploy software, you have to get the permission of the union to deploy software. We've been working on this release for a while and we finally got their permission to do a silent release. We can actually deploy this big release and then we can just deploy per feature and you'll never know. When you're ready for us to turn it on, we'll turn it on and we were able to convince them to do that and so they did.
They pushed a big deployment out to get that feature into production. We knew it didn't destabilize things. Then, as they've been adding features, adding reports, that kind of stuff – they're at the tail end of their product backlog – they've been pushing those into deployment. Deployment takes longer to email than to actually do the deployment. The business doesn't know, it's like quiet. Getting them to agree to that was, I think, one of the most significant organizational changes we've had recently: that they trust IT to deploy multiple times a week.
I mean, that's a dramatic change from where they were just a few years ago, right? I really don't think the business trusted us.
Yes. One of the things I've been trying to do is kind of collaborative learning with our teams.
One of the things that went really well this summer was I took the teams that were pretty far along in their automation or testing – because you know I have a lot of teams but not all amazing. I mean, they're all amazing. They're not all doing high-end DevOps amazing things. Then there are some that have more technically challenging environment than others.
What we're doing is we're taking the teams that are farther along in their automation journey. I had the lead tester from a team that had gone through this transformation to automated testing, stand up in front of all his peers and explain what it was like. He said, "The first time we ran a test it took like a week. It was really hard. Then I figured it out, I can do the same thing in 30 minutes and now we run this test and we're done" I let somebody else stand up and say, "Actually for me what really helped is that I sat down with the developers and they helped me learn how to do this. Once I started working better with them," you know.
So they gave their testimonies on what was like to go through the transformation. I thought it was way more powerful than anything I could have done in terms of helping other teams do it. So we want to do more of that. I'm trying to find ways that we can have people help each other across the program and kind of learn better. That's what we're trying to get to because it can't just be like “go take a computer-based training” or “go see a lecture”. It's got to be really a discovery of learning this.
Yes. I mean, there's always those groups that are concerned. Security in our world is very significant, for sure. We deal with a lot of privacy and now there's security data in our agency and that's top of mind.
We were able to bring the security group in collaboratively and they run tests on the developers development environment, not just their production stuff.
The fact that they're allowed in earlier has been really helpful but there are still a lot of people in that process/function – I mean, not all of them but there's quite a few of them – that I would describe it as, their job for so long was to sort of be the “no” stamp. They had this process and it's all about following this process.
Getting them to a place where the point isn't to follow the security checklist, the point is to build a secure app. I think my security team is really great. They really understand the core goal. The goal is to build a secure app. But we hit random people in the other parts of the organizations that don't share that goal.
Once we get out of that nice collaborative team environment, every now and then we bash up against, we have Change Control Boards at agency level, we have gates and things. For the most part, we've been really lucky being an agency that has minimized that, but we still hit them and it's very frustrating to kind of hit those processes.
It's a constraint, we just work with it. I hope we'll get to a point someday where we can effect a greater change and we can get to a better place of control where we don't feel like we need to have a gate or a meeting or a process, or you can't do this until you draw me a Gantt chart, and all these other things.
Maybe we'll get to a place where we can get away from that. But for now, it's a constraint that we just work with. We're still able to accomplish a lot even in spite of some of that. That's so true anywhere. You're not going to get rid of the bureaucracy on day one.
8. What have been the greatest achievements and failures, so far, in the journey?
I guess I'll go with the achievements first. The greatest achievement I think has really been seeing the teams figure this out for themselves. I am really impressed. Two of my teams did some really amazing things that I was impressed with just how much they figured it out on their own.
The team that migrated their application out of the data center to Amazon, I kept worrying like, "Oh, my gosh, they're going need somebody to tell them how to this." But they didn't, they didn't at all. Actually they completely knew how to do it. They had this very learning mentality and they really approached it great. Seeing a team that's a great cohesive team and a really good learning team do that, it's really nice. Not just what they did technically but how well they've approached it as a team.
The same for one of the other teams that are in pilot, they've really been through quite a journey in terms of both improving their overall Agile but then taking this DevOps transformation. Something that previously, for them, used to take 48 hours – to get a staging environment – they've got it now. So when the developers are code complete, the full test suit tuns. Runs the everything test, it's like 15 minutes. Runs a deployment, that's 12 minutes. The test team verifies and tries to kick the tires and when they say done, it's in staging in 12 minutes.
That used to take no less than 48 hours. For them to make that kind of transformation, that they can basically be done coding and showing it to a user under an hour is pretty amazing because that lets them get that shorter feedback loop. It's changing how they work. It's helping them.
I mean, there's still things that they want to fix, they've got a whole list of things that they want to keep going on in their improvement journey. It's not a destination. It's a place they're still going to but I think it comes back to watching a team that's really working well, improving itself which is the most satisfying.
Failures. I love failures. In the times I guess I've done the transformation, one from a previous project was that it was kind of a mixed blessing. It was a success and a failure in that we successfully did this sort of technology side of DevOps. We migrated the agency website to AWS. We moved some key apps, some key portals, all customer facing, into that environment.
We did it with weekly deployments. We did all of the automation, all the great things, DevOps things. Unfortunately, I think that I didn't affect the culture change in that organization that I wanted to see. I kind of reached a point where I really had to walk away because I was that like DevOps flogging them a little too much.
It was a little disappointing for me to sort of have to walk away from that knowing that although we technically accomplished what we set out to do, we haven't really seen the cultural change. I checked in with that team and they're still struggling on the culture.
I feel a little happier about the place where we've made the culture change and the teams are really changing than the one where we did this amazing automation but nothing changed in the culture. So that, to me, is a little bit of a wound that I didn't succeed there. But I learned a lot from it. I learned a lot about how much more I need to focus on the culture and not just go and doing tech stuff.
Sure. We take a lot of data. We've got corporate metrics and metrics for my own project. We have not sampled every project in our company for anything but, just on average, most of our projects that migrate to DevOps deployed between 200% to 400% more than they did before.
We see quality increases on average go up into the high 90s. We see with just some of the testing automation, we did the math across six projects that automated testing. We took the level of effort they spent writing the test. Sometimes I estimate it, sometimes from story points and other things that we are capturing. Then we took the actual number of times the test ran and then put an equivalent of what it would have been if we'd have run that test manually. Our average calculation across all those teams was that, on average, automating saved us six FTEs worth of work for every one tester.
It's not that we replaced testers. We just tested more. We see the real benefit in just straight numbers. So I have no doubts, like in that example, from 48 hours to 12 minutes. I mean, that's a crazy improvement. People are like, "Really?", and I'm like, "Yes, really. I can show you." It's real. The team did a great job of building that out. It didn't take them long to do it and it has an amazing impact on them.
I have no doubt that that's not an anomaly, that is exactly the kind of improvement you get with DevOps.
We measure lead time and I measure total cycle time and that kind of stuff. But I think one of the most creative measures was by one of the teams that we were trying to do this testing pilot with. We were going over the usual measures, code coverage and that kind of stuff but we wanted to make the culture change.
We challenged the team: how would you know that, as a team, you've changed how you work? That automated testing, test-driven development, and acceptance-test-driven development, this is how you work? And how do we know that?
Somebody suggested, "Why don't we make like it a sentence, like an oath? At every retrospective, can you sign up to the following statement: “I believe in testing and I do it all the time and it's how I work." I can't remember the exact wording of it but that's what they came up with.
This oath that says, "I understand it, I believe it, and I do it." If 100% of the people in the team say it, then we know it's part of the culture. Right now, the team is somewhere around 60%. I think they're stuck, they haven't quite crossed into it. They know it. They do it. But they're not quite like "I wouldn't go back."
That's been a really good measure because that's how I'm measuring – not test coverage because I can tell, test coverage is out, that's great – but that's how I'm trying to measure that it's really affecting how they work as a team. I like that they came up with that as an idea. I definitely think that's a good way to measure more than just the numbers, though I love the numbers too.
12. It tells us a bit about why the people would go back or not to their old ways of working?
Yes. It tells you whether you are doing it because I told you to do it or you are doing it because you've really seen the value on it. What sort of created an up-tick was there a particular sprint where they broke the build and testing saved them a little bit and suddenly a few more people agreed to that number.
I mean, I did this myself, right? I was a developer and I learned to write automated tests. So having been through that, I know that it's not like "Oh, this is great." I thought “This is really stupid. I don't know why I'm doing this.”
I know that from my own journey, that you have this a-ha moment but you don't get it right away. So you have to kind of wait until people get it. That's, I think, a good measure of "Are you there yet? Is it really in the culture?”
In my current role one of the challenges is that we've run out of “easy”. We solved a lot of the easy things that were in our domain to solve.
We've actually done a lot of collaborating with some other contractors on the field that we work with that are part of our delivery cycle but not part of our team. But there are some groups that we haven't really touched yet. So I'm trying to figure out how to collaborate better with them and their process and help them out. I think that's kind of the next journey.
For example, networking is not part of our team. They don't sit with us and they're different. So it's one of the harder groups for us to collaborate with because we're not like hanging out with them every day. That's an area that we certainly have to tackle and other areas of the operations, other places that we touch, applications that are outside of ours, other agencies even. It's harder because we don't have a natural relationship with those groups.
But trying to facilitate those conversations and solve those problems, that's where we're trying to hit next. And scaling, a little bit. I've got some good teams and I've got some teams that are still trying to get on to this way of working. Trying to help the teams that don't know how to do this yet learn from their colleagues. I think that's kind of the next focus area.
Manuel: Thank you very much, Paula.
Thanks.