Yes, sure. So I'm Aimee Bechtle. The MITRE Corporation is a federally-funded research and development corporation. The government provides us with funding and sponsorship to do research and development to help them solve some of the country's most critical problems. We are a company of systems engineers. We are a not-for-profit corporation of about 7,000 employees. We're a very risk averse company. We are subject to intense scrutiny from the government as to how we spend our funds.
Manuel: Your talk here at Agile Conf is focused on the DevOps transformation that you have been a part of in your organization. Can you talk us through first how you recognized that you needed to change, particularly in that kind of environment, as you said, usually risk averse and very controlled? Maybe even not only your organization but your end customers as well.
Correct. We recognized the need to change as some of our application development teams started practicing Agile practices. They reached a bottleneck when they came to my services. I run a release and deployment services team that holds the keys to the kingdom of the test and production environments.
As their cadence got faster and they were delivering releases and developing faster, they hit a bottleneck and slowed down when they came to us because we were manually deploying their apps. So they got increasingly frustrated, started exposing the bottleneck and the length of time with metrics on how long it took and then comparing to how long it could take and started threatening to find alternative ways to go around us.
We started investigating their suggestion to look into continuous integration, continuous delivery and DevOps practices to alleviate the manual processes and the bottleneck.
3. Did you face some resistance to start that kind of change inside your team?
I did a little bit inside my team as more of a fear factor, being afraid they might be automated out of the job. But instead, there's still always a place for manual deployments. So I just split my team and cut some people on the manual side and then developed new skill sets and transformed some people now into release engineers who got more technical and innovative skills.
It's not that much different. We're not allowed to have profit or revenue, obviously. So we have to show more accountability for our spending and we have to reinvest back into some of our programs and portfolios and initiatives.
5. In terms of cultural aspects, you don't think that there's a big difference?
There's a little bit of a difference. We do have more of a government culture than a lot of for-profits. So maybe deadlines aren't as hard for us. We're not driven by commercial profits and trying to beat people to market. It's more of an internal responsibility to deliver on time, be more conscientious because we are dealing with the government's money. We want to be cost-conscious.
We might invest more in open source or we might really look at how long a project is taking and whether we're being efficient or not. So just because we're not-for-profit and we do have a government mentality or culture doesn't mean that we don't look at efficiencies or cost-effectiveness.
Yes, it does. We are still working on taking down our silos even within Dev and Ops. While we implemented DevOps more for application development, we didn't focus on infrastructure until recently. So now, infrastructure's coming into the fold and starting to adopt DevOps because the silo got even worse when we automated the application and database builds and deploys but not the infrastructure.
7. So are those the main changes that you're working on in terms of technical changes?
Yes. Similar to when you renovate your house. You buy a 1960s house. You renovate everything except that one bathroom. Then now, you're looking at a modern house but then the bathroom is still pink and blue tiles. So we were staring at the pink and blue tile for a long time and now we're getting rid of the pink and blue tile.
It was really easy to get the application developers who were doing web application development into the mindset. There's other areas, for example, ERP areas, PeopleSoft, Oracle Financials, they're not into Agile as much and DevOps. So they're not as focused on improving or changing their application delivery process.
But on the infrastructure's side, we have a lot of very traditional systems administrators who are very task and command line-oriented or shell scripts. They need to - and have - stepped up to get new skill sets and start learning how to do automated provisioning, automated configuring, using Chef, Puppet. But it has been a hard sell, we have been trying to evangelize them for about a year now. They've just recently brought in expertise to help them.
They are just results from the changes that we made. They were not set up. We went in more as a grassroots movement.
10. How did you measure those numbers?
We used the legacy ticketing system that helped us manage the manual process before. So from the time that the ticket was created until it was closed, we measured that to compare it to the time that code was committed to our source code repository and implemented to production.
11. What were your business and customer's reactions to this reduction in delivery time?
We're a victim of our own success. They've embraced it. Again, most of the web application teams, almost all of them have moved on to it. I don't have personal experience exactly with what the customers are saying because I'm more in the background of things. But I would venture to say that they're pretty happy. They're getting their feature requests, some of them within 24 hours from making the request.
Yes, sure. We're really far with automating Linux VMs, about 100%, I believe, they have the automation for a baseline VM pretty much complete. It's missing the request interface for a user to just press a button and fill out a form. But that's coming.
Then we're 50% to 60% with automating our windows infrastructure or Windows VM. We are having a learning curve with Microsoft's DSC technology. We're also learning Systems Orchestrator, working with that technology.
Yes. We did a lot of Chef training, sent a lot of people through Chef. The DSC has been more learning as we go and self-taught. What we did with was we went out and secured professional services. We've had a consultant with us for about 14 months now helping us get the whole infrastructure set up and the entire continuous delivery pipeline for both applications and infrastructure end.
Yes. So we only have three products that are really audited and that's our time recording application, Oracle Financials and PeopleSoft. PeopleSoft and Oracle Financials have not moved over to automation. So auditing was focused more on the time reporting system. We set up our continuous integration and delivery system with a lot of traceability and logging and querying, being able to query as well, and to provide the evidence to the auditors.
So we used reporting tools, basically. To be honest, I'm not familiar with monitoring. It's a separate part and the auditors are still with the infrastructure folks on that.
Absolutely. I actually think that automation is a boost or an augmentation to the ITIL process. I think it's comforting to know that your tests might be automated and repeatable and have more coverage, more so than subject to a human-being testing, with human error and bias. And automation is more repeatable and, as far as ITIL goes, we don't have the incidents and the issues because of our environments are matching and our deployments are cleaner.
So that part has been more married with ITIL but one of the things that we're doing is... I'm the Enterprise Change Manager, so I'm responsible for how changes are defined and what category they fall into. We are enticing people to use automation, to be qualified as a routine change and not have to go through the approval process because they meet the criteria of being scriptable, repeatable, rollbackable, and that's actually less risk than if it was a manual process.
So what used to take up sometimes on a Wednesday night - which is our manual change deployment window - up to four hours and that's if it was successful - if it was unsuccessful, it could take all night - has now been reduced to less than 30 seconds at 6:00 a.m. on a Wednesday morning or any day of the week because we've changed the change windows.
A person who was normally working a Wednesday night deployment is now putting their kid to bed and reading them books. Or if they're spending all weekend doing a deployment and then rolling it forward, rolling it back, they might be spending an hour now on a weekend morning. Availability's up and windows are short.
19. Are you aiming at a sort of continuous delivery?
I want hot deployments. I want no downtime. I want to do it at lunch and have nobody work off hours. That would be ideal. Of course, there's always going to be the COTS products and the big ERP packages that haven't moved to the model or are difficult to handle in that model, that we can't avoid.
20. What are the next steps in this DevOps journey?
To continue focusing on automating infrastructure and provisioning. Transforming our skills to be everything as code. To continue expanding the continuous delivery to support better deployment models, the blue green or hot, whatever you want to call it. So that there is higher availability, less risk, and people aren't working out of hours.
Manuel: Thank you very much, Aimee.
Thank you.