BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Interviews Aimee Bechtle on Not-for-Profit & DevOps

Aimee Bechtle on Not-for-Profit & DevOps

Bookmarks
   

2. Thanks for accepting our invitation. Could you briefly introduce yourself and the organization you work for to our audience?

Yes, sure. So I'm Aimee Bechtle. The MITRE Corporation is a federally-funded research and development corporation. The government provides us with funding and sponsorship to do research and development to help them solve some of the country's most critical problems. We are a company of systems engineers. We are a not-for-profit corporation of about 7,000 employees. We're a very risk averse company. We are subject to intense scrutiny from the government as to how we spend our funds.

Manuel: Your talk here at Agile Conf is focused on the DevOps transformation that you have been a part of in your organization. Can you talk us through first how you recognized that you needed to change, particularly in that kind of environment, as you said, usually risk averse and very controlled? Maybe even not only your organization but your end customers as well.

Correct. We recognized the need to change as some of our application development teams started practicing Agile practices. They reached a bottleneck when they came to my services. I run a release and deployment services team that holds the keys to the kingdom of the test and production environments.

As their cadence got faster and they were delivering releases and developing faster, they hit a bottleneck and slowed down when they came to us because we were manually deploying their apps. So they got increasingly frustrated, started exposing the bottleneck and the length of time with metrics on how long it took and then comparing to how long it could take and started threatening to find alternative ways to go around us.

We started investigating their suggestion to look into continuous integration, continuous delivery and DevOps practices to alleviate the manual processes and the bottleneck.

   

3. Did you face some resistance to start that kind of change inside your team?

I did a little bit inside my team as more of a fear factor, being afraid they might be automated out of the job. But instead, there's still always a place for manual deployments. So I just split my team and cut some people on the manual side and then developed new skill sets and transformed some people now into release engineers who got more technical and innovative skills.

   

4. You mentioned that you work for a not-for-profit corporation. So how does that make a difference compared to a regular private company in terms of the DevOps adoption?

It's not that much different. We're not allowed to have profit or revenue, obviously. So we have to show more accountability for our spending and we have to reinvest back into some of our programs and portfolios and initiatives.

   

5. In terms of cultural aspects, you don't think that there's a big difference?

There's a little bit of a difference. We do have more of a government culture than a lot of for-profits. So maybe deadlines aren't as hard for us. We're not driven by commercial profits and trying to beat people to market. It's more of an internal responsibility to deliver on time, be more conscientious because we are dealing with the government's money. We want to be cost-conscious.

We might invest more in open source or we might really look at how long a project is taking and whether we're being efficient or not. So just because we're not-for-profit and we do have a government mentality or culture doesn't mean that we don't look at efficiencies or cost-effectiveness.

   

6. In terms of silo cultures, does that happen as well? Like each team focusing just on what they have to do and not so much on the end to end flow of work?

Yes, it does. We are still working on taking down our silos even within Dev and Ops. While we implemented DevOps more for application development, we didn't focus on infrastructure until recently. So now, infrastructure's coming into the fold and starting to adopt DevOps because the silo got even worse when we automated the application and database builds and deploys but not the infrastructure.

   

7. So are those the main changes that you're working on in terms of technical changes?

Yes. Similar to when you renovate your house. You buy a 1960s house. You renovate everything except that one bathroom. Then now, you're looking at a modern house but then the bathroom is still pink and blue tiles. So we were staring at the pink and blue tile for a long time and now we're getting rid of the pink and blue tile.

   

8. In cultural terms, were there any changes, maybe some practices that you adopted or you tried to raise awareness of the need for a DevOps mindset?

It was really easy to get the application developers who were doing web application development into the mindset. There's other areas, for example, ERP areas, PeopleSoft, Oracle Financials, they're not into Agile as much and DevOps. So they're not as focused on improving or changing their application delivery process.

But on the infrastructure's side, we have a lot of very traditional systems administrators who are very task and command line-oriented or shell scripts. They need to - and have - stepped up to get new skill sets and start learning how to do automated provisioning, automated configuring, using Chef, Puppet. But it has been a hard sell, we have been trying to evangelize them for about a year now. They've just recently brought in expertise to help them.

   

10. How did you measure those numbers?

We used the legacy ticketing system that helped us manage the manual process before. So from the time that the ticket was created until it was closed, we measured that to compare it to the time that code was committed to our source code repository and implemented to production.

   

11. What were your business and customer's reactions to this reduction in delivery time?

We're a victim of our own success. They've embraced it. Again, most of the web application teams, almost all of them have moved on to it. I don't have personal experience exactly with what the customers are saying because I'm more in the background of things. But I would venture to say that they're pretty happy. They're getting their feature requests, some of them within 24 hours from making the request.

   

12. In your talk last year at the DevOps Enterprise Summit, you mentioned that you were in the process of adopting automated provisioning and infrastructure as you mentioned. How far are you now and any lessons you'd like to share on this journey?

Yes, sure. We're really far with automating Linux VMs, about 100%, I believe, they have the automation for a baseline VM pretty much complete. It's missing the request interface for a user to just press a button and fill out a form. But that's coming.

Then we're 50% to 60% with automating our windows infrastructure or Windows VM. We are having a learning curve with Microsoft's DSC technology. We're also learning Systems Orchestrator, working with that technology.

   

13. Do you have a lot of training going on since you apparently have a lot of different technologies to handle?

Yes. We did a lot of Chef training, sent a lot of people through Chef. The DSC has been more learning as we go and self-taught. What we did with was we went out and secured professional services. We've had a consultant with us for about 14 months now helping us get the whole infrastructure set up and the entire continuous delivery pipeline for both applications and infrastructure end.

   

14. What about other aspects like system monitoring, auditability, and compliance? Did you, in your DevOps transformation, face challenges in these areas which are typically very controlled in federal agencies?

Yes. So we only have three products that are really audited and that's our time recording application, Oracle Financials and PeopleSoft. PeopleSoft and Oracle Financials have not moved over to automation. So auditing was focused more on the time reporting system. We set up our continuous integration and delivery system with a lot of traceability and logging and querying, being able to query as well, and to provide the evidence to the auditors.

So we used reporting tools, basically. To be honest, I'm not familiar with monitoring. It's a separate part and the auditors are still with the infrastructure folks on that.

   

16. In the past, ITIL processes and certifications have been accepted as signs of correct levels of control and traceability. In your opinion, can the mindset of ITIL, of risk reduction, cohabitate with the need for continuous delivery and automated deployments that you expect in DevOps?

Absolutely. I actually think that automation is a boost or an augmentation to the ITIL process. I think it's comforting to know that your tests might be automated and repeatable and have more coverage, more so than subject to a human-being testing, with human error and bias. And automation is more repeatable and, as far as ITIL goes, we don't have the incidents and the issues because of our environments are matching and our deployments are cleaner.

So that part has been more married with ITIL but one of the things that we're doing is... I'm the Enterprise Change Manager, so I'm responsible for how changes are defined and what category they fall into. We are enticing people to use automation, to be qualified as a routine change and not have to go through the approval process because they meet the criteria of being scriptable, repeatable, rollbackable, and that's actually less risk than if it was a manual process.

   

18. Again, in your talk from last year, you said that the quality of life improvement for your staff was the ultimate goal of your DevOps transformation and that's something that Gene Kim, the author of "The Phoenix Project," also stated as his goal for all his work in the DevOps movement. In which aspects did this quality of life improvement manifest itself?

So what used to take up sometimes on a Wednesday night - which is our manual change deployment window - up to four hours and that's if it was successful - if it was unsuccessful, it could take all night - has now been reduced to less than 30 seconds at 6:00 a.m. on a Wednesday morning or any day of the week because we've changed the change windows.

A person who was normally working a Wednesday night deployment is now putting their kid to bed and reading them books. Or if they're spending all weekend doing a deployment and then rolling it forward, rolling it back, they might be spending an hour now on a weekend morning. Availability's up and windows are short.

   

19. Are you aiming at a sort of continuous delivery?

I want hot deployments. I want no downtime. I want to do it at lunch and have nobody work off hours. That would be ideal. Of course, there's always going to be the COTS products and the big ERP packages that haven't moved to the model or are difficult to handle in that model, that we can't avoid.

   

20. What are the next steps in this DevOps journey?

To continue focusing on automating infrastructure and provisioning. Transforming our skills to be everything as code. To continue expanding the continuous delivery to support better deployment models, the blue green or hot, whatever you want to call it. So that there is higher availability, less risk, and people aren't working out of hours.

Manuel: Thank you very much, Aimee.

Thank you.

Nov 26, 2015

BT