How Compuware Escaped Its Waterfall for True Mainframe DevOps

If you toured Compuware’s Detroit headquarters five years ago, you would have witnessed a struggling software monolith in decline, a radically different existence from what we live today in our innovative culture of continuous improvement under Agile and DevOps.

Back then, we were mainframe-powered and mainframe-focused, as we still are, but we were stagnant. We were delivering software maintenance updates every 12 to 18 months via linear, slow, waterfall development. Our culture was uninspired, our processes slow and long, our tools lacking. Worst of all, we put too little focus on solving customer challenges.

You would also have met many of the same developers who work at Compuware today. They once cherished silos and feared change, but now they adapt quickly and collaborate to overcome new challenges with agility; they also work in two-week sprints to develop innovative mainframe solutions that integrate into DevOps toolchains. Every 90 days, they are able to deliver technology that supports customers’ digital transformations. This wouldn’t be possible if we, as a company, didn’t first transform ourselves and our own approach to innovation.

Using DevOps, our mainframe software company fought gravity and began innovating again—without losing or replacing most of our staff and without losing focus on the computing platform that got this company going in the first place over 45 years ago: the mainframe. Along the way, we identified three core philosophies and an abundance of best practices about innovation, speed, and quality that guided us on this DevOps journey.

Why We Shifted

In our digital age, established enterprises that once ruled markets are failing because they fear and resist change. But, it would be naïve to assume every employee and leader in those organizations feels this way. In fact, there are innovative, engaged, passionate explorers in those enterprises who could change the fate of their companies, if they were so empowered.

As we were struggling in 2014, many of us knew our deliverance from possible dissolution would require momentous changes to our culture, processes and tools, but we needed leaders who wanted and knew how to drive that transformation, who were willing to empower passionate explorers, invest in experimentation and enable change. Fortunately, when Chris O’Malley joined Compuware as CEO, he saw an opportunity for revitalization, an opportunity to defy the odds through a transition to Agile and DevOps.

Three Guiding Principles for DevOps Transformation

With a new vision and mission, we identified three core philosophies that would guide Compuware’s DevOps journey. Adhering to these across our organization, from IT to HR, was crucial to a successful transformation from Waterfall, and to our survival.

1. Empower Ideation and Innovation

To have confidence the technology we delivered would meet or exceed the needs of our customers, we needed to develop a culture of continuous improvement that encouraged employees to become passionate explorers. In a development environment, passionate explorers are those eager to try and learn new things, even if it may mean failure—and they use these lessons for transformation.

These people would be critical to advancing company objectives and inspiring others. They would be the people driven to experiment, take risks, and dream up competitive solutions to the massive challenges we and our customers faced on the mainframe. They would be the ones to help us make the mainframe as agile, as integrated into a DevOps ecosystem, and as important to a CI/CD pipeline, as any other platform.

So, where were the ideas? They were there, but they were hidden. The people who had them weren’t empowered to build on or share them. We simply had a culture that did not allow the great ideas throughout our company to come to the surface. We realized we had to cease treating our developers like people rowing a ship to the beat of a drum, and start treating them like high-performance athletes who require time and space to continuously improve. They needed to feel free to share wild ideas, experiment, and iterate until something great came to fruition.

2. Fast Beats Slow

We also needed speed. We couldn’t be confident in innovations that took over a year to deliver, and which came from ideas potentially even more dated. Like the many massive companies trailing today, we experienced that big no longer beats small—fast beats slow.

We needed to boost our agility to manifest great ideas into software deliverables that fill business needs and outpace competitors. Meeting customer demands more rapidly—in other words, rolling out solutions quickly to help our customers address their own mainframe modernization challenges—would require reducing the time to move ideas into production, creating rapid feedback loops, shortening release cycles, and generating greater efficiency across the entire organization, among other things.

Agile Development—adopting Scrum and working in two-week sprints—became crucial. We had to move from project management to product management and start focusing on building minimum viable products we could deliver quickly and iteratively to meet customers’ most critical needs. But speed couldn’t be our only focus.

3. Quality Is Still Essential

In mainframe development, quality takes priority, given the mission-critical nature of mainframe transaction workloads. That’s why most mainframe teams have historically relied on Waterfall methodology, which follows a rigid 12-to-18-month sequence of setting requirements, designing, developing, testing, analyzing, and finally deploying—as we did prior to our transformation. While this methodology may have worked in a pre-digital economy by helping companies maintain high quality standards, it’s dangerous to follow today, because it prevents the velocity and efficiency today’s organizations need to compete.

Our new agility allowed us to be fast and efficient enough to compete in a digital age. It also forced us to approach quality differently, simply because the time to ensure it would be reduced. Giving ourselves new freedom to fail fast, as we developed new ideas and experimented to deliver unique, competitive solutions, also required us to constantly maintain, measure, and improve quality at all costs.

Continually increasing the number of automated tests and code coverage, combined with an increased frequency of executing these tests, helped us better ensure quality. Our automated test suites now run nightly so new code is validated more quickly when completed. Automated test processes are now much more streamlined and efficient, which has enabled us to increase the frequency.

Our Biggest Obstacles to DevOps

We knew following these philosophies would require a shift to an Agile and DevOps mindset if we were to propel our company out of a downward spiral and into a market leadership position. But, there was a chasm between where we were and where we needed to be. We had to find a way across, and that required overcoming several obstacles related to our culture, processes, and tools.

Culture of Apathy

Becoming Agile and embracing DevOps are changes that require executive buy-in and strong leadership, if they’re going to prevail against the status quos of an entire organization. Chris O’Malley embodied these qualities as Compuware’s new CEO, but as expected, that didn’t resonate with everyone.

Not everyone was on board with Agile and DevOps, and many of those who weren’t knew how to do their jobs better than anyone else. They had processes, habits, and routines they relied on and never expected to change. The status quo was comfortable, it worked—for them. But apathy toward change didn’t work for the company or our customers. As Chris has said: "Apathy settles for ‘good enough’ instead of striving for excellence and innovation." Apathy was a virus, and it was killing us.

Siloed Processes

Now, we have multiple teams for our various software products, but five years ago, these teams didn’t communicate or collaborate frequently. Each team specialized in what it did, and individual contributors specialized in what they accomplished for their teams. This created unshared silos of knowledge and skills, which made us inefficient, slow, and less likely to improve overall quality and innovation across our toolset.

Amidst all of this, we were losing experts through attrition, as is the case with most mainframe companies. Any knowledge these people hadn’t shared with colleagues would eventually go with them, leaving us stranded with work no one knew how to accomplish.

Redundancy in processes was another major consequence of siloed work. There were several processes in place for related tasks across teams, rather than a few standards for everyone to follow. This simply created more technical debt, more chaos to sift through, and made it difficult to set objective standards against which we could gauge our entire organization, rather than measuring teams at a subjective, localized level.

Outdated Tools

From a technology perspective, our mainframe tools were a manifestation of our silos. They didn’t integrate easily across platforms, making it difficult for our customers and our own teams to work fluidly. The mainframe-specific technology we offered used the classic green screen 3270 emulator interface, and other non-mainframe tools used Eclipse-based interfaces, preventing teams from having what was needed to be productive—common interfaces and tools.

These tools even impacted recruitment. Because of the stigma attached to mainframe development (not to mention our issues related to culture and processes), we had difficulty attracting and onboarding next-gen programmers, who were unwilling to put up with old-school ways. We needed to accommodate them.

To attract next-gen talent, we needed tools that were:

• Modern, intuitive, and uncomplicated
• Supported Agile and DevOps processes
• Integrated into a cross-platform DevOps toolchain with best-of-breed solutions
• Enabled less-experienced developers to efficiently update and enhance complex programs

We also needed to provide these kinds of tools to our customers, so they could be successful.

How We Solved These Problems

These were big issues preventing our movement from a state of collapse to a state of continuous improvement. By driving the right changes through Agile and DevOps, we’ve successfully overcome them. Here’s what we did.

Culture: From Apathetic to Awake

To rid our organization of the apathy virus, we needed to inspire our developers to reject the status quo, and feel empowered to embrace creativity and innovation in advancing our toolset, rather than serving as mere caretakers of our products.

To move beyond concepts and catalyze inspiration for our teams, we began making bold promises to customers. We hadn’t released a new product since the ‘90s, and we decided to create a lifecycle of new functionality to be delivered to customers every 90 days. The first product came from an idea one of our passionate explorers in engineering brought directly to Chris O’Malley. It was a feature that provides unprecedented graphical visibility into often-complex interactions between mainframe programs.

A diverse team of developers—some with 35 years of experience, some with only two months under their belts—took this product from concept to delivery in 84 days. This success inspired everyone to believe they were capable of repeating this effort, over and over and over. That release became the first of our ongoing quarterly releases of new functionality—19 consecutive quarters as of this writing.

Processes: From Siloed to Synergistic

As we drove a new inspired, empowered culture, admittedly much less comfortable than the status quo many of us had embraced, we needed leaders to guide our new cadence of 90-day Agile deliveries.

We began training leaders in Agile methodologies. Roles shifted, and new roles like product manager, product owner, and scrum master emerged. Our teams became less homogeneous and more blended, as Development, Operations, Security, QA and other roles throughout the organization came together in Scrum teams.

We began breaking down silos to encourage transparency, communication, and cross-team collaboration. People crossed product-team borders, and new product teams with more disparate skills formed. Knowledge sharing increased and improved, becoming a necessity and an outcome of this collaboration and communication as processes and skills were shared.  

Prior to our transformation, our mainframe developers and non-mainframe developers had very little interaction, which caused delays in delivering enhancements. New teams came together in a broader team to work on enhancements in their entirety. This meant scrum teams would have both mainframe and non-mainframe developers working side-by-side every day. Siloed teams were no longer working on an enhancement without knowledge of the other platform interaction. Initially, this created some tension, but as common tools were introduced, such as Jenkins and SonarQube, the teams were able to function together better. DevOps, continuous integration, and continuous delivery forced all team members to interact.

This culture spread to the rest of the company, as we began holding bi-weekly town halls that kept everyone aware of the successes and failures of teams and the company. The new agility and DevOps-enablement was infectious. Every organization within the company would now have to pay closer attention to what the company was doing, and move faster in their roles to ensure whatever we put out every 90 days was supported effectively, from marketing, to finance, to legal.

Tools: From Outdated to Modern

Making all of this happen required more than changing our mindsets and processes. We also needed tools designed to support Agile and DevOps approaches to building great software.

We began developing a foundational software suite called Topaz, to support modern mainframe development and testing, and we also began innovating our existing "classic" toolset on a quarterly basis—for customers and ourselves, as our own teams use Compuware tools to build our software. Topaz Workbench, an Eclipse-based IDE, gives our developers and customers a modern, easy-to-use, intuitive interface that was an alternative to "green screens."

Products like Topaz for Program Analysis and Topaz for Total Test also allow developers to better understand mainframe code and automate the creation and execution of unit, functional, and integration tests.

As we improved our toolset, we began partnering with other mainframe and non-mainframe software vendors and integrating our tools with theirs to build a comprehensive toolchain that allowed the mainframe to be integrated throughout the entire DevOps lifecycle.

The Fruits of Our DevOps Transformation

Adopting DevOps was particularly challenging. We had to commit to the change by transforming the entire organization, which was made up of a large number of very experienced developers and a small number of next-gen developers. Our 'burn the boats approach' risked alienating development teams and jeopardizing the move to use DevOps and Agile. So, rather than send everyone for training, scrum masters and product owners were sent for certification training. They returned and became trainers for their new teams. The teams watched videos and read about the steps needed to transition to DevOps and Agile. This approach gave each unique team the opportunity to have input and take ownership of the new development methodology.

These changes have required significant effort from our teams. But, they have been part of our DevOps journey and we’ve continuously improved upon them since committing to change. They are the outcomes of implementing Agile and DevOps best practices, and are things on which we continue to iterate and transform each day.

We’ve reached the other side of the chasm, and after 19 quarters (and counting) of consecutive quarterly deliveries of new products, product enhancements, integrations, and partnerships under Agile and DevOps, we can look back across and see how far we’ve come from where we once stood—on apathetic ground. We’re now achieving greater quality, velocity, efficiency, and growth every quarter, every year. We now:

• Identify 25% more trapped defects while decreasing our number of escaped defects
• Deliver double the amount of code per developer
• Experiencing revenue growth
• Have made five acquisitions in the past three years
• Have increased our workforce numbers, with millennials over 30% and growing

What We’ve Learned

Our transformation is continuous, as is our learning. Here are some of the important things we’ve learned and continue to improve upon through DevOps:

Innovation

• Be Customer-First: We keep customers the focus of our efforts. It is our customers who inspire us to deliver innovations, so they can be better equipped to do their jobs.
• Empower People: Empowering individuals and teams to allow them to thrive and deliver innovation is critical to sustained success.
• Embrace Challenges: Challenging ourselves and others to do better, be different, and take chances helps us achieve success.
• Seek Out Ideation: Generating new ideas is essential to growing a business. In a bigger company, new ideas can come from anywhere. We’ve learned that creating an environment of collaboration and transparency allows great ideas to come to the surface.
• Find the Right Leaders: Executive sponsorship and strong leaders create vision and provide the environment that allows the entire organization to thrive and grow.

Velocity (Speed), Quality, and Efficiency

• Set High Expectations: We set high expectations for ourselves. We hold ourselves accountable to deliver every 90 days and have invited customers to join the journey.
• Fail Fast, Succeed Faster: We’ve learned failure can be good when perceived appropriately and used to do better in the future.
• Rely on Metrics: It’s important to understand how much you can accomplish (velocity), what is most important to accomplish (efficiency), and whether something will work well with a low number of defects (quality). Measuring velocity, efficiency and quality shows how far we’ve come and how far we still must go.

It’s easy to become complacent, accept the status quo, and ignore the need for change. It’s difficult to be creative, collaborative, and expose failure for the sake of finding ways to improve.

Agile, DevOps, culture, processes, tools, people—these are how we describe the transformation of our company. Continuous improvement of velocity, quality and efficiency, while delivering innovation every 90 days is what we do. In doing those things which are difficult, we have become an innovative force in the mainframe ecosystem.

About the Author

David Rizzo is Vice President of Product Development at Compuware. He has been with Compuware for more than 19 years. He has been instrumental in determining the technical direction, design, testing, new feature implementation, and ongoing maintenance for the Compuware product portfolio.