Retiring Mainframe Programmers: Should I Care?

We take Uber, browse Pinterest, send tweets, and update Facebook. We hear every day about instant millionaires and the growing number of billionaires that built the latest gadgets in our high-tech world. But we ignore the fact that over 70% of all business transactions are processed on mainframes. Our visual and audio world is presented with tools hastily slapped together by young hotshots yet the reality is that the chairs we sit on, the paychecks we cash, the health care we utilize are available through data managed by mainframes. That’s right, over 80% of manufacturing, banking, and healthcare industries are on mainframes.

We hear so much hype about monetization of new markets that we fail to acknowledge the importance of mainframe code. Well, there’s this little issue… most mainframe developers are baby boomers who are approaching or past retirement. The harsher side to this reality is that the aren’t just retiring, they are expiring. (Short prayer for Jim Stanicki and John Stalher who were two exemplary mainframe developer friends of mine that died in the last few years.)

I Smell Rewrite

Shouldn’t those old mainframe applications just be rewritten? It ain’t that easy. Yeah, I know, you’ve heard about rewrites for years. But the reason why most of those Visual Basic, dBase III, and PHP apps (that’s right, I’m saying they weren’t mainframe apps) were rewritten every 5 years is because they weren't written that well to begin with. Meanwhile, the mainframe apps have been running well for decades. The Return On Investment (ROI) for rewrites of mainframe applications just hasn’t been there. Case in point: In the mid ‘80s I wrote a traffic system for Hanover Brands Inc. that is still in use today.

But then there’s this retiring and expiring thing. Why not just bite the bullet and do the rewrite?

Rewrites are never easy and, for huge applications, they are often failures. Just a few weeks ago, I did a rewrite of a little, itty, bitty, PHP application to Ruby and Rails. Now, I’m pretty good with Ruby and OK with PHP but, even though it was just over a thousand lines, I still missed stuff. Mainframe Cobol and RPG applications are a wee bit more complex. It is common for an RPG program to be ten, and Cobol to be twenty, thousand lines long. Multiply that by hundreds and hundreds of programs and you have an application that has a mega-million lines. Worse that that, many of those programs were written before modular programming techniques became available. Typically, all variables in one of these behemoths are global. I remember, dozen years or so ago, I had a jest-quest in articles and seminars of a Diogenes-like search for a local variable in mainframe code. Diogenes never found an honest man and I had problems finding local variables in circa-70s code.

RPG specifically can be crazy hard to read and understand. For years RPG could only use six character variable names. Actually, its was worse than that. RPG had a bug where if you used the same column name in two different tables, they would share the same memory space. So RPGers used two to four of those precious six character column names to identify the system and table they were associated with. The bug was fixed decades ago and today the variable and database column name length supports at least thirty two characters. But six character variable names are still prevalent in RPG programs.

Somewhere around 1992 I gave a presentation on Cobol modularizing techniques to the team of Circuit City coders of which I was a member. After the presentation, one of Circuit City’s best Cobol programmers said she didn't see the benefit of modularizing Cobol.

Understand that I can get in a fist fight with other Ruby developers over a discussion on whether a method should be no more than 6 or 9 lines long. In Ruby development, wanton use of global variables might get you fired. So I kind of chuckle when I recall working on these 10-20,000 line programs. Maintaining Cobol and RPG monstrosities is often more voodoo than mastery. You try something that you have a gut feel will work and you light incense, sprinkle holy water, and pray to a variety of gods that your change will work.

The prevalent development practices of Cobol and RPG propagates use of outdated syntax. We are talking ugly code. Often thousands of lines of code in one program have been commented out. Many sections are completely unused. Looking at some of this code is like walking into a hoarder's house -- it's full of useless junk. To push the analogy perhaps too far -- the more junk that piles up the more chance of rot and, yeah, stench (and we talk of code smells.) The thing is: processes and materials you use everyday flow through this old rotting code the maintainers of which are retiring and expiring.

Agility

Let me be honest… The reason I migrated my career away from the mainframe development workspace is velocity -- or the lack thereof. Development practices and toolsets for RPG and Cobol have languished. Test driven development, source control, modern editors, refactoring, agility… for years I proselytized such concepts in articles and seminars and not only was I, for the most part, ignored, but I couldn’t find projects to work on that followed such practices.

The essential word in my last paragraph was agility. Because mainframe application development practices lack agility they are slow to adapt to market demands. Often, the tarnishing of these old apps and the impending retirement of support staff cause new hot-shot C-level executives to suggest the procurement of costly ERP systems or complete rewrites. And we’ve all heard horror stories about such projects.

Modern Machines

Understand that the mainframes are not antiquated. They are not the System 360 and AS/400 of yesteryear but the IBM z/OS and IBM i 64-bit operating systems with reliability and scalability that Linux and Windows can’t approach. They also have a lower total cost of ownership for complex data centers. You scale mainframes horizontally, rather than vertically. Those mainframes can run the latest software as well. Case in point: you can run thousands of Docker images on one mainframe. DB2 for i is arguably the best database on the planet. As to hot technology, A few years ago I was on a team the moved Ruby and Rails to the the native IBMi operating system. Banking applications run on mainframes primarily for security reasons. And it was a banker that funded the Rails port to the IBM i platform. There are huge advantages to mainframes: ranging from huge horizontal growth to the ultimate in security and very close to 100% reliability. It may be true that IBM is selling less mainframes but they are doing quite well upgrading and expanding the existing machines as they have astronomical horizontal scalability.

What is aging is not the machines but mainframe applications and application programmers.

So, We Are Screwed?

Absolutely, just like we were when we hit year 2000 and everything crashed. It didn’t. We were just fine. We were fine because management finally started taking the two-digit to four-digit year thing seriously. If management begins to take the mainframer skills loss thing seriously, we will be fine.

Let me summarize my solution to the retiring and expiring mainframe coders thing before going into more detail:

• Modernize the database
• Refactor application code
• Train existing mainframers on agile and as trainers
• Coerce smart people into a career in mainframe development
• Convert existing code into APIs

Modernize the Database

Modernizing the database is step one. A wealth of data kept in the world’s best database and operating systems is hidden behind applications that are twenty to forty years old. Many of the mainframe databases were created before today's well known database normalization and optimization techniques were created. At little as ten years ago I put a web front end on a table that was clearly housed, at one point, in a card deck. There are tools and available techniques that allow you to mock the old database schema’s structure so legacy programs may continue to run with little or no modification. From experience I know that refactoring the database is not that difficult to do.

Refactor Application Code

I made a big stink earlier about the state of mainframe code. Know that mainframe code can be refactored to the point that development can be much more agile. Understand that I am not talking rewrite. Refactoring is the process of restructuring existing code without changing its behavior. The C-level executives should not have fear of the downtime that would be probable with a massive rewrite.

The first step to refactoring is to put code under source control. I highly recommend git. Perhaps the mainframe code is already managed by a source control package but my experience with mainframe source control packages is that they propagate sluggish development and dissuade common refactoring strategies. As soon as the code is in git, remove the commented out code -- it’s source control’s job to retain old versions of code.

The second step to refactoring is updating the development environments. Many mainframers still use green-screen editors even though powerful IDEs have been available for at least two decades. These modern IDE are bundled with refactoring tools.

The third step is to set up a unit testing strategy. Unit tests typically have very specific and detailed tests for program behavior. We don’t have time for that. What I recommend is to follow the Approvals Testing strategy developed by Llewellyn Falco (http://llewellynfalco.blogspot.com/). The basic concept of Approvals Testing is that you take snapshots of state before and after execution of a routine. That snapshot could be anything from a database query result, to a PDF, to a CSV. Be creative.  With the snapshot stored, you modify the routine and use the before and after images to verify that the refactoring did not change behavior. You may end up using a testing infrastructure that uses Java or Ruby or Python to invoke the mainframe routines but this layer will not be complex.

With a unit testing strategy constructed, the refactoring should begin with making variable names readable and understandable. Then start to whack away at reducing the use of global variables as you move into modularization. Duplicate code is rampant in mainframe code so use tools to find that duplicate code and then create common modules for them.

Empower Existing Developers

The golden key to a refactoring strategy is your best mainframe developer(s). And now I’m going to get a bit nasty:  Mainframe developers have been treated by their employers like bus drivers. Management feels it is the programmer’s job to move data from point to point and, when something breaks, they are expected to twiddle with the engine until they can get back on the bus and begin moving data again. From my personal experience, they are often underappreciated and underpaid. Many of these mainframers have no degree and deal with C-level executives with Ivy-league MBAs that seem to lack the ability to consider the ROI for more than this next quarter.

These mainframers need to be empowered. The will need retraining on agile development practices and they also need to become trainers themselves. They will need assistance in the refactoring and the later creation of modular APIs. And then there’s the issue of retiring and expiring so it will be part of the empowered mainframer’s job to manage the training of new developers.

If one of your existing mainframers doesn’t stand-out as a project lead, go out and get one. The problem is that many of the folks out there that have more than enough ability to help have moved on. They’ve moved on to management, or training or, like me, they’ve moved on to other programming languages and platforms. The problematic thing with these people is that, decades ago,  they grew tired of the revolving door of C-level executives who turned down their recommendations for modernization. Many of these ex-mainframers may simply tell you to buy an ERP,  give Oracle a call, or do a complete rewrite (which they know from experience will probably be a failure.) So be prepared to wheel and deal with these guys.

Where might you find these ex-mainframers? Troll Linkedin for those folks that are attempting to hide mainframe experience yet have excellent communication skills and experience with new technologies and agile. Go to Java, Ruby, JavaScript conferences and talk to the folks that are over 50. Based on their age, they probably are ex-mainframers but they are also well versed in new tech and, more importantly, agile development. 

Once you know what you are looking for you might not have to look too far. This paragraph may be cut because my editor is an ex-mainframer and an Agile coach. He probably will cut this paragraph because he left his mainframe bus driver job years ago. Go find a person like him to manage your mainframe modernization project. [EDITOR: No - I left it in because I agree]

Coerce Smart People Into Becoming Mainframers

Colleges aren't teaching Cobol and RPG. There are blog posts and articles on Skill Loss that suggest colleges need to add mainframe course and otherwise attract millennials to careers in mainframes.

I don’t think millennials are the solution. I would not suggest a career in mainframe development to any youngster. There are far less jobs available and the slow development velocity makes for a less satisfying career.  My suggestion is 1) retrain existing non-IT staff and 2) coerce folks in their 30s and 40s into a mainframe career. Both of those suggestions sound crazy but I question the capabilities of a young IT graduate that is willing to begin their career on a mainframe.

The technical knowledge required to be a mainframe developer is more focused than the polyglot, multi-platform, programmer of today’s vogue developer. What companies need is not someone with NodeJS and functional programming skills but business acumen, a desire to learn, and, quite simply, smarts. There are plenty of smart people with non-IT mid-career blues that would be willing to have the opportunity to potentially double their salary after a year of on the job retraining. One area to tap would be retired military.

Convert Code into APIs

The final step in a mainframe modernization process is to turn your refactored code into APIs. The will them be reusable software components. Often the parameter list of mainframe code will be so complex that it might seem that creating an API for it would be impossible. For those you create one or more wrapper programs (which can be written with Cobol or RPG or your new language of choice.) One technique I liked to use was to create SQL Stored Procedures that wrapped the legacy modules. With SQL Stored Procedures available, anybody with an SQL interface (JDBC-, ODBC-driver or whatever) can use those routines.

With legacy code available via an API (SOAP, REST, or otherwise) and unit tests for everything, development can be fully agile. New code can be written in whatever language you choose. And, yeah, you can hire millennials.

The more sensitive and complex programs that show up high on your churn reports can then be slated for conversion to a new language.

Mainframe Agility

Mainframe applications, while being crazy reliable for all but half-a-century, are brittle with little, if any, automated tests, and the mainframe workforce either does not know about refactoring strategies or hasn’t had the C-Level management backing to start it. In short the code is not adaptable to changing business requirements. Existing mainframe programmers are retiring at an alarming rate and the workforce is not being replenished.

Moving off the mainframe is far from the most optimal solution. Complete rewrites or converting to an ERP is costly and fraught with peril. The mainframe boxes themselves are not aging. In fact they outcompete Microsoft and Linux on features like performance, scalability, security, and reliability. It’s not the machines but applications and programmers that are aging.

Modernize that database, refactor that code, become agile, shore up programming staff with internal training. And finally, where it shows benefit, start to move your newly modularized code into a new language.

The thing is: these solutions are dependent on C-level executive buy in. For them to do that they need to look past quarterly results and think about what the next two to five years will bring as they lose their undervalued mainframe developers.

About the Author

Don Denoncourt is a developer for simplethread.com. He has been coding since before Windows and Linux, much less the Internet. In the early nineties, Don moved from RPG and Cobol to C and C++. He adopted Java before it was real: 1996. After coding his way through the proliferation of Java frameworks (including Struts, Spring, and EJB) Don pined for the Convention-over-Configuration framework of Ruby and Rails. Don did Groovy and Grails before finally moving to Rails in 2011. Don enjoys writing and has published a couple of books and hundreds of technical articles. Don has been working from home since last century. When Don is not working, he loves spending time with his 3 grandchildren. To keep his mind young, Don reads and listens to novels in Italian. And, to keep his body young, Don is an avid off-road and street unicyclist.