Untangling an API-First Transformation at Scale. Lessons Learnt at PayPal – Part 3

This is the third in a three-part series that explores how PayPal has adopted a more API-first approach to building platform services. In the first part, we explored the reasons for moving to a more loosely-coupled, API-driven architecture as well as some of the infrastructure needed to make the process work at scale. In the second, we looked more deeply at how the APIs themselves are managed. In this article, we’ll take a closer look at the program aspects and some of the execution and operational challenges that had to be overcome.

Deepak Nadig: What was the biggest challenge when you first kicked off the program?

Erik Hogan: A few come to mind.

Initially, we were faced with an engineering and product organization that was used to the status quo and was already going through a waterfall to Agile transition. The organization had a not-invented-here attitude and there was a vocal, stalwart group of long-time developers who were highly resistant and very influential. Similar programs designed to refactor the platform or do an SOA transformation had been tried a few times in the past with mixed success. There were legitimate concerns that this new effort would just result in more complexity, more confusion, and more tech debt. Simply getting them to buy in and support the program was a hurdle.

Understanding the scope and capabilities of the company was another huge challenge. Fifteen years of development produces a lot of code and solves a lot of use cases, most of which were not really documented. Getting a good understanding of existing business processes was necessary to form a reasonably accurate vision of what we wanted the API product portfolio to look like in the end.

The last was how to structure the program. There was no blueprint and we had to develop a strategy that was both feasible and accommodated the nuances and maturity level of the organization. That took time. Business was not closely engaged in past efforts. We basically started from zero.

Deepak Nadig: What did you do to overcome these obstacles?

Erik Hogan: I’ll start with the strategy.

There was a fair amount of top-down pressure to make progress. The CTO, CEO, and a fair number of technology VPs understood that we needed to change and looked to the program as a part of the solution. In that sense, we didn’t have to sell upper management on the value. They understood that we couldn’t increase our agility and bring new products to market fast enough if we continued with the status quo. They were as frustrated as everybody with the slow pace of innovation and the high cost of every new integration.

We explored a couple of different engagement models. One of the first was to recruit technology leaders within each domain to become “champions”. The idea was to create experts that can evangelize the principles and standards within their groups. We assembled them, had meetings, did some education, and then – nothing much happened. The advocates loved the idea, but had no real incentive to follow through as they still had their day jobs. The other cadre had more or less been “volunteered” by their managers to participate. Not only did they not necessarily believe in what we were trying to do, they weren’t really being held accountable for it either. It fizzled.

We also explored extending development tools to encourage the design patterns and standards we were putting together. While this in itself can provide a lot of value and is actually necessary for success, it alone is not enough to re-shape and solidify the capability boundaries of the service portfolio. If you have a pig, applying lipstick doesn’t fundamentally change the fact you can only build bacon and pork chops. You need to think much more deeply if you want a nicely decoupled and agile platform.

After much meditation, we finally started to form a vision that we thought might work. Being data-driven, management needed some simple metrics they could use to track progress and hold their teams accountable. They were primarily interested in how much progress we were making and when we’d be “done”. We also knew that this was going to take a long time – years – and that we needed a model that gave us the flexibility to transform in stages.

We settled on the idea of an API maturity model that encapsulated the standards we’d developed. We wrote objectively verifiable criteria and plotted them on a scale of 1 – 5. In terms of priority, we wanted to emphasize well designed interfaces, conformant to the business capability model as a first goal. The idea being that if we can get the interfaces right, we can establish stable boundaries between business capabilities, begin to migrate legacy traffic, and gradually refactor the underlying service implementations. We’d eventually achieve a nicely encapsulated set of microservices, but we could get most of the business agility and integration cost benefits provided by the APIs without absorbing the entire refactoring hit up front. Maturity level 3 represented this step. APIs that achieved full data and service encapsulation were maturity 5. The other, interim steps allowed for a smooth transition.

To calculate a percentage done metric, we needed a denominator. Our approach was to define a business capability model and mock what we believed the primary resources and endpoints would be in the end state. A team of six or seven business architects spent the better part of a year analyzing use cases, interviewing domain leads, doing code forensics, and documenting business processes to arrive at the model. It was by no means perfect, but it did give us some semblance of a denominator we could use to track progress.

Finally, we sold management on the idea that we’d build a target percentage of the portfolio to maturity level 3 in the first year and, critically, we got their commitment to cascade this goal to all their managers. This applied to both the product and engineering organizations. In one step, we addressed the measurement, step-by-step progression, and priority alignment problems. Critically, we also had a simple, numerical representation of our progress that everybody in the company could understand. From an internal marketing perspective, that was extremely powerful. We set a target of 75% of the portfolio exposed through RESTful, standards-compliant APIs by the end of 2014. We ended up beating that by a significant margin.

Deepak Nadig: How large was the team and what roles did they play?

Erik Hogan: A platform product group was formed as a result of this initiative. It started off with around seven people. This is the core group involved in decomposing the business architecture, defining business capabilities, managing the API product portfolio, and managing the infrastructure and tools roadmap to make the entire program possible.

These are very senior and technical product managers, mostly with engineering backgrounds, who had a real passion for designing great platforms. Every API built is reviewed by them and evaluated against the target business architecture. They determine naming and try to ensure product consistency across the portfolio. Most importantly, they try to always represent the customer’s perspective and ensure internal concepts, acronyms, and terminology don’t “leak” into the external interface. Developers tend to focus on a specific problem. These Portfolio Managers are there to apply a holistic, platform view and help shape the work of developers into APIs that are as reusable as possible.

Another very key group is something we call API Designers. These are very special engineers who have a great mix of architecture, interface design skills, customer focus, and a passion for education and mentoring. Possessing a few of these skills is not uncommon, but finding them all in the same person is unusual. We were lucky to find some really exceptional people who were able to build this team into a core strength of the program. Their job was not only to define and maintain standards, but also to evangelize them, educate the organization, and review API spec pull requests against the criteria in the maturity model. This direct interaction with developers is where most of the education and learning actually happens. It’s also where we get a lot of on-the-ground credibility. We had three or four people in this group during the initial build-out year. They reviewed all APIs and were extremely busy.

The third group was a small Scrum team focused on the infrastructure required to scale the program. At the time we started, we weren’t happy with any commercial offerings, so we built our own, internal developer portal. It provides discoverability of all APIs, automates the maturity score calculation, provides some basic workflow, tracks lifecycle state, and enforces versioning policy. This core system also produces the necessary reporting  on the status of the program and roll that up to the various team and business organizations. Over time, this team has also built libraries, APIs, and plugins that streamline the SDLC and make it easier for developers to deploy their APIs.

The last group was program management, whose job it was to coordinate activity across all teams and generally control cross domain risk. In the beginning, especially, we discovered a lot of duplicate development in different teams and business units across the organization. A significant benefit of the portfolio-driven approach is a reduction in duplicate investment and greater reusability. This required some tough decisions by senior leaders and, ultimately, decisions that resulted in teams being reassigned to other projects. Program management played a key role in facilitating these outcomes.

Deepak Nadig: Besides setting top level goals, what have you done to get buy-in and encourage developers and product managers to participate?

Erik Hogan: To be fair, most developers actually supported what we were doing from the beginning. We were proposing moving from proprietary solutions to more industry-standard approaches like REST and OAuth2. Anybody who had seen the outside world recently and wasn’t personally invested in what was going to be replaced, intuitively understood the value of adopting the same standards as everybody else. It also interested them personally since they’d get to use more modern tools and technologies that improved their marketability.

Most of those that weren’t immediately on board changed their minds after we did a little more education about what we were trying to do. There were a few die-hards that held out until the end, but the momentum clearly shifted after we articulated a clear vision and a plan to get there.

We tried to find creative ways to “pull” developers in. We created a monthly award for the best new service and publicly recognized the team in all-hands meetings. After a few months, teams were actively lobbying the program to win the award, so it definitely had some effect.

We also hosted hackathons that encouraged creative use of the new APIs. These were important not only to motivate service developers, but also to iron out bugs and gaps in the SDLC. A side-effect is it showed service developers that they have customers outside of their immediate project. To many, it was a strange idea to think about somebody integrating with their service without even asking them. It wasn’t what they were used to. That’s a cultural shift that also illustrates why good documentation is actually really important. APIs aren’t just projects. They’re products that require ongoing support and, hopefully, a good developer experience.

Deepak Nadig: There is always the potential for programs like this to start strong and fizzle. What have you done to ensure ongoing, long term success?

Erik Hogan: This is very true. There is often a strong, initial push, it’s a top priority, some progress is made, and then the organization shifts to the next, new thing. To some extent, this was also true in this case.

I think the key is to always think about your long-term strategy and react quickly to changes in your environment. The program itself is like a product. Putting on your product hat forces you to think about your customers, what they need, what your competitive threats are, and how to maintain growth. Programs exist to fulfill a business purpose and you need to be sure you’re doing a good job contributing expected value.

In our case, we generated momentum in the first year with a lot of high level executive help. The second year, there were a lot of organizational changes, we got a new CEO, we went public, and there were a very different set of top level priorities. Our visibility dropped dramatically.

The positive outcome is it forced us to double down on the satisfaction of our developer customers. We became much more customer-focused and systematically improved the areas that were irritating them. We matured our tools, sped up the review process, increased developer education, and eliminated some non-essential maturity criteria. Friction in the process reduced and more developers actively engaged.

The other thing we noticed is repeat customers came back earlier in the process and asked for more help earlier in the SDLC. At its core, the product we were “selling” to them – API design expertise – was recognized as really valuable to developers who wanted to build great services. We ended up saving them time, in many cases, and they produced better products as a result. More native evangelists emerged from within each domain and this had a positive, flywheel effect.

Another strategy we employed was to decentralize as much as possible. This was always part of the original plan, but we felt like we needed to keep things centralized until we established a repeatable process and mature the infrastructure to support it. By the end of the second year, we began to see some success in training developers in each domain how to do the API maturity assessment process themselves. We made a big education push the third year and actively recruited and trained enough developers to handle the majority of API reviews. Domains are now creating their own plans to have a “certified” API designer in every team as they’ve seen the positive impact on the quality of the services they build. We’re shifting into more of a support and oversight role and the methodology itself becomes a native part of every team.

To reinforce long-lasting presence, you should also figure out how to integrate and contribute to each phase of the SDLC. Initially, we primarily focused on the design phase as we wanted to be sure we had good API designs and followed a design-first methodology as much as possible. We then moved into the development phase by providing a library that developers can use to verify their service implementation matches their API spec. This leverages their functional tests to produce a report identifying where there are discrepancies. It has been very effective at identifying bugs and backward incompatible changes that snuck into the implementation, but not the API spec. Lastly, we’re more closely integrating with deployment tools and production environments to audit ongoing API conformance and install checkpoints that ensure minimum maturity levels are met.

Overall, we’ve managed to maintain momentum and grow the program despite some pretty drastic organization and priority changes. In my experience, that’s something most programs struggle with.

About the Interviewee

Erik Hogan has been learning what it means to be a great Product Manager for almost two decades. Much of that time has been spent understanding how to be a customer champion and trusted leader who can scale. He derives great satisfaction factoring complex problems into reusable parts and giving engineering exactly what they need to execute quickly. Lately, he’s been applying fundamental product concepts of simplicity, storytelling, and focus to affect organizational change - a very different type of "product". Also, his wife still gets mad when he asks for her success criteria.

About the Interviewer

Deepak Nadig is Distinguished Architect at Intuit. Prior to Intuit, Deepak was Head of API Platform Engineering at PayPal, and responsible for leading and transforming PayPal APIs used internally and externally to process $100 Billions of payments. Deepak has 16+ years of experience in leading engineering and architecture organizations, in high scale online websites like eBay, as well as enterprise software companies like VeriFone and HP. Deepak is a frequent speaker at conferences and is passionate about the use of technology in transforming businesses.