There's recently been a lot of discussion about the convergence between SOA Governance and API Management. While much of the latest discussion has come from vendor blogs and whitepapers, the concept seems to originate from last year’s Gartner Magic Quadrant for Application Services Governance in which Gartner introduced the term "Application Services Governance" to "refer to the union of SOA governance technology functionality and API management...with the view that these aspects will eventually be unified." The earlier published Forrester Wave for API Management report, while acknowledging that SOA and APIs are different observed significant overlap in management requirements and the effect of this overlap on API management products.
So what is the state of SOA and API convergence? Is it a real phenomenon driven by parallel evolution of technologies?
The Panelists
Ed Anuff – Vice President, Product Strategy at Apigee.
Chris Haddad – Vice President, Platform at WSO2.
Roberto Medrano – Executive Vice President at SOA Software.
The Questions
- Protocols and data formats aside, are there fundamental differences between APIs and SOA?
- What are the main shared concerns for SOA and APIs?
- Runtime governance, policy management, is common to both SOA and APIs. But design governance processes have so far only been applied in the SOA domain. Would APIs benefit from design governance?
- Data schemas and interface specifications have been a strong focus of SOAP services, but are relatively new to REST APIs. Is this simply a matter of tooling or are there inherent differences in requirements?
- APIs have focused on developer engagement as a critical concern, driven by an open and competitive API marketplace. Within the enterprise, developer engagement for SOA has been somewhat challenging. Do you think SOA can learn from APIs in this respect?
- So if I'm an enterprise architect sitting on ten years of SOA infrastructure, what should I be planning for the future?
InfoQ: Protocols and data formats aside, are there fundamental differences between APIs and SOA?
Ed: The fundamental differences tend to be around governance and scope. Most SOA initiatives span enterprise-wide data sharing and application integration concerns and are managed through a much more formal and heavyweight governance process that is highly prescriptive in nature. API initiatives tend to be more agile in nature and are more focused on actual developer adoption and usage. Agility of application delivery is the primary measurement of API success.
Chris: SOA is a well-defined architecture style that addresses a fundamentally different set of goals and constraints from APIs.
SOA is a design discipline that focuses on implementing shared functionality as reusable services. From a technical perspective, a SOA must exhibit three key design principles; service orientation, clean separation of concerns, and loose coupling. Service orientation is gauged by service reusability, granularity, and architecture simplification. Clean separation of concerns is gauged by testing separation of business logic from infrastructure, interface from implementation, and interface from capability. Loose coupling is gauged by measuring interoperability, transaction control, and mediated interactions.
APIs are a strategic component supporting your Service Oriented Architecture initiative. The API technology space is loosely defined as a network interaction style and interface definition. For example, APIs are associated with RESTful endpoints accessed via the HTTP request-reply interaction style. An API may apply RESTful constraints on the interface definition and interaction style, but not all APIs are RESTful. All APIs clearly separate back-end implementation from consumer facing interface definition.
When development teams publish APIs, they explicitly separate service interface from service implementation. Managed API endpoints are lightweight proxies enforcing security, monitoring usage, and shaping traffic. The API proxy enables a clear separation of concern between consumer interface contract and back-end service implementation.
To define a clear separation, an API gateway is often used to enforce an API facade pattern and expose an API proxy for the service implementation. API Management incorporates SOA success lessons by allowing teams to easily apply adoption, trust, security, and quality of service best practices to APIs. These four focus areas are hard to include with traditional SOA infrastructure.
Roberto: Both SOA and API management technology have important things to say about the future capabilities of our IT infrastructure. API technology brings a strong focus to the consumption of your backend IT resources within a well-managed community of API developers and mobile app developers. SOA brings a strong focus to the production of reusable enterprise services that expose your backend IT resources. Only a unified SOA and API management solution will prepare you for the benefits of the convergence of these two technologies. APIs are essentially an evolution of SOA, imbibing a lot of the same fundamentals around creating and exposing re-usable services. However, the core difference between them being that APIs are focused more on making consumption easier, while SOA was focused on control and had an extensive and well-defined description language in the form of WSDL. APIs are managed more as a product, are well documented, humanly readable, which lends it-self to easier adoption by developers. APIs are often available in a self-service mode, where-in a developer can essentially provision access to an API and start using it with little or no human intervention. Many description languages like RAML, Swagger etc. are emerging, but they are still not as mature or complete as WSDL. We believe you should use a single unified solution to realize the benefits of both technologies.
InfoQ: What are the main shared concerns for SOA and APIs?
Ed: APIs and SOA are built on service endpoints and data protocols that must be robustly built, securely exposed, properly documented, and reliably supported.
Chris: On the surface, RESTful APIs are simply a specialized version of web services, and provide similar technical benefits. Both REST API design and SOA service design intend to expose discrete functionality via a well-defined interface. The API endpoint and the service endpoint both serve as a core design unit, and teams deliver technical solutions by accessing, aggregating, composing, and orchestrating endpoint interactions. Successful and scalable API and service-based solutions both require Internet messaging infrastructure, service level management, and security.
SOA and APIs share common integration concerns of endpoint location, message models, security, and quality of service. Managed APIs are ‘SOA done correctly’. A managed API is:
- Actively advertised and subscribe-able
- Available with an associated, published service-level agreement (SLA)
- Secured, authenticated, authorized and protected
- Monitored and monetized with analytics
While services should exhibit managed API characteristics, most service endpoints are deployed on platforms that don’t deliver management characteristics. By applying the API façade pattern, teams can layer network addressable endpoints, monitor usage, enforce usage limits, shape traffic, and authorize consumers.
Roberto: Governance, control and security remains the primary concern across SOA, APIs and other services initiatives for enterprises. In additional to proper documentation and ease-of-consumption, the biggest driver for both SOA and API adoption and success is whether they are planned right, built right, and work right and they do not breakdown due to internal changes and do not lead to unmanaged or uncontrolled revisions, factors which are critical to continued usage of these services. On the publisher side, it is critical that the service provider retains control over who has access to these services and they are able to govern not only access but are also able to log, monitor and analyze usage. Having proper governance and approval controls ensures that SOA and APIs are built right and operate efficiently. The entire lifecycle of APIs and SOA should be managed and governed in an agile manner, such that there is sufficient control but at the same time supports agile development and DevOps methodologies. Since both are similar they should be governed, managed and secure with the same products and not use different products for each.
InfoQ: Runtime governance, policy management, is common to both SOA and APIs. But design governance processes have so far only been applied in the SOA domain. Would APIs benefit from design governance?
Ed: While design governance has many benefits, in practice it has proven to be a double-edged sword, and is much of the reason why SOA adoption and service reuse have waned. Design education has proven to be more effective without the bottlenecks that most organizations inevitably impose as part of a design governance process.
Chris: Applying design governance to APIs would help promote an architecture instead of Just a Bunch of APIs (JABAs), overcome challenges when applying REST HATEOS principles, and encourage safe API evolution.
Many development teams publish services, yet struggle to create a service architecture that is widely shared, re-used, and adopted across internal development teams. Instead of creating consistent service architecture and demonstrating service re-use, teams inadvertently produce Just a Bunch of Web Services (JBOWS) or Just a Bunch of REST Services (JBORS). A single application often consumes a service, and a spaghetti web of One-to-One connections exists between service provider endpoints and consumers. Many teams find a SOA or REST focus may not improve IT agility, but result in simply swapping out IT toolsets, message formats, and protocols. Design time governance can help teams achieve architecture coherence and agility.
To easily evolve APIs, think through versioning, eventing, and hypermedia maturity; these are design governance concerns. API evolution must consider forward compatibility, backwards compatibility, and client processing flexibility.
Roberto: We are seeing this changing. Enterprises are realizing that unlike small start-ups who can keep iterating on their APIs without any real business impact, enterprise class APIs with real customers do need to be stable and reliable and built well from the get-go. This requires a certain amount of development and design governance to ensure alignment of API programs with strategic business objectives and make sure that enterprises build the right services at the right time. Design time governance allows API providers to achieve quicker return on investment, making sure APIs are built to plan and built to priority. At the same time we are also seeing that governance itself is transforming to adapt itself to agile development methodologies to support quick and rapid development.
Enterprises must be able to track all of the elements of their API and services architecture, changing them as needed while maintaining a clear understanding of the underlying interdependencies thru a management platform that automates machine and role based validations and sign-offs across the software development lifecycle for APIs and web services.
InfoQ: Data schemas and interface specifications have been a strong focus of SOAP services, but are relatively new to REST APIs. Is this simply a matter of tooling or are there inherent differences in requirements?
Ed: This is a good question without a simple answer. Sophisticated tooling really requires some level of specification detail. Some of this will come from the adoption of API description mechanisms like Swagger. The shift of API usage to being driven by application development efforts, particularly in the mobile space, will likely lead to similar types of IDE support as what is present for SOAP services.
Chris: The requirements are the same between SOAP services and REST APIs, but the approach is different. SOAP service data schemas and interface specifications are focused on strongly typed interactions with a well-defined data schema. REST APIs tend to abide by Postel’s Law, “be conservative in what you do; be liberal in what you accept from others”. A more conservative, simple API accessed by a client with liberal consumption requirements doesn’t require a formal contract.
API developers traditionally eschewed heavy weight, formal data schemas and interface specifications and prefer human readable documentation. API developers believe a simple, straightforward textual web page suffices. However, the RESTafarian backlash against schema, description languages, and machine-readable policy is starting to abate. Just take a look at the formality building in the Swagger 2.0 specification.
When developers consume or compose APIs or services across organizational boundaries, they find documentation and contracts describing how to interact with the API or service will strongly influence adoption.
REST best practices are starting to influence SOA service design and description. Whether building RESTful APIs or SOA services, teams should:
- Not publish endpoints exposing rigid data models and requiring complicated interaction patterns
- Document how session context and session state influences interactions
- Document the message model
- Describe usage limits and security concerns (i.e. acceptable authentication credentials, authorization protocols)
- Hide internal details from the external interface
SOA and API tactics should include defining metadata and policy frameworks that provide the scaffolding required to identify, classify, and describe consumption candidates. Most API management platforms bundle policy options into service level tiers that can be flexibly applied to APIs.
Metadata is the design time, development time, and run-time information used to describe a service or API. Policies are used to map metadata to deployed service instances and endpoints. Metadata can span the following concerns:
- Overview (e.g. name, description)
- Lifecycle Attributes (e.g. version, version relationships, lifecycle status)
- Classification (e.g. basic, composite, infrastructure, business)
- Endpoint Deployment Attributes (e.g. protocols, location, WS-* specifications)
- Data Model (e.g. JSON Schema, RAML, XML Schema, WSDL, version, semantics, validation)
- Service Level Requirements and Policies (e.g. availability, capacity, responsiveness, security, transaction rate)
- Mediation (e.g. routing, queuing, caching, transformation)
- Dependency Attributes (e.g. APIs, services, databases, directories, frameworks)
- Physical Instance Dependencies (e.g. application platform, security, management)
- Business Process Model (e.g. UML diagram, business classification)
- Contract information (e.g. consumers, providers, utilization)
- Usage Guidelines (e.g. time of day, availability, # of users. throughput)
- Accounting or remuneration options (e.g. pay per use, subscription, chargeback amount)
Metadata enables formal description of the endpoints and promotes discovery and re-use by other teams. Most organizations don’t start with a full inventory effort, but instead simplify the project's scope. They work towards describing a single candidate service/API that is core to the business, refining their metadata taxonomy, and gaining buy-in from solution teams on possible re-use.
Roberto: SOAP did have well defined data schema and interface specifications. WSDL 2.0 and WADL provided early support for APIs, but they couldn’t truly capture the essence of RESTful APIs. Recently we have seen the emergence of several description languages. New ways to describe and document APIs have emerged such as Swagger, RAML, API Blueprint and others, each taking a slightly different approach and they are still yet to achieve the level of maturity that we had with SOAP. However we are hopeful that these standards will evolve rapidly and probably converge into a mature and well accepted description language for RESTful APIs. SOA Software is already on the working group of RAML and following development in other description languages closely.
InfoQ: APIs have focused on developer engagement as a critical concern, driven by an open and competitive API marketplace. Within the enterprise, developer engagement for SOA has been somewhat challenging. Do you think SOA can learn from APIs in this respect?
Ed: I think enterprises must learn from this. I've heard that most SOA implementations have a very low level of service reuse. While many new API projects are solely focused on supporting specific applications and so are also not concerned with reuse, it is definitely the case that there are a number of open API programs that have demonstrated remarkable reuse via developer adoption.
Chris: Traditional SOA and integration platforms enable rapid development, but they don’t encourage cross-team developer engagement. Development teams commonly operate independently and without any knowledge of useful service building blocks. Within a typical enterprise-scale organization, hundreds of developers write new APIs and services; few people know:
- Who is consuming APIs and services,
- Who is writing re-usable APIs and services, or
- How APIs and services are being used.
In order to overcome common SOA anti-patterns of ‘Not Invented Here’ (NIH) and ‘Build, and Build Again’, SOA developers can learn from API marketplaces by offering managed services through a portal (aka Service Store). A Service Store is a venue to find, explore, subscribe and evaluate available resources. Using the Service Store, internal development teams can quickly find relevant services. Once a candidate service list is identified, the Service Store provides a structured environment for exploring the services and understanding solution fit.
SOA can learn from APIs about how to extend governance meta-data to include service provider details, service consumer subscriptions, and actual service usage. Self-service subscription and automated traffic monitoring provides an efficient mechanism to capture valuable information that an API provider team can use to identify popular APIs, analyze API dependencies and potential impact when evolving API, justify service consolidation activities, and guide monetization strategies.
Roberto: We already are seeing a convergence of SOA and APIs. This is leading to creation of what we called Internal API Catalogs. The API catalog combines the collaborative, open practices of external API Portals with search, controlled visibility, selective provisioning, and integration with enterprise security that are required for internal development. This public-private fusion offers enterprises a refreshing and productive way to harness the potential of APIs for software reuse and business value creation. So the combination of Internal API Portals with lightweight governance being adopted by enterprises to achieve the best of both SOA and APIs.
InfoQ: So if I'm an enterprise architect sitting on ten years of SOA infrastructure, what should I be planning for the future?
Ed: An enterprise architect should need to think about whether their API (and SOA) strategies are prescriptive and about central governance or if they're about developer empowerment and agility. Services design needs to start to focus on the needs of the application developer rather than trying to be about creating levels of service abstraction that don't meet the needs of these developers. The actual SOA infrastructure isn't going anywhere, but it's hard to understand where it's best applied until the organization develops a more agile model for delivering API-centric applications.
Chris: Traditional SOA infrastructure provides valuable capabilities (i.e. protocol mediation, message transformation, content routing, monitoring, security), yet doesn’t address core IT agility and trust concerns. A focus on self-service adoption, service level management, simple interfaces, and governance will help teams succeed. SOA governance, API management, and micro-services can stand in the gap and improve architectural coherence.
Service Oriented Architecture initiative success requires creating loosely coupled consumer-provider connections, enforcing a separation of concerns between consumer and provider, exposing a set of re-usable, shared services, and gaining service consumer adoption. Many development teams publish services, yet struggle to create a service architecture that is widely shared, re-used, and adopted across internal development teams. Many teams find a SOA or REST focus may not improve IT agility, but result in simply swapping out IT toolsets, message formats, and protocols.
Focus on encouraging adoption, trust, and monolith decomposition with governance, API management, and micro-services.
Roberto: SOA is an architectural methodology that is still valid. Enterprise architects should still continue to use SOA where it works for them. They should also look at adopting an Internal API Catalog in favor of registries and repositories to encourage developer adoption of these services. Documentation is the most important aspect of API portals and catalogs, which leads to the self-service nature of APIs. Enterprise Architects should build in the detailed, yet easy to consume documentation best practices in their governance model and enforce it as part of publishing APIs to the internal catalog. Lastly, protocols come and go. Enterprise Architects should not deploy and build an infrastructure that is tied to a specific protocol, but rather leverage an infrastructure that is inclusive and open to multiple protocols, both past and future. REST and JSON are a welcome improvement over the complex and bulky SOAP based protocols, but we are already seeing some newer protocols (such as AMQP, MQTT and web sockets) emerge to support Internet of Things and other upcoming trends.
About The Panelists
Ed Anuff has over 20 years of experience as an entrepreneur, creating innovative consumer and enterprise products and starting and selling several companies, and as an executive, defining product strategy at early stage and publicly traded companies. Ed currently serves as the VP of Product Strategy at Apigee, the API company, where he manages product and technology strategy and is directly responsible for mobile and developer products. Ed is the founder and previous CEO of Usergrid, a start-up that created a mobile BaaS to provide cloud services for powering mobile and rich client applications. Usergrid was acquired by Apigee in 2012.
Chris Haddad leads platform evangelism at WSO2. He lives in Space Coast Florida, where he watches rocket launches, rides the ocean surf, and writes about architecture best practices. Chris converges pragmatic hands-on practitioner experiences with IT business goals to communicate strategies and tactics when adopting Cloud, DevOps, and API-centric architecture.
Roberto Medrano: A recognized executive in the information technology fields of SOA, internet security, governance, and compliance. Medrano has extensive experience with both start-ups and large companies, having been involved at the beginning of four IT industries: EDA, Open Systems, Computer Security and now SOA. Prior to joining SOA Software, he was CEO of PoliVec, a leader in security policy. Before joining PoliVec, he was one of the top 100 Sr. Executives at Hewlett Packard. At Hewlett-Packard (HP) served as the General Manager of the E-Services and Internet Security Divisions. Medrano holds an MBA from UCLA, a MSEE from MIT, BSEE from USC.