Jaime Ryan, Partner Solutions Architect at Layer 7, in an article titled "Rethinking the ESB: Building a simple, secure, scalable Service Bus with an SOA Gateway" has discussed the emergence of the SOA gateway as a viable alternative for the ESB. The core of the recommendations are based on modern SOA gateway's functional capabilities around three typical ESB usecases: standards-based endpoint abstraction, data and transport mediation and message routing, which he terms "the hallmarks of traditional ESB", in addition to a host of other non-functional requirements.
InfoQ spoke to Jaime Ryan to better understand the underlying reasoning behind various recommendations and the timing for these recommendations.
InfoQ: What are the painpoints of the current state of ESBs that motivated the need for looking for a replacement?
Once a company has decided to go down the Enterprise Service Bus route, they often get caught up in the complexity (and cost) of a mammoth software suite provided by major vendors attempting to equate the ESB pattern with a particular product. These suites, while they tend to be relatively comprehensive in terms of adapters, are often far more than the average enterprise really needs. When the goal of the architecture is agility and the ability to respond to customer needs, a solution that takes a week to install is rarely the answer. These are heavyweight, adapter-driven solutions that take months to years to roll out in any meaningful fashion, and often require a lot of code for mapping protocols and data formats along the way. They’re also focused mainly on connectivity and very little on security, which is what initially drove the interest in applying SOA Gateways to this pattern.
As enterprises successfully deployed SOA Gateways at the edge of their networks for security and governance, it was a natural transition into the intranet for the same purposes. Once deployed between internal applications as a security shim, customers started to recognize that more and more of their ESB needs could be met with these lightweight, easy-to-use, lower-cost alternatives to the traditional ESB platforms they had been implementing with varying degrees of success. At that point it becomes a requirements exercise…do these gateways meet my needs, and if so, why am I continuing down a slower, more costly, more laborious path to the same end goal? SOA Gateways are easier to configure, they scale more consistently, and they’re more manageable from an operations perspective. They provide value to the security architects, application developers, network administrators, and operations personnel; perhaps most importantly, their agility and flexibility help increase the bottom line for the business, and no pain is greater than a pain in the pocketbook.
InfoQ: What is new to the concept of implementing certain ESB patterns using SOA gateways and why should it be of concern now? Haven’t we been witnessing this role expansion since the early years of the previous decade. For example the incorporation of DataPower into IBM’s ESB family of products and the use cases it addresses?
Certainly the concept of using an SOA Gateway as an ESB has been around for quite a while, as several vendors have been selling in this space for almost a decade. However, these solutions have gotten more sophisticated, as have the applications and services to which the ESB connects. As the number of supported protocols and message formats expands, SOA Gateways are becoming increasingly comprehensive. Ten years ago, these solutions were limited to HTTP and HTTPS, and focused almost entirely on security. The range of applications and interfaces to which they needed to connect to provide a full solution, on the other hand, was quite broad. The gateways could therefore only meet the needs of a limited number of large enterprises.
Fortunately, there has been movement on both fronts. As Gateway features evolve and the applications they touch provide newer interfaces, the ability to meet a customer’s needs greatly increases. I would say the market can now meet the needs of 85-90% of customers with very little customization or product enhancement; this is true of most of the SOA Gateway vendors. And when there’s a data format or service interface that doesn’t conform to the supported standards, the Layer 7 Gateway supports a custom component SDK for customers to incorporate it into our policy language for further processing. Similarly for new transport protocols, we have made it very easy to extend our network-layer support to meet customer demands. At the same time, packaged app vendors are modernizing their interfaces as well, meaning there’s no need for complex additional-cost adapters. This is true of major packaged app vendors (SAP, JD Edwards, Peoplesoft, etc.) as well as many of the smaller players. Just type “YourPackagedAppName web services” into Google, and you’re likely to find an easy process for integration.
Beyond general integration support, much of the recent interest in a more lightweight, agile alternative to traditional ESBs is due to changing business climates and technical platforms. IT shops are being asked to do more with less and quickly respond to customer demands. The days of the three-year services engagement are quickly becoming a thing of the past, as turnaround times need to be measured in weeks or months rather than years. Configuration-driven gateways that can expose existing data in novel ways generate new income quicker than long-term projects with requirements that could change before they’re ever completed. This is especially true of new technologies around mobile and cloud. Traditional ESB implementations aren’t easily portable to cloud environments and are often more focused on 30-year-old technology than 3-year-old technology. Layer 7 Gateways are portable to the cloud as either a VM deployment in a public or private IaaS/PaaS environment, or as a public marketplace instance in the Amazon EC2 or vSphere environments. On-premise gateways can connect via simple templates to SaaS applications in the cloud, and we support new identity and access control mechanisms used by both cloud and mobile interfaces, such as OAuth, two-factor authentication, etc.
Obviously an SOA Gateway can’t and shouldn’t do everything, just as any other ESB-targeted product shouldn’t attempt to be all things to all people. When writing code for business logic, standard programming languages running on application servers is still the way to go – there’s no reason to put complex business logic in the ESB itself. When working with complex business-driven rules, a rules engine should be employed to externalize them to business users. When interacting with humans and performing long-running tasks, stick to a BPM solution.
InfoQ: In your article you focus on three main use-cases: “standards-based endpoint abstraction, broad data and transport mediation capabilities and dynamic, intelligent message routing”. There are some who believe data mediation being a functional requirement should be encapsulated in transformation components on the endpoint, what is your take on it?
While the application of true business logic to a service request or API call is best left to the endpoint, there are several compelling reasons for moving data format transformation off the endpoint to a more centralized enforcement or mediation point. These essentially boil down to performance, cost, and complexity.
When attempting to expose a new standards-based interface for a backend application, the go-to choice is often XML exposed as a SOAP or REST service. While self-descriptive and (in theory) human readable, XML is also very verbose and memory hungry when it comes to processing. As a result, parsing, schema validation, and transformation are overly painful in a software solution that’s not built for these tasks. Having a gateway provide this functionality, especially in an appliance form factor with dedicated XML acceleration hardware, reduces the overhead on the application itself and increases the overall throughput of the solution.
In the real world, this performance impact equates to real dollars. While many legacy platforms are beginning to modernize and provide new interfaces on their own, this progress is slow and counterproductive. As an example, recent versions of one leading mainframe transaction product have added a “for Web Services” designation and developed a way to map an application to a defined Web service interface and perform the data format conversion on the mainframe itself. So first you have to make sure you’re running a recent version, which at an enterprise like a bank is no small task. Then you have to dedicate development resources to write, test, and deploy the code that maps between the mainframe formats and the XML interface. Even in the most successful case, when everything is in production and working as intended, you have now increased the processing overhead on the mainframe. Why add MIPS to the mainframe, resulting in additional cost, when you can externalize this transformation to a dedicated resource in a distributed architecture?
Lastly, relegating all transformation to the endpoint also puts a heavy burden on your IT resources. Even in a simple case where an application provides a REST interface and a customer decides they would prefer a SOAP-based Web service, your development team suddenly has to support multiple interfaces for the same underlying application. The underlying data is the same, but they’re writing additional code to support one customer. Then what happens when a new version is released but you have to support the old version as well. This suddenly becomes a development, deployment, and management nightmare, all built into the application itself. The transformation capabilities within an SOA Gateway can expose new mappings such as SOAP-to-REST using simple, standards-based transformation capabilities and configuration-driven manipulation of HTTP verbs – no code necessary. They’ll support multiple versions running at once, dynamically chosen based on user, namespace, or even message content, all connecting to a single backend interface and completely transparent to the end user.
In the end, it’s very difficult for any endpoint-based data mediation to approach the simplicity, scalability, performance, and Return on Investment of an SOA Gateway performing the same role.
InfoQ: Are there any non-functional advantages apart from OOTB support for secure protocols that SOA gateways should replace ESBs?
Security was really just a starting point for SOA Gateways being used in this capacity, but it’s still an important factor. Transport-layer security, message-layer security, threat protection, identity and access control – security is truly a first class citizen, and the gateways have the certifications to prove it. I’ve already talked about the performance benefits of doing transformations and protocol mediations in a dedicated appliance, and the ability to just drop a new appliance into a cluster when you reach a saturation point is vastly preferable to rolling out a new software installation. In fact, there are actually environmental and cost savings in terms of datacenter capacity, electrical consumption, and management overhead when you start replacing tens of traditional application servers with a single purpose-built appliance.
Ease of use and flexibility are perhaps the most important advantages. SOA Gateways provide the ability to create policies that perform complex operations like digital signature validation, threat protection, transformation, and context-based routing, all without writing a single line of code. Functional requirements are the primary consideration, but it’s the non-functional requirements that truly make SOA Gateways a preferable option.