Quest for True SOA

Since the time I first discovered Quest for Fire by J.-H. Rosny Aîné at the age of twelve, I have been fascinated by the Quest genre. To me, the essence of a quest is a story of a group of assorted characters who embark on a long and perilous journey, pursuing an elusive goal, and end up finding something else, typically more valuable and important than what they looked for in the first place. From The Wonderful Wizard of Oz to The Lord of the Rings, I could never resist being pulled into such a story.

I think Quest serves as a perfect metaphor for the evolution of my views on SOA and sheds some light on why my vision of Governance differs from those prevailing in the industry. Back in the fall of 2004, I was working for a leading vendor in the EAI and SOA Platform space. Back then, SOA was hot, and, according to our marketing, our product was SOA. Needless to say the competition was saying the exact same things about their offerings. However, as we put more and more enterprise-wide implementations under our belt, it became apparent that the marketing message was not entirely accurate. If you take our products, adopt our methodology and implement them following our guidelines and best practices (even using our professional services), the end result would not be SOA. At least not the way SOA was supposed to be. It would work and serve the specific purpose for which it was created, but it would do little to deliver on the original promise of SOA as it had been envisaged several years earlier. And it was that promise, which included versatile, coarse-grained, loosely coupled, dynamically discoverable, easily composable and effortlessly reusable services, that made the concept of SOA so appealing to both business and IT, which in turn made it a must-have on every CIO’s strategic plan, RFI/RFP and product feature list. This is not a critique of our product – we had as much right to call it SOA as anyone else at that time. The problem lay within the definition of SOA itself. When the idea became so popular, there was a rush to declare it “done”. To do so, both the vendors and IT organizations alike were subtly changing the definition based on what their solutions supported. As a result, everyone had SOA, and almost no one benefited from it.

Against this backdrop, my team asked an almost existential question: What would it take to build such a platform that no one could say is lacking anything SOA should have? Our answer was conceived in a form of a forty page whitepaper which defined our vision by describing what we believed to be the core characteristics of enterprise-grade SOA services and the supporting infrastructure:

1. Self-containment and Modularity of services, including modular decomposability, modular composability, modular understandability, modular continuity, and, modular protection;
2. Interoperability between the services, consumers, infrastructure, tooling and the legacy world;
3. Loose coupling between consumers and providers;
4. Support for Orchestration and Composability of services;
5. Discoverability of services and Dynamic Binding between consumers and providers;
6. Location Transparency of services, consumers and infrastructure components;
7. Pervasive Security which by design ensures trust throughout the SOA ecosystem (including services, consumers, agents, composite applications and infrastructure) by addressing some or all of the following security aspects: Authentication, Authorization, Integrity, Confidentiality, Accountability, Identity and Policy;
8. Support for Sustainability and Self Healing which includes ability to detect, predict and mitigate undesirable and unintended behaviors and conditions as well as draw and release resources on demand;
9. Service Versioning which supports temporal evolution of the SOA ecosystem via facilitation of transparent co-existence of multiple versions of service implementations and consumers;
10. Service Lease which explicitly defines and maintains the relationships between service consumers and providers and disambiguates potentially dangerous mutual assumptions;
11. Network-Addressable Service Interfaces with flexible invocation mechanisms supporting the multitude of transports, protocols and synchronicity options required to maintain a vibrant and versatile SOA ecosystem;
12. Coarse-Grained Service Interfaces which can be easily refined to support a multitude of service consumption scenarios;
13. Service usage Metering (both real-time and historical);
14. Monitoring, Management and Control of services and their implementations;
15. Exception and Alert Handling supporting Correlation and Root Cause Analysis;
16. Effective Service Virtualization ensuring complete separation of service Interfaces and Implementations;
17. Published, verifiable and enforceable Quality of Service (QOS) including Performance, Reliability, Availability, Regulatory, Accessibility, Integrity, Security, etc.;

This enumeration also gave us a neat scale for assessing the maturity and completeness of SOA products and implementations. When applied to our flagship product, it yielded 40±5%, depending on how kind one wanted to be to own code. Thusly, without giving it a second thought, we set out to build “the remaining 60%”. And this is what it was called until its release 1.0 in the fall of 2005.

As we started building that thing, we realized that we had missed a couple of characteristics. However, the 19 core characteristics of SOA did not have the same nice ring as the 17, so we started calling them “the 17+” which was how we first stumbled upon the idea of Aspect-Based SOA infrastructure. The premise was that there is a fundamental reason why we could not enumerate all the characteristics of SOA services that would make them suitable as the computing platform for the Enterprise. That reason was that these characteristics were based on the actual business, regulatory and other requirements which were bound to vary by industry, geography, from client to client, and, most importantly, with time. What the market needed was a platform that could realize and support a nearly arbitrary set of service characteristics, and allow the users to add, change and remove them on a whim, or, more appropriately, based on what was important to business today.

This fairly radical idea was validated during our very first client encounter. The client, a defense force, fairly paranoid even by military standards, came up with a laundry list of security-related requirements, most of which we were able to either support directly or delegate back to their own security infrastructure. However, there was one requirement which made me raise an eyebrow: the client wanted to prevent a poorly written service from being able to accidentally disclose classified information to legitimate, properly authenticated and duly authorized users who lack the sufficient clearance. My initial reaction was: hire good developers, do thorough QA, and you’ll be fine! Then, I realized that, from the client’s point of view, dealing with threats of an unforeseen leak was as important as all the other service characteristics which the rest of the industry thought it would be reasonable for them to care about. Within the next hour, we designed a Censorship Aspect that examined the service responses on the way to the client, determined the classification level of the message and/or individual fragments, compared it with the clearance level from the profile which was made available during the inbound authentication, and took a necessary corrective action, ranging from partial obfuscation to total intercept.

In the meanwhile, with the release 1.0 out of the door, the marketing needed a name: for some strange reason they were not keen on starting a campaign around “the remaining 60%”. After toying with SOAi and SOAf (standing for Infrastructure and Foundation respectively), someone brought up Governance, probably because it was becoming a buzzword of the day. I was not familiar with the term at the time, so I googled it a bit and for the life of me could not understand what it has to do with the technology we had developed. My main concern was that there were no clear definition and too many unclear ones.

Most vendors in that space defined SOA Governance simply as “what we can do”: Systinet was pushing the idea that it is a system of record for WSDLs and supporting artifacts coupled with taxonomies, publication workflows and subscription/notification mechanisms. AmberPoint saw it as autodiscovery of dependencies and “rogue services”, light-weight security, monitoring and end-to-end fault analysis. Appliance vendors saw Governance as Web Service Security plus whatever mediation functions they had. The pundits were coming up with cryptic and nebulous definitions, like the one below from the OASIS SOA Reference Model, which would make the Pythia proud:

SOA Governance: the art and discipline of managing outcomes consistent with measurable preconditions and expectations through structured relationships, procedures and policies applied to the organization and utilization of distributed capabilities that may be under the control of different ownership domains.

After giving it some consideration, I realized that there must be an underlying elephant behind all of this and set out to define it. The rest of this article will describe the vision of SOA Governance as the defining characteristic of SOA that ensued. We will also illustrate it with an example.

Since the entire problem domain of SOA is plagued with ambiguous terminology, let us start with laying out some key definitions. First of all, we need to begin with what to govern. The answer seems quite obvious: services! However, this term has been used to describe so many different things, most of which have little to do with SOA, that I feel a need to bring some clarity, which I usually do through qualification.

An Enterprise Service is a self-contained component delivering business functionality combined with an extendible set of non-functional, policy-driven qualities (such as security, industry/customer defined service policies, management, monitoring, and lifecycle management) which responds to requests through a well-defined, standard, published interface.

Another way to describe an Enterprise Service is: A service whose name is familiar to the CEO or line of business owner, and the latter cares about what it does. According to this definition, invoiceCustomer, dischargePatient and launchICBM qualify as enterprise services, while invokeBAPI, saveLogRecord or generateToken clearly do not. In the light of this definition, SOA itself can be described as:

an architectural style that provides and facilitates evolution of enterprise IT towards an ecosystem of Enterprise Services.

And finally, the SOA Governance simply becomes:

A combination of processes, practices and tools which facilitate lifecycle of Enterprise Services and provide means for creating, communicating, enforcing and managing compliance to corporate policies regarding the non-functional service characteristics that are of importance to business today.

According to these definitions, SOA Governance is what turns Enterprise Services from digital artifacts into true business assets by allowing responsible reuse across the domains of control of SOA participants. To make this easier to digest, let me describe a hypothetical example.

Imagine a multinational company Hot Dog, Inc. that has two business units (divisions) - Buns Research and Sausage Development. They have separate management, P&L responsibilities, IT departments and budgets but operate within the same corporate entity.

At some point, the leadership of Buns Research decides to implement a new application on which they intend to run their core business functions. They do so at a cost of one million dollars, and, being good corporate citizens, they spend some additional $50k to package the newly developed functionality in the form of reusable services, and expose it for the benefit of the entire company.

Next year, the management of Sausage Development realizes that they need to build a new application on which to run their core business, and it so happens that this application needs exactly the functionality that had been developed, packaged, deployed and made available to them courtesy of Buns Research six months earlier. [Sounds like a textbook business case for SOA, doesn’t it?] Sausage Development’s IT now has a choice either to reuse those services or spend another million dollars (re)developing this functionality from scratch. I would stipulate that if the underlying technology is say CORBA or ungoverned (according to the above definitions) SOA, the only responsible course of action for Sausage Development’s management is to go ahead and spend the money to implement the required functionality within their domain of control. And here is why: the original services were implemented and are being hosted by Buns Research, so although they happen to be a 100% match from the functional point of view, Sausage Development has no way of knowing any of the following factors, all of which are vital in determining the usability of these services within Sausage Development’s business context:

• What are the expected availability, reliability, MTBF and Time-to-Repair?
• What security and accountability mechanisms are in place?
• Have these service implementations been audited and found in compliance with the relevant regulations (SOX in USA, privacy in Europe, HIPAA in healthcare etc.)?
• What performance levels are supported and what load levels are acceptable?
• Are those constant or do they vary by time of day, day or the week, season, etc?
• How is service usage metered and accounted for?

Even if Sausage Development’s management finds out all of this information from their counterparts in Buns Research, one vital question remains: how do we know that all of this is not going to change next week (month, year)?

When the thought process was complete and had crystallized into the vision of SOA Governance described above, several things became apparent (at least to us):

1. Similarly to the way Einstein’s Theory of Relativity extends Newtonian classical mechanics into the realms where the latter was not applicable, this unified vision of SOA Governance does not negate, but, instead, builds upon the fragmented definitions and approaches mentioned above and described in detail in my article SOA governance – the perspectives.
2. Implemented properly, this vision will yield a solution capable of meeting all of the 17+ core characteristics of Enterprise SOA described earlier.
3. Such a solution has the potential to take the imperfect SOA platforms and implementations of today and transform them into true Service Oriented Architectures implementing the original vision and capable of delivering all the promised benefits.

In the end, building of this Governance platform turned out to be easier than we thought. And, as dictated by the Quest genre, along the way, we made many unexpected and valuable discoveries, such as: Delegated Governance, Analysis-time Governance, Brownfield-friendly Governance and unified Governance Information Model, all of which would make excellent topics for future articles.