"SOA Governance" Revitalized

Reviving an often-discussed but poorly understood topic

The term “SOA Governance” has been used in the industry for years already, but it is still considered an arcane topic. Anyone who reads this article should be able to understand the following things:

1. Why are people pursuing SOA, isn’t it dead?
2. What is the relationship of SOA Governance to SOA itself?
3. What is SOA Governance?
4. How does it differ from Management?
5. How does SOA differ from Integration?

While being able to think and speak clearly about these topics may not win you as many brownie points as it would during the time when this was a “hot topic” for Enterprise IT, it will help you understand why SOA and SOA Governance continue to be significant issues for the Enterprise despite the ups and downs of market hype cycles.

Why are we still talking about SOA?

Anne Thomas Manes posted her famous “SOA is Dead” post at the beginning of 2009. Since then, some of the “fizz” has gone out of SOA. But by making two irrefutable statements, we can clarify why, in the words of Gartner Analyst Paolo Malinverno “SOA is Inevitable”.

1. Enterprise IT continues to suffer from bad architecture
2. Since SOA was “declared dead”, no viable alternative solution for this problem has been proposed.

In the spirit of InfoQ, I challenge anyone to either

1. Propose a viable alternative that addresses the problem of “Enterprise Architecture Syndrome” as defined below. Or
2. Show that a viable alternative has already been proposed
3. Make a case that “Enterprise Architecture Syndrome” is not a significant problem.

Feel free to attack the premise that Enterprise Architecture is in bad shape as well. Please keep in mind that:

1. This is an Architectural problem and requires an architecture solution (so programming styles like Events and Objects solve problems at a different level)
2. Cloud Computing is not an Architecture, it is a delivery mechanism
3. Business Process Management without SOA does not reverse “Enterprise Architecture Syndrome”

One of the challenges in maintaining a paradigm such as “Service Oriented Architecture” is that during the “peak of inflated expectations” in its hype cycle it became a “big tent” under which many many software companies huddled. Everything was marketed as “SOA” in the hopes that it would sell better. Of course the billions of dollars spent helped to blow up the hype around SOA to epic proportions.

Throw out the SOA Baby with the SOA Bathwater

One of the difficulties associated with SOA is the length of time it has taken for the industry, products and implementation methodologies to mature. Because of this achingly long time span, contemporary SOA bears little resemblance to the fanciful ideas of a few years ago.

So the question naturally arises—has the term outlived its usefulness? Wouldn’t the community be better off throwing out both the baby and the bathwater and just moving on to the next concept? This seems a very legitimate approach, but since renaming my blog http://soacenter.com to http://whatevercenter.com in response to Anne’s “SOA is Dead” post, there has not been a viable alternative to solving the disease that I call “Enterprise Architecture Syndrome.”

The Disease: Enterprise Architecture Syndrome

Most large organizations suffer from a degraded architecture that I refer to as “Architecture Syndrome”. Much like the human disease called “Metabolic Syndrome”; this disease has many risk factors including a heterogeneous and aging technology infrastructure, a company that has grown through acquisitions or geographic expansion, an industry undergoing consolidation, a stiff regulatory environment as well as the commoditization of platform software such as ERP systems.

Symptoms include an inflexible IT System (including hardware, software, integration and IT processes) that makes any changes into a nightmare of schedule slips and weeks becoming months. Diagnosis can include the creation of insane-looking architecture diagrams with hundreds of different product names and thousands of object models, data tables, events, processes, organizational charts, components, lines, integrations, connections, network topologies, standards, applications, schemas, documents and other abstractions.

Condition: Entropy

We best capture the degraded state of Enterprise IT Architecture by using the word “Entropy” from the physics of thermodynamics… defined as the amount of energy in a system that is unavailable for work.

There are many ways to measure the energy in Enterprise IT that is “unavailable for work”, but you can really look at it as time and budget wasted. Wasted on throwaway systems, redundant systems, wasted on useless meetings (productive meetings are ok), maintenance and other abuses. Looking at the percentage of the IT budget that goes to move the organization forward, it’s clear that for most large organizations the majority of the IT budget goes towards “keeping the lights on” and leaving not much room for anything else.

So this is a pretty dismal state of IT Architecture. One of the first things people notice when they decide to fix their architecture when they look at it from a global perspective is the number of redundant systems. This is typically looked at as a bunch of “Silos” and the Enterprise wide goal of reducing costs aligns with the idea that these redundancies can be “rationalized”. This cost-cutting alliance between Corporate Finance and Information Technology spawned a field called “IT Governance” and SOA Governance is their red-headed stepchild.

How did Enterprise Architecture get so degraded? People with “Metabolic Syndrome” could ask themselves the same question. Typically the answer is decades of neglect.

Perhaps a deeper question is: what caused Enterprise IT architecture to be abused and neglected?

IT Project Management

The IT organizational competencies and processes that evolved during that era involve a mentality around disposable “projects” and a short term framework around “ROI”. As long as a project could return its initial investment it was a justified expense.

There’s nothing inherently wrong with ROI. However, the way ROI is typically measured in large organizations is typically as a forecast used to justify a project. Even organizations with the discipline to measure ROI after the project gets started rarely measure it over the lifetime of the project.

Let’s face it, a “successful” IT project creates a “legacy”. Very few ROI models measure the value of something over the time span of “until the organization disappears”. These stable long-running “reusable” assets form the platform atop which the business functions.

IT goes through a Volcanic Hot Age

During the mid 1990’s the volatile fuel of Venture Capital and IPO ignited into a “Hot IT Revolution” culminating in the Y2K spending cycle.

Carrying the thermodynamic perspective further, What do we mean by “hot”?

1. Larger overall IT spend
2. Faster change in the technology landscape
3. Less mature technologies
4. Expansionistic—building out rather than cutting back

IT funding and project management disciplines were evolved for this type of environment, one in which time to market was more important than cost.

During an expansionistic phase, the ROI calclulation of “pays itself back” translates easily to “let’s do it!” But like the hangover after partying every night for a week, the longer term consequences of “let’s do it!” need to be considered. Consequences that affect not only the project but the prospects of the whole Enterprise.

IT Goes into an Ice Age

After Y2K, the IT environment began to cool off. The industry, products and marketplace began to mature. Enterprise IT began to enter a contractile period.

If you have fewer pennies to spend, what types of considerations come to mind?

A few of the trends in recessionary IT seem to make Enterprise Architecture Syndrome worse!

1. IT headcount is drastically reduced: Lots of people are laid off, significant expertise about legacy systems is lost.
2. Short Term Thinking prevails: new CIOs are hired who have substantial cost-cutting agendas for their “first 100 days”. Long term planning goes out the window.
3. Draconian measurement systems are put in place: programmers are measured by lines of code produced, QA testers are measured by bugs fixed and everyone starts to focus more narrowly on their jobs.
4. What’s best for the Enterprise is thrown under a bus: the dominant pattern becomes “whatever saves my job/group headcount”.

All of these trends cause even more friction between organizations—people feel more tightly squeezed, and when people feel pressured, their tendency to care about people outside of their group (or in worst cases, outside of themselves) decreases.

SOA is Dead, Cloud is Water Vapor and Enterprise 2.0 is a Crock.

In a period where the industry “cools down”, large trends in IT tend to be based on “coalitions” of vendors associated with themes like SOA, BPM, EDA, Enterprise 2.0, etc. Of course alongside these “cooler” times, the amount of time for technology to be adopted in the Enterprise goes up. Enterprises narrow down the list of “approved” vendors to only the larger vendors and they apply much more stringent critera for approving projects. During this type of environment, vendors huddle together for safety and these hype cycles take a longer time to develop. During this kind of elongated time frames, it become easy for critics to take “pot shots” at the coalitions, especially since most coalitions consist of a cluster of strong and weak players. Some of the weaker players huddling under the SOA banner make themes like SOA easy targets for critics. “SOA is dead”, “Cloud is Water Vapor” and “Enterprise 2.0 what a Crock” have variously been said of these three IT trends.

Why SOA makes sense even in recessionary times

And a few of the trends make the logic of Service Orientation more compelling

1. “Use what you have” becomes a key. Getting the most out of the IT infrastructure you have becomes a battle cry for the Finance group. This means a “leave and layer” approach to developing new IT capabilities. “Rip and Replace” goes out of fashion. Finance will not buy IT any fancy new toys until it shows that it can play with the ones it already owns
2. Consolidation becomes attractive. Any redundancy or perceived redundancy in IT systems is looked at as an opportunity for consolidation. Where there were two, now there is one. This is where the concepts of “rationalization” and “reusability” come from.
3. IT becomes more centralized. Instead of having dozens of buyers, central IT buys infrastructure in bulk. Just like a recessionary family buys in bulk at CostCo, a recessionary enterprise buys IT in bulk. Do the business units get as much say about the brands it prefers and features? No, but by amassing all purchases through central IT, the costs go down. The procurement office rises to ascendency.
4. Opportunity Cost becomes important: The criteria changes from “let’s do it!” to “is this the best use of our time/resources?” Opportunity cost becomes a serious issue since multiple projects compete for the same penny. “Software Portfolio Management” becomes more popular.

In geology, heat and pressure make diamonds (which we know last “forever”). But periods of prolonged cooling produce sedimentary rock—rock that is formed from layers.

But at the same time IT begins cooling off, the Business World is also cooling. Large enterprises are forced to compete in more mature markets.

Business gets Process-Oriented

The definition of a recession is multiple quarters of negative growth. So the impact to the business as a whole is either less growth or negative growth.

During the throes of such recessionary trends, one naturally sees a shift from product innovation to process innovation. After World War I, Henry Ford developed the Assembly Line that radically altered the field of Business Process Management and the Assembly Line created process flow visibility by making inefficient segments of the assembly line obvious to any observer. This innovation excelled at creating non-differentiated commodity products. Henry Ford said of his Model –T “you can have any color you like as long as it’s Black”. The industrial growth sparked by this and other such innovations launched the “Roaring Twenties” economic boom.

Across the Pacific, another stalwart in Business Process Management, Toyota uses Kaizen (continuous process improvement) to transform itself from a producer of the “Toyoda Model G Automatic Loom” into a car company.

Difficult economic times provide impetus for a significant push around “Business Process Management”. The emergence of Business Process projects on the business side has led directly to the connection between SOA and BPM. Early efforts to connect SOA and BPM through standards such as BPEL fell quite short when practitioners realized that what business people thought of as their processes were not represented by such low level programming languages. Although BPEL served its purpose as a low level orchestration flow language and a declarative way to express endpoint semantics, true business processes were not expressible in BPEL leading to the rise of such efforts as BPMN and others.

Business shifts to Niche or Vertical Markets

It’s a well known fact that as companies mature and markets mature, growth slows. So how do organizations compete in slower and more mature markets? The best contemporary description of product marketing for mature markets can be found in Geoffrey Moore’s book “Dealing with Darwin: how great companies innovate at every phase of their evolution.”

In this book Moore describes a mature market where product and service fragments into niches as a “Fractal Market”. He suggests that once a market is saturated with a single product, that product competition drives niche product development and creation of products with (for consumer) narrower demographics or (for enterprise) more verticalized solutions. The example he provides is that once every household has a “regular” telephone, the market is forced to product dozens of niche variations including car phones, cell phones, game phones, PDA’s, camera phones, etc.

An extreme form of niche marketing was documented in Chris Anderson’s book “The Long Tail”. Prior to this book, the organizational focus on the Pareto Distribution was on the top 20% of the customer base that makes up 80% of the revenue. The “head” end of the distribution curve deserved the lion’s share of attention simply because focusing on a smaller number of customers costs less, and empowering those customers gains you 80% of your revenue.

In “The Long Tail”, Chris Anderson explores the 80% of customers that typically account for only 20% of revenue. By studying internet giants such as EBay and Amazon, he explored the potential of the “long tail” and the ability for organizations to focus on serving the 80% of the customers on the “other side” of the Pareto distribution.

The inherent problem with serving such customers is being able to differentiate products at a finer level of granularity, essentially niche marketing. One solution to this problem is to develop a core set of “platform” services that can be used and recombined in billions of possible ways, and by pushing those combinations closer to the customer. This means personalization, mass customization, self service and combinatorics. Starbucks coffee, Build-a-bear workshop, Apple iPhone AppStore, FaceBook Apps, eBay Marketplace, Dell.com “custom built” computers and multi-colored iMac Computers and iPods all fall into this category.

Near-infinite consumption patterns with finite provisioning

One of the properties of fractal shapes such as Benoit Mandelbrot’s famous “Mandelbrot Set” is that it is a 2 dimensional shape with infinite surface area that bounds a finite volume. In market terms, a product that has “infinite surface area” means that every single consumer experiences the product in a unique way, but the underlying supply chain is the same. An example of this kind of product is Starbuck’s coffee. Many millions of possible drinks can be composed (e.g. Venti nonfat sugar-free hazelnut decaf latte with an extra espresso shot, served hot) but these variations do not disrupt the infrastructure requirements to deliver this product at the store. Hot water, milk and coffee beans go to the store and the biggest complexity is maintaining sugar-free hazelnut syrup on site.

In business terms, the same hot water, coffee beans and milk are “mass customized” into millions of different products. Notice the term is not “reused”.

So whether business demands process oriented consumption of IT services or other forms of mass customization such as mash-ups, “platform” services, utility services, App Stores or self service, we are talking about the emergence of complex consumption patterns for IT (Complex Demand)

Enter the Alternative IT Provider

Business has complex demands for IT. If IT is unable to supply this demand, a host of alternative sources of IT capabilities emerge. Among these include traditional outsourcing, Cloud Computing, Offshoring, and a host of emerging options for business leaders.

Although the relationship of the business to an external IT provider is usually more well defined that the relationship of in-house IT to the business, outsourcing is not a panacea.

Other than in smaller organizations, the wholesale outsourcing of all IT functions to external suppliers is not realistic. In cases where only part of the IT portfolio is outsourced, this adds additional complexity to the model. Now organizations must “organize and utilize” capabilities from on-premise and off-premise and manage new organizational agendas. Although external providers of IT may keep internal IT “in check” and force them to compete with aggressive outsiders, ultimately the wholesale outsourcing of IT puts an organization into a vulnerable state where the external provider has an excess of bargaining leverage over the Enterprise.

Complex Demand, Simple Supply

Consider the example of the electricity grid. Although the standardization of the AC power outlet (it’s irritatingly heterogeneous across countries) provides for billions of different appliance configurations that can be plugged into it, the supply of electricity is a reasonably homogeneous problem. Although energy has many sources such as fossil fuel, nuclear, solar, wind power, etc, once it has been translated into electricity it has become a commodity.

One rational business response to commodity market is monopoly. We can build an IT delivery system that has a similar complex pattern of consumption but has a large monolithic supplier. If managing a supply chain of heterogeneous IT suppliers becomes too complex and expensive, a monolithic supplier of IT becomes a reasonable alternative.

One example of this is the emergence of the acquisitive Oracle Corporation who is attempting to build a single technology stack to commoditize IT. The problem with this approach is that IT is not a commodity in the sense that electrical power is. The biggest issue is not that an ESB from Oracle is so dramatically different from an ESB from some other company—the issue is that in the provision of electricity, no supplier has extraordinary pricing leverage as a function of high switching cost. So the solution of a monolithic or monopolistic IT provider in order to simplify the complex IT supply chain is a difficult solution. Another problem with this model is that it would require an already heterogeneous IT landscape to “rip and replace” their solutions with those of the commodity provider. Although electricity is highly interchangeable, the history of IT has proven that swapping out IT providers is difficult.

Perhaps a more viable model for simplifying the supply of IT is Cloud Computing. By building very large “utility” companies for IT infrastructure, we can simplify the supply of IT services into the Enterprise. This is an excellent model, and the shift to Cloud Computing enables a huge cost benefit via commoditization and economies of scale. Very large providers of Cloud Computing will emerge and due to the disruptive pricing model, new entrants will likely surpass the old providers in this field.

Simplified delivery of IT capabilities is a wonderful cost-saving device. It’s well known that IT in the Enterprise is too complex. Unfortunately, the Enterprise is already filled with heterogeneous and complex IT supply chains and in order to keep the business running, it’s difficult (or near impossible) to switch off those systems. One future scenario might be that the inefficiency of complex Enterprise IT is so prohibitive that new business entrants fueled by “clean” utility IT providers in the cloud will outgrow and replace the old Enterprises.

This model does not speak to the logical response of Enterprises to the emerging need to support complex consumption patterns for IT Services. The correct chess move for large dinosaur enterprises is not to roll over and die in favor of small mammal enterprises. It’s certainly theoretically possible for large enterprises to “spin out” dozens of agile Cloud-powered mini-me enterprises. But radical moves such as this are typically not well rewarded at the Board and Executive Management level of most large organizations. So Large Enterprises will need to come up with a solution for IT that combines both complex supply as well as complex demand.

Don’t get me wrong, Cloud Computing is happening and it’s going to happen big. Most if not all large enterprises will have significant investments in Cloud. But most large Enterprises will be unable to rip and replace existing IT infrastructure with a monolithic and simple cloud solution. This results in a complex IT supply with complex demand problem.

Complex Demand, Complex Supply

If you consider the Starbuck’s Coffee example, Starbucks sources coffee from many sources all over the world and manages the complex relationships between thousands of suppliers to deliver a well-branded experience to the consumer. Although coffee beans are arguably a commodity like electricity, the ability for Starbucks to offer supply transparency (Shade-grown Fair Trade Ethiopian Coffee Beans) or at least the simulation of transparency to the consumer requires complex supply management practices.

So for IT, how do you take this complex legacy history, complex sources of capability and layer on a new set of capabilities that can be recombined the way Starbucks does with their coffee?

IF you’re challenged with a complex IT supply and complex IT demand situation, SOA is the only rational response. In the spirit of InfoQ, again, please feel free to challenge that statement.

Tribalism

If the supply of IT were just a problem of managing monopolistic vendors trying to lock you in, it would still be very challenging. As if that were not hard enough, even within Enterprise IT there is increasing heterogeneity and complexity in providing capabilities to the business.

As described in my free eBook, “SOA Adoption for Dummies”, one of the significant challenges in managing in the Enterprise is the existence of “Tribes”. A “tribe” is a self-interested organizational unit. Each tribe has its own leadership, budget, priorities, performance metrics, world view, and systems.

Tribes fall into a pattern where they behave in ways that “good for me, bad for you”.

This is not garden-variety selfishness. In this difficult economic environment, not only does management obsess about squeezing every dime of value out of every employee, but each group manager is fighting to keep all of their headcount. The combination of draconian measurements and individual contributors trying to save their jobs leaves very little room for thoughts of other groups. The metrics each group are measured by drive that group’s behavior and little else.

If you want to talk about complexity of supply, the existence of Tribalism both within and outside of IT organizations dramatically increases the complexity of supply.

Defining SOA

The statement that SOA is the only rational response to these myriad concerns is perhaps overstating the point. Lets explore why.

At this point, the astute reader may wonder if I’m playing fast and loose with the definition of “Service Oriented Architecture”. So I would like to refer to the OASIS SOA Reference Model definition, which states: SOA is a paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains.” (Emphasis added).

How is SOA Different from “Integration”

If the aim of SOA is to “organize and utilize” capabilities, presumably through a “leave and layer” strategy, how is this fundamentally different from Enterprise Integration?

The “baggage” associated with the term “Integration” is simply that it concerns the interoperability and interconnection between systems and applications. While this is certainly a useful set of competencies for SOA, what makes SOA a different layer of abstraction is hinted at in the Reference Model Definition. The notion that the distributed capabilities may be “under the control of different ownership domains” is at the heart of SOA, and in fact poses the greatest challenge to SOA Adoption, the realization of the SOA benefits.

Of course in practice, SOA operates at a higher level of abstraction then Integration, and covers a broader set of agendas… and Integration sophisticates already understand the problem of ownership domains. But it’s the reason why we need a new word. Whether we need a new word for SOA in order to dispose of the “baggage” associated with it is subject to debate. InfoQ readers, have at it.

Reining in SOA

The idea that an Architectural paradigm can “organize and utilize” capabilities does seem like a bit of a tall order. This misunderstanding is one of the failings of SOA as realized today—that Architecture by itself does not organize and utilize capabilities. Architecture does not produce such effects; Architecture done properly only creates Architectural Blueprints.

An Architectural Blueprint is sometimes referred to as the “To-Be” or “Target State” Architecture. Without an aggressive program of implementation (which is often referred to as “SOA Adoption”), the Blueprint will remain a pipe dream.

So suggesting that SOA (which isn’t even an Architecture, it’s an Architectural Paradigm) by itself will produce positive business results is like saying that the Postmodern Architectural movement will provide housing for thousands of people. Before you realize the business benefit, you need to instantiate the architecture in the form of your blueprint. Even more to the point, you need to realize your blueprint in the form of SOA Adoption.

The SOA Baby in the Bathwater

If you follow the earlier phrase about not throwing out the SOA Baby with the SOA Bathwater, what fundamental insights do we need to preserve in order for the promised benefits of SOA to realized?

Let’s go back to the definition and the words “ownership domain”. The realization that the distributed capabilities may have different owners changes the equation from technical “reusability” into a problem of creating a framework of agreements that support sharing.

What about reuse?

The concept of “rationalizing silos” speaks to the idea that we have one service instead of two. You could look at it as that single service being “reused” two times.

This is a fairly unproductive way of looking at it.

What gets people’s blood moving these days is the idea of “good use of time/resources”. Reuse is such a dead tired word that it hardly ever gets people interested in SOA. We spoke earlier about “Entropy”—the amount of energy in a system that is unavailable for work. Another way to describe it is waste.

If you take a look at the “good use of time/resources” measurement, any project that:

1. Produces a redundant function (bad use of your time)
2. Will be thrown away (bad use of your time)
3. That is limited to one business use case (bad use of your time)
4. Produces a security risk (bad use of someone else’s time)
5. Crashes or otherwise fails (bad use of someone else’s time)

It’s typical for someone who is part of a siloed organization to not care as much about other people’s time—this is a natural product of working in a large enterprise where responsibility is diffused. It becomes “somebody else’s problem”. Ultimately it results in lots of waste—code being ripped out, underused, marginalized, consolidated and otherwise wasted.

Complexity is not the Enemy, Entropy is

We spoke earlier about the complex supply and demand challenges facing IT. The complexity of Enterprise IT is a direct function of the size and longevity of the Enterprise. Although size and complexity is correlated with failure, complexity itself is not the enemy, Entropy and waste are the enemy. When we look at a system that converts energy into form, we can take inspiration from the Biosphere. The Biosphere takes in solar energy and converts it into all forms of life on earth in its tremendous complexity. Complexity in the biosphere is a measure of its strength, adaptability, scale and resilience. One could argue that Enterprise IT is a “closed system” and therefore does not have the boundless solar energy input needed to reverse local Entropy as we see in the earth’s Biosphere. But that argument doesn’t hold water—Enterprise IT continuously takes in energy in the form of IT budget expenditure. Large Enterprises simply aren’t known for “pulling the plug” on IT, and in the most extreme cases will simply outsource as much of it as possible—but energy is still being expended. Much like the Biosphere, complexity is a measure of success—it demonstrates the ability to support tremendous business variation and for an organization to attack a larger number of economic niches. Those who hate “complexity” often confuse organic complexity (which is simply a beneficial form of complexity that they just don’t understand) with overcomplexity. They may try to drive the system into a state of oversimplicity. Both overcomplexity and oversimplicity are forms of waste and therefore Entropic.

Services hide complexity

The concept of a “service” is simply the notion that consumers are shielded from the complexity of the implementation. Because of this, SOA can be seen as a way of hiding complexity. Hiding organic complexity can be useful in situations where the total system complexity makes visualizing solutions difficult for the business. However, in a business context being “shielded” from complexity essentially means abdicating control over the situation. There’s actually nothing wrong with abdicating control over a process in the business world, as long as you trust the provider or the framework of accountability. What I mean by the framework of accountability is that we “trust” that other drivers on the road will not smash into us because we “trust” that they do not want their insurance premiums to go up and the hassle of having to deal with a crash. Similarly, provider/consumer contracts can provide a framework within which consumers don’t have to explicity trust a given provider but can trust the framework of accountability.

But they require trust

Ultimately, the trust issue is the biggest stumbling block for SOA. What we’ve learned is that lack of trust in business is tremendously expensive. The speed, coordination and agility exhibited by high performing teams is a function of trust—team members pass a ball or puck to where they “know” their teammate will be—without having to look. Lack of trust means that agility grinds to a halt. Everything must be verified, double checked and legislated. There’s an old saying that if people are pulling contracts out of the filing cabinet to re-read them, it’s a sign that the business relationship is ending. Similarly, in SOA overgovernance can produce as much entropy as undergovernance. Ultimately it comes down to waste, waste is the enemy.