SOA Governance: Crucial Necessity or Waste of Time?

The term "governance" has been regularly appearing in IT publications and conferences for some time, but among technical circles, such discussions are often yawn-provoking at best. This article provides a developer-friendly guide to SOA Governance, starting with the general notion of IT governance down through design-time and the second runtime Governance.

Imagine an organization with an ongoing SOA initiative. Everybody is excited, the initiative has a generous budget, there are new business and technical opportunities - it smells like change, but with a certain amount of pressure on everybody. In the midst of this situation we find "Mrs. S. Governess", who is responsible for ensuring that everything concerning services runs smoothly. She cares about the SOA big picture & the interests of the organization. Our governess will not be involved in day-to-day business- or technical details, but she will setup strategies, define responsibilities and constantly monitor the initiative. She has to ensure, for example, that SOA budget is spent to achieve the best possible ROI and that business services are supported as best as possible by the service implementations. Although well respected throughout the enterprise, Mrs S. Governess can be pretty strict sometimes, overruling even project managers or experienced architects. In the end, be assured, she has the direct line to the uppermost management levels and the shareholders... which makes her arguments pretty convincing. In other words, Mrs. S. Governance wields the insignia of power, at least concerning the corporations' SOA.

But wait: Isn't that the management's job? Shouldn't management care for SOA success? In theory, yes. In practice (which most often differs from theory, as you know!), managers usually have their own (sometimes even hidden) agenda, which too often collides with the organization's long-term goals. For example project managers put more emphasis on their projects' schedule than on strategic goals. Therefore Mrs. Governess represents some kind of super-management, being devoted only to the organization itself. She encourages and enforces "desirable behavior". Her job is about control, about law and order - not about being polite and gentle.

Her job description serves as blueprint for the new discipline "SOA Governance". It's not about creating fun at work or having the latest technologies in a developer's hand - but about the relationship between SOA-budget-spent and return-received. That's why governance isn't too popular among us software developers and architects. Their lives (it's mine too, btw!) will probably be less fun after the governance program has been set up. But be assured: It'll keep your organization or enterprise fit and healthy for the future - and that's worth a little hassle.

Before we go into the details of SOA governance, let me explain one of Mrs. Governance favorite axioms from her vast knowledge repertoire: "Enterprises and organizations need two seemingly contradicting things:

At first, they need "order and control": they need laws, jury and a and police/enforcers.. That's where governance comes into play. Be warned: As software developer or architect you might not like it, but you'll absolutely have to have it!

Secondly, they need freedom, room for creativity and a positive working environment, especially for brainworkers. The whole "agile" movement is targeted at improving this aspect of work.

Especially within IT organizations, the order-and-control aspect is often regarded as being counter-productive with brainworkers. I'm not talking totalitarian here: Governance tries to add the appropriate amount of control, with the idea of optimizing the (often cited but seldom achieved) business/IT-alignment. Pretty often, IT people don't care about business and business-people seem to ignore us IT guys. But only by close cooperation can they ensure enterprise success!

Therefore, governance is a crucial necessity. The right amount of governance makes enterprises successful. Harvard professors Weill and Ross could prove in [Weill-Ross] that successful IT-governance implies higher revenues! A lack of governance implies a high risk of long-term failure.

The context

Before we dive deeper into details, we shall look at the whole governance clan (see figure "Governance hierarchy"): Mrs. S. Governess lives in a larger family. Her big sister calls herself Corporate governance, and that lady is nicknamed "Mrs. Compliance". Her goal hangs only an inch below world peace: Ensure that everything happening in the enterprise is directed toward its benefit. In short: She's the one caring for corporate values and assets.

Figure 1: Governance hierarchy

Next in line comes the smaller sister, the IT Governess. Her tasks focus on the relationship of IT and business. Many online and paper publications describe her job (IT Governance) in significant detail, for example the "IT Governance Institute" (ITGI) with [ITGovBB] and the excellent book "IT Governance" by Peter Weill and Jeanne Ross [WeillRoss]. Her responsibilities and tasks determine those of her little sister, Mrs. S. Governess we met before.

The Figure "Focus areas of IT governance", taken from [ITGovBB], gives you an impression of IT-Governance tasks. Pretty high-level, huh?

Figure 2: Focus areas of IT governance

IT-Governance

IT governance provides the foundation for SOA governance - in SOA organizations both should go hand-in-hand. IT governance ensures that all IT-related activities are directed towards the organization's goals, supports its long-term well-being. IT-Governance gurus Peter Weill and Jeanne Ross brilliantly define IT governance as "decision rights and accountability framework to encourage desirable behavior in IT." (from [WeillRoss]). Say it again: Governance encourages desirable behavior. It provides the appropriate "order-and-control" framework which leaves enough freedom for the business to flourish and applies the necessary level of control over individuals and processes to avoid anarchic and chaotic behavior.

When I started with governance, I found it really difficult to boil down theory to practice. Therefore let's have a look at a practicle example of desirable behavior - taken from [Ashar+07]:

"Why have so many hybrid vehicles been registered in the state of California in the last 12 months? Is it the more than $1500 in federal tax credits given to hybrid owners, or the luxury of cruising down the high-occupancy vehicle lanes solo during commute hours? Or is it that Californians are becoming more environmentally aware? Whatever the true reason, the reality is that policies were put into place to encourage desirable behavior--the purchase of low-consumption vehicles. This is an example of governance: when policies are put into place to encourage desirable outcomes."

Easy, isn't it? Now you'll ask exactly the right question: How can IT-governance achieve that goal? How can "desirable behavior" be achieved in IT?

The answer lies in four questions that have to be answered by Madame IT-Governesse:

1. Which IT-related decisions shall be taken?
2. Which roles or people shall carry out those decisions?
3. How shall these decisions be taken?
4. How are the results to be monitored?

Doesn't sound like too much work, does it? The figure "Key-governance-issues" summarizes those key governance issues:

Figure 3: Key Governance Issues

Benefits of IT-Governance

But, you may ask, "why such a fuss about it?" What benefits result from IT governance? I'll show you a pretty impressive list of advantages (don't hesitate to forward it to your managers...)

• Higher Profits: Independent studies have proven that profits are higher in organizations with effective IT Governance - Harvard Business School quantifies it to be in excess of 20 percent.
• Control over Capital Investments: IT spending is a major capital investment - many enterprises spend around 50 percent of their annual investment on IT.
• ROI and Value Achieved: Current organizations invest in IT, but very often fail to monitor the value created by their investments. Governance implements monitoring mechanisms to facilitate ROI measurement.
• Effective Decision Making: The still rising number of people involved in IT-related decisions provides a good reason to have effective decision-making processes in place. For example, even business line managers influence IT spending today.
• Balanced Conflict Resolution: Conflicting business goals of cost-effectiveness and flexibility require balancing across the organization, which IT-governance is happy to provide.

Just in case your organization or corporation still hasn't got any IT governance in place: Your SOA initiative gives you the perfect reason to start both IT and SOA governance today...

SOA Governance

Now that we had a brief overview of IT governance, what does "SOA Governance" really mean? I assume that you are familiar with Service Oriented Architecture (SOA) in general. An organization adopting and implementing SOA needs to care about a plethora of issues within the whole service lifecycle. I'll name just a few:

• Business process design: Create business processes in a service-oriented way, analyze their functional and nonfunctional requirements.
• Re-use services to leverage existing investments and increase overall value of existing services.
• Offer services (with defined service level agreements) to other organizations and/or clients.
• Assure quality of services and test them, with respect to both functional and non-functional requirements.
• Architect, design and implement technical service infrastructure, enabling re-use of fundamental services like transformation, encryption, compression, authentication, authorization.
• Appropriately document both business- and technical aspects of services, with respect to service consumers and providers.
• Monitoring of service behavior at execution-time: How often is a specific service called, with what kind of parameter, from which consumers? Did they use the premium (paid!) variant, or did they stick to the freely available version?

There will be many people involved in all those issues, literally dozens of documents, models, logfiles and other artifacts created during the long way from business requirements over service implementation to service provisioning and operations.

At the beginning of this article I talked about the need for control - I'll come back to that term here: SOA needs tight control over its documents and artifacts. Everything needed to get and keep services operational needs to be governed - simply to avoid the not-invented-here syndrome and to maximize re-use. As (agile) software developers, you might not like it - I predicted that at the beginning... But if your organization wants SOA to be successful in the long run, you need that control! Just in case you don't believe me: Even the techno-friendly guys from Gartner Group predicted that most SOA-initiatives are likely to fail due to a lack of governance than due to technical reasons ([Gartner]).

Now let's get down to necessary actions: There are two distinct phases (Lori MacVittie called them timers in her excellent online article [MacVittie06]) when you need to control your SOA (or your IT in general): The first being design-time or development-time and the second runtime or execution-time. You should care about both, as they have a few crucial aspects in common (namely metadata!) and will therefore influence each other. Let's discuss them in turn.

Design-Time Governance

Design-time governance controls all phases and activities around the software development lifecycle. It begins as early as requirements management and spans over architecture, implementation, test, quality assurance, documentation - until your service is operational. Imagine a business- or requirements-analyst jotting down the required features of the next-generation business service. Design-time governance ensures that the analyst will find every piece of information on related stuff that's already available - cutting down the time he needs for his job. Slightly afterwards, your software-architect or service-architect starts designing an appropriate solution architecture. Again, governance ensures that existing artifacts, documents, (UML) models or even service contracts are readily available (making searches quicker and facilitating reuse at the same time).

This form of design-time governance has its price: Control over every asset (like documents, models, presentations, bug-reports, evaluations, concepts and especially service-descriptions, service-interfaces and the like) created or used in the whole development cycle. Many developers will regard this kind of control an impediment, not a kind of support. An organization has to think and act with a long terms outlook - educate your developers accordingly and give them the proper tools (see below), which minimize the impact of this control.

Finally, when service implementation is completed, service level agreements have been negotiated and everything has been appropriately documented and tested... your service is ready to run! Now the second phase of SOA governance begins.

Runtime-Governance

Runtime-governance covers everything about service execution and operation. You need to recognize which of your services are called, by whom, with what kind of parameters. Your runtime environment should detect potential performance bottlenecks before they occur, should care for agreed service levels on both provider and consumer side, observe logfiles and exceptions. In short: It should constantly monitor every aspect of service execution. In contrast to design-time governance, these tasks will be carried without many people involved. People, especially Mrs. S. Governess, need to evaluate condensed reports on the results. They will receive loads of feedback for the next service design and implementation iteration (you work iterative, don't you?).

Governance Tools

The software industry responded with two different categories of tools to the demand for SOA governance support: registries and repositories. Both can and should be used to store service metadata, that is everything about services. Don't care about terminology here: You'll need proper tools for both design- and runtime SOA governance. From my (vendor-neutral) viewpoint it makes sense to establish your own SOA processes first with some smaller projects or prototypes and start your SOA-governance-toolchain shopping tour afterwards. But: Do not neglect appropriate support for your governance. Strive for smooth integration into your specific SOA lifecycle - otherwise your stakeholders won't accept it - in that case even the nicest Mrs. Governess won't help.

What features shall a SOA governance tool-chain provide? For the answer, re-read the preceeding paragraphs: You'll need both design and runtime support. For your specific feature list, look at your specific SOA lifecycle (yes, it will be your specific one). Every role and activity in there should receive its appropriate support. Easy to write, hard to achieve! Nevertheless I strongly suggest to evaluate tools to support Mrs. S. Governess - otherwise your SOA will likely mutate into a service mess with maximized entropy and zero ROI.

Summary

Now you might think: "SOA Governance is easy. Just get a cool tool and we're set". Wrong. Entirely wrong. Hear out our experienced Mrs. S. Governesse again, she proposes the following steps to effective SOA governance:

1. Get your organization-specific SOA process up and running. Design and develop a few services, gain experience in consuming and providing services. Establish roles and assign responsibilities for your SOA activities.
2. Evaluate which types of artifacts (document types) are useful in your SOA process. Subject these to SOA governance - control them!
3. Evaluate which types of information you need to monitor at service-runtime. Do you need to track SLA's? Do you want or need to inspect logfiles for service-calls? Again, your specific requirements count.
4. Align SOA governance with IT governance. Pretty likely your IT governance will profit from your SOA governance program (and vice versa). Call that synergy - surely a good thing to have.

With these few steps, you can set up very effective SOA & IT governance programs. May Mrs. S. Governesse be with you:)

References & Links

• Corporate and IT Governance
• [ITGovBB] Board Briefing on IT-Governance. IT Governance Institute, available online: http://www.itgi.org
• CorpGov
• ITGI: Institute for IT-Governance
• [Weill-Ross]: Peter Weill, Jeanne W. Ross: IT Governance: How Top Performers Manage IT Decision Rights for Superior Results, Harvard Business School Press, 2004.
The seminal book on IT-Governance. If you want to read only one piece on governance, I strongly suggest this Weill & Ross. They are precise, concrete, understandable and experts. Additionally they have mastered the art of writing well!

SOA Governance

• [MacVittie06]Lori MacVittie: Understanding SOA Governance. Online: www.intelligententerprise.com
• [Afshar+07]: Mohamad Afshar & Benjamin Moreland: Key to successful SOA Governance, August 2007. Online: http://www.ebizq.net/topics/soa/features/7680.html?page=2&pp=1
• [Gartner]: "Lack of working governance mechanisms in SOA projects will be the most common reason for project failure (0.8 probability). (Jess Thompson, Gartner)"
• [Forrester-Jun06]: Forrester Research, June 12, 2006: "The Scope and Focus of SOA Governance" by Randy Heffner and Larry Fulton with Christine E. Atwood

About the author

Gernot Starke works as independent consultant and coach for software-architecture and agile development practices. His clients from various industries (finance, logistics, telco, manufacturing and public administration) rely on his 20+ years hands-on experience in software architecture, -development and -management. He co-founded the arc42 portal for software architects and authored numerous articles and books, primarily on software architecture. He's co-editor of the (German) book "SOA Expertenwissen". Gernot lives in Cologne, Germany, with his wonderful wife Cheffe Uli and their two children, practices yoga and sometimes (tenor) saxophone. More on http://www.gernotstarke.de.