Software AG has released a user survey analyzing the best practices for SOA Governance. Software AG got responses from a wide range of customers. One of the key takeaways from this survey is that SOA is real and SOA is happening in the enterprise on a large scale. Connected systems are a reality and they are delivering value. The survey identified that
SOA Governance plays a key role in creating sustainable, enterprise-wide implementations
as 91% of the respondents indicated that governance was critical or moderately important to their SOA strategy.
The respondents agreed overwhelmingly to say that the following tools were needed for SOA Governance:
- Registry
- Repository
- Security
- Policy Management and enforcement
- Service Lifecycle Management
- Performance Monitoring
- Testing
The survey also asked which standards were important for SOA. WSDL and SOAP got a score of about 90% followed by UDDI, WS-Security, BPEL 2.0 and WS-Addressing. REST, SCA and JBI got a score of about 10% or less.
InfoQ asked a few questions to Miko Matsumura, VP and Deputy CTO at Software AG to get his opinion about the survey's responses
InfoQ: The survey's introduction does not mention the word "reuse" is it an oversight or on purpose?
Miko: We tend to be very careful with the word "reuse" because the word has different meanings depending on who you speak to. If you are a developer, people don't really believe it. They have heard that word before in reference to "reusability" of Object Oriented code libraries. Obviously reusing a static library that needs to go through all of the pains of Enterprise lifecycle provisioning, deployment, quality assurance and everything else is very different from connecting to an already deployed service--but developers don't see it that way. It is almost always easier to crank out new lines of code than to "reuse" anything. If you talk to someone in the business, it is hard to get them excited. They are more excited by increased agility and speed of development. They are more responsive to the idea of being able to connect chunks of functionality into business processes. The thing in common between business and developers is that both are a bit tired of how complex and slow the IT restrictions are around policies and processes. Neither one particularly responds well to the term "Governance" either!
The only group that responds positively to the word is the finance function. They are excited about it because of the cost model--the concept of turning a recurring cost (IT) into a reusable asset is very appealing to them and they don't have the cynical experiences of the past. They just think... hmm this is obvious, let's do it.
So I would be cautious to use a word that has such different meanings to different people.
InfoQ: What have been the major hurdles people encountered in their SOA?
Miko: There are 4 or 5 that are really important. One very important one is funding and being able to justify both the initial and the ongoing effort. It gets to be a very interesting challenge, people tend to become fairly enthusiastic about the architecture blueprint. Everyone is always a bit upset about the as-is architecture, there are so many things to complain about today. They love their to-be architecture. However, architecture is never something the business gets excited about. The business has a mentality that is part of the struggle of enterprise IT: what's in it for me? It makes harmony difficult to establish. The initial establishment of ROI requires trust and alliances. The IT group cannot generate interest and funding, it needs to partner with the business to achieve that.
Then the question is how do you sustain this effort? Once you have proven ROI you still need to establish sharing and ownership of the different services. Business Units don't necessarily want to own business services. They certainly don't want to work for another Business Unit! So who pays for changes? Does a business unit "own" a service? What benefit would it be for them to share it?
They are implicit challenges in migrating services into a shared environment, especially if there is data involved, they don't want to share the data either. There is also all the chargeback mechanisms that can prove challenging. Obviously this doesn't even get into the complexity and cost of getting people to agree on the data schema level.
Another hurdle is the establishment and adoption of Governance. This is a very sensitive one, especially to the technical audience because people don't want to be governed. People don't put enough emphasis on generating the consent of the "governed". People are not necessarily sharing their rationale for governance. The principle to follow is to develop an 'informed consent'. They consent to be governed because they have a deep understanding about why a service is designed a certain way. People have a hard time with governance, particularly technical folk. The reality is that automated policy enforcement of things like interoperability and security make the evil parts of IT faster, not slower. You just have to get used to the fact that you're going to have to comply and it will either be automated or it will be very slow and painful manual reviews.
I would also say that reuse is challenge. It takes a long time to learn what someone has done. You may need to request changes. It is not even clear that the changes will be better than writing entirely new code. But people often oversee the cost of new code: bugs, reliability, increased maintenance over duplicated code …. People often forget that a service is in a "steady state", and that has value. It's the difference between "reusing" a blueprint for building your own car and "reusing" your friend's car. The actual car will get you to where you want to go faster. "Just let me deploy my code" is a very seductive idea, but I've found most Enterprise IT unwilling to accept the risks associated with that. So instead we rely on using what's already there which makes a lot less hurdles to jump through.
InfoQ: Is there an ideal mix of people for governance?
Miko: The ideal mix has to be representative of the organization. If you look at a government there are representatives of people being governed. The second dimension is that it has to be someone who they trust to represent their interest. You need to find people that can actively represent their interest and at the same time be capable of communicating why some policies make sense. I think it was Winston Churchill who said that Democracy is a very stupid form of government but it's better than all others. This suggests that your SOA "Center of Excellence" (CoE) should be a representative body--that everyone affected by the policies created should have a voice there.
Within the context of a CoE you can't be intractable, there is always a give and take approach to resolving conflict, you need to be both aggressive and questioning. So if you're the developer representative in the CoE, be sure to question everything, but don't be an ass about it. Ultimately you will have to work with these people.
InfoQ: How does governance works in B2B?
Miko: There are two ways to look at it. In general you have some SOA principles and practices supporting a model based on many consumers and one provider. This is a supply chain type of model, say like a large OEM, or like WalMart or Boeing, …everyone comes to them through these sets of interfaces. What you have typically is bunch of one-to-one relationships. Governance in this context is not that difficult because it's almost all on the provider side. Where things get more sophisticated is business process orchestrations across network of providers and consumers.
From WebMethods perspective we found quite a bit of success on provider B2B gateways. We don't see situations where you have many-providers to many-consumers (like an "ebay" model, an open marketplace). There are few "Vertical Marketplaces" (they were hot during the dot com boom). The general pattern is more like one provider to many consumers.
InfoQ: Where is the level of maturity going? What's next in the enterprise for SOA?
Miko: The survey indicates pretty clearly a notion of awareness. The next step is to move from awareness to actions. We see more clarity around technical and organizational requirements. The next step is blueprinting: implementation and governance. People have a strong understanding, this is necessary, but it is also insufficient.
InfoQ: What's different between SOA governance and IT governance? Why is not just part of IT governance?
Miko: Logically, SOA governance should sit within IT governance. In fact if you study IT governance practices like ITIL, you get a lot of clues about optimizing system behavior. IT governance is strongly focused on connecting the financials with IT, especially on the foundation of cost, there is a very strong orientation to control cost. You also tend to see a provider orientation, this is very much focused on CMDB, tracing assets through the lifecycle. Through the "help desk" function you see a lot of pain, by tracing these effects to their root cause via the CMDB, you can optimize your IT assets. IT governance is a suitable precursor (emphasize things like reliability, TCO,…), and now ITIL is starting to take on some of the SOA aspects. What is unique about SOA Governance is the emphasis on business process and agility. Culturally, what SOA injects is the prime externality: the service consumer, the business unit or the customer. Now you are not just optimizing for the cost center for the service consumer.
By aligning the business function and IT function, SOA governance has the potential to change the culture. You see, the concept of "Change Management" within ITIL and IT Governance is focused on helpdesk and typically has nothing to do with developing or deploying new functionality. In IT Governance, "change" means something bad happened, and you need to trace the "root cause" and try to find who was at fault. In SOA, change can mean new functionality, and agility comes through reuse and composition and configuration.
InfoQ: SOA Governance is evolving into a complex discipline? (registries, repositories, security, policy management, logical data model,..performance monitoring…), is it going to make SOA successful or is it a burden for SOA? Could SOA be so complex?
Miko: This is certainly a fear that a lot of people have. This is where we believe that governance automation has a large role. What we are seeing is that best practices are emerging, when you create uniformity in your SOA, you want interoperability to be the corner stone. If you have a bunch of dependencies between providers and consumers, you want to make sure that there are some SLAs in place. The system can be made more manageable with policies to help manage behavior. These are mandatory for a connected system to operate.
Governance is a complex system: you want to minimize the impact on people. Automation makes dealing with complexity more manageable. There is a lot to keep track and people can't keep track of everything in their head.
To be successful in SOA governance automation, you have to iterate and drive topics security, interoperability, maintainability….
InfoQ: Versioning has been a major hurdle for all the remoting technologies. How do people approach versioning? How tool simplify versioning?
Miko: The first law of the land is that the principles of evolvability are the key principles. In some ways there is never a right answer. You can perfect granularity and reusability and then you discover their a new scenario that it cannot perform. The other trap is to spend years of your life on perfect reusability and granularity and perfect schemas. You'll never ship! So instead you just ship the darned service as best it can fit the business need, and try to imagine all future uses if you can. That's when you need to version. How do you avoid version fragmentation and avoid killing the service as a result?
The key is to establish a binding policy that introduces a level of loose coupling between the consumer and the provider. Use an intermediary: the end point must be made virtual. To some degree you can use techniques to transparently migrate and mitigate versions. The big issue here is that if everyone is tightly coupled to an endpoint, it means first of all that you need to discover who is bound to the service. Without some kind of governance system you won't even know who is consuming. Then if you don't have any intermediary, you have to get all the consumers to recompile their code to reflect any new version. So you basically have to hunt down everyone using the darned thing and then get them to recompile their code to reflect the new version. It doesn't work, be sure to set a binding policy where there are no anonymous consumers (except ones you don't care about, and even those you need to regulate their consumption level to prevent Denial of Service) and such that all consumers are binding a "virtualized" service endpoint. That way the intermediary can redirect them to whatever you want.
Another mechanism is run-time discovery, but we don't see a ton of that. There's agent based discovery like Amberpoint or Actional and then there's network based discovery like Managed Methods.
With the use of taxonomies, and role based access control, you can also hide some of the complexity of versioning. That way different groups feel like they are looking at the same service even if there are in reality several versions. Having several versions may be unavoidable.
They are more mitigation style responses. We can't deny the existence of the problem. Things will have to mutate and evolve.
There are 3 choke points for governance:
For a developer developing a service, they should go through the reusability check-point, validate that a service can be or must reused, do we need a new version? The policy should of course be in this case that if there is a service you can reuse, please do so. Another choke point is the Run-time governance system: if we have a new version, can that version be made transparent to old users through transparent redirection as well as some degree of XSLT transformation or massaging the schema. The third choke point is during discovery where we can make discovery of the new version transparent and less invasive. If there are many versions, we can at least hide the ones that are inappropriate from various consumer groups.
InfoQ: How can you fund SOA governance?
Miko: The first pattern is top-down pattern: an architect and some VP level IT executive agree to the benefits, there is a high level of trust, and this is how you get funding.
The second pattern is a smuggling operation: business units tend not to fund what anyone else needs. On the plus side, business units are great funders because a little SOA goes a long way. A little SOA tied to a BPM project can generate tens of millions of dollars of ROI straight out of the gate. So it's easy to embed some governance or shared infrastructure components in there without too much fuss. This requires some effort to embed governance activities in a project.
You can also establish some kind of shared infrastructure with costing mechanism and put in place some kind of charge back mechanism, but this requires some pretty serious executive commitment.
InfoQ: How do you generate ROI from SOA governance?
Miko: SOA governance and SOA adoption are realization strategies, tied to the benefit of SOA itself, one of the good thing that is emerging is that we are increasingly seeing the benefit of BAM, runtime governance, where you can measure on-going economic benefit of services through their use and reuse. You end up with a pattern where the reusability of the service shows growth and value. It becomes exciting to the business when you see the value of a service increasing, because you see new consumers reusing the same use care, or increasing the number of services.
In short, if you can show an increasing utilization by an increasing number of users using an increasing number of services, you have a "hockey stick" growth curve. All that remains is to show how valuable that usage is and you have ROI.
We are seeing good successes and people are able to justify themselves.
InfoQ: There is a lot of debate in the industry about SOAP/WSDL versus REST. The survey seems to be unequivocal?
Miko: People in the enterprise are pretty expedient on the use of standards, people harvest best practices and interoperability from standards. If you look at those drivers, you can understand why the standards shake out. Some of them have been in the WS-I basic profile. We see a lot of SOAP based activities. Messaging approaches augment what the enterprise is doing in general. If you have web-based approaches, with scalability concerns then REST seems a better fit.
SOAP is the least common denominator that all systems are ready to deal with, so we are seeing heavy adoption of that. Obviously REST is even more lightweight, but you can get into JBOWS (Just a Bunch of Web Services) that way (I don't mean WS-* "web services" I just mean a REST service that you can use on the web).
With respect to SCA, Software AG was one of the early participants in that effort. As yet at the customer level there is not yet a lot of excitement, but from the perspective of having a way to architect systems it's pretty handy. We will be supporting a number of SCA related features in the webMethods and Centrasite product lines going forward, for example the ability to dynamically parse elements in an SCA assembly package and to visualize those dependencies for Governance within Centrasite.
InfoQ: We are in 2008 and "lack of skills' still come first. How can we solve that today? What's the path moving forward?
Miko: Enterprise IT is a business, so far, we have had a long line up around SOA, with people poking around here and there but not enough adoption. After many years, the adoption is finally here. What's clear from the Survey is that SOA is real. The skill shortage is going to resolve itself because real money is coming into it, people will get training and they will also see a career opportunity.