We're at QCon and talking to Anne Thomas Manes. So Anne, can you tell us a little bit about yourself?

Thank you for having me. I'm Anne Thomas Manes. I'm a Research Director with Burton Group. My primary area of focus is service oriented architecture, but my group covers all things related to building and managing application systems.

You are quite well known in this community and you have been covering it for a long time. What is your current view on the state of SOA?

I've been talking about SOA since 1990 when we first started looking at CORBA and distributed object systems. It's been around for a long time, although I think that the technology has improved and our understanding of how to build service oriented systems has improved a lot since those days. Where people are today? Well they're either just doing experimentation and they are really focusing mostly on the technology. Or they've gotten beyond that stage and they realize that SOA has nothing to do with technology and has everything to do with how you build systems. And they are at that point trying to figure out how to reign things in and get them under control. Or they have actually got to the point where they realize that governance is the main piece of making SOA work and there are not very many people at that point yet.

What is the difference between the promises that are being made and the actual reality? Are the promises true? Do you have experiences that actually people managed to get it all together correctly?

I don't think anybody has actually reached the point where they are really getting the promise of SOA because I think SOA is a very long term effort and to truly get to the point where you've got complete flexibility and agility that's going to be 15 to 20 years from now. But people are definitely experiencing benefits from creating services and designing systems according to SOA principles. I think that every single application that you implement that is using SOA principles is going to make your systems more manageable. And so you should start to see real benefits for every single project that you do.

The topic of your talk here at QCon is how to sell SOA and how to get funding for that. Can you elaborate a little, how do I get funding for something that will take me 15-20 years?

SOA is about reengineering the IT organization and changing the way you design systems and the way you manage your systems. Today about 80% of an organization's IT budget is spent on legacy systems and only 20% is spent on new development and innovation. And that 80% of the IT budget typically accounts for something like 70% of a corporation's capital expenditures per year. And what's really amazing is that most organizations never bother to look at this enormous allocation of funding. But that's actually where you want to apply your SOA principles, to that existing system. And if you think about how much money gets sunk into legacy systems you realize there are a lot of opportunities for us to fix IT and that's what SOA is about, it's about fixing IT. IT is fundamentally broken and you need to fix it and the only way to do that is by going in and reassessing and reengineering the IT process. And SOA is a completely different way to design systems that is no longer looking at it from an application project by project basis but looking at it in terms of very expensive assets that need to be managed more effectively.

That all sound reasonable but that requires the organization to actually recognize this, to perceive this as the problem and what if they don't?

That's actually the big issue, that's why it's so hard. There are great methodologies out there for helping you examine your current system application portfolio management and application life cycle management are the core technologies and methodologies that people are using today to assess their current system and to understand how much money they are spending and whether or not it's a wise investment. And it's actually a very academic exercise, you just sit down and do an analysis of each application and you have to come out with a standard set of metrics to do this but you have a standard way to assess the value that the application is providing to the company and how much it costs on an annual basis to manage and maintain that application. And if cost exceeds value you got an academic opportunity here to say "it's time for us to decommission the application". And if you look at most IT organizations they have hundreds and hundreds of applications. The average large enterprise has 4 or 5 hundred applications and many of them have as many as a thousand or more than a thousand applications. And most of those applications are doing exactly the same thing but they are doing it for different groups within the organization. SOA is about reducing redundancy in your system, it's like "if I have a dozen applications that are all doing the same thing, I should decommission those applications and create a common service and have just a single service that implements that capability".

You're argueing that starting from application life cycle management to portfolio management that's the right bridge towards a SOA approach?

You need to have an enterprise architecture effort within the organization, the organization has to recognize that there is a problem in IT, and that you do want to do something to fix the issue, and it's really an academic exercise, what's the cost what's the value. When cost exceeds value it's time to get rid of this investment. If you treat these application systems like real assets, then it's very easy. In fact business people understand this idea: cost exceeds value, get rid of it. If this were a financial portfolio you can guarantee these people would be scrutinizing it a lot more. Now the challenge is if you can't convince management to do this high level assessment of your environment, in that case my recommendation is that you adopt a more stealthy approach to SOA – "Stealth SOA". You have a project, it's funded by a business group, the business group recognizes that this is a project that we need to do, and now take a look at this in terms of how can we apply SOA principles to this system. And you simply apply SOA principles to every project that you do. It takes a lot longer to achieve the true benefits of SOA, if you're doing it in a stealth manner but once you've got a couple of examples in place it's much easier to demonstrate the value of SOA then you get more funding for the process.

That naturally leads us towards the issue of governance. If you start those stealth projects, and have everybody do a ‘bottom up' approach, how do you actually manage that and how do you ensure that it conforms on standards, can you elaborate a little on your view on governance?

Yes, the governance is b far the most important aspect of SOA and you want to make sure that people are actually designing systems in such a way that they can be reused, you want to make sure that they are supporting true interoperability. The problem is we haven't identified what the best practices are for SOA so therefore there isn't a single book you can go to and say "Ah, here's how I build a good service". And so it's very much an art form at this point. Governance tools help you because it gives you the ability to identify the right way to do it and define some compliance tests, make sure that when people build a WSDL it complies to the WS-I Basic Profile, you can define some basic requirements for how to build schemas and things like that. When you put something into production you want to make sure that it's managed, secured, you want to have standard ways to represent this information and governance tools can help you in that regard. But if you can't get the budget to go get the governance tools, that means you have to do it on the side.

What sort of tools would you classify as governance tools? Could you give an example or a product a category?

The primary governance technologies are: registries, repositories and web services management systems. There are also a number of other smaller products that do policy management, and that's one of my favorite in that category. Policy management allows you to define and codify what's the right way to do something, and typically they provide compliance tests that help you verify that your artifacts conform to the policy that you identified.

You mentioned schemas and WSDLs and other web services artifacts, how much do you equate SOA with web services, is it just one way, is it your preferred way?

I would say at this point it's probably my preferred way but it's certainly not the only way to do it and in fact SOA is not about technology, it doesn't matter what technology you use to implement services, but the type of technologies that you use is going to impact how useful the service will be. One of the primary goals you have in creating the service is enabling interoperability at least in most circumstances. There's some times when performance far outweighs requirements for interoperability, or scalability far outweighs requirements for interoperability, and in that case you have to use an appropriate technology to support those requirements. The essence of SOA is the fact that you are designing a service in such a way that it's implementing a core capability that other applications can consume. And that you expose its capabilities using some type of protocol and an interface and a description that other systems can figure out how to work with it. Web Services is a popular way of doing it. The primary reason why Web Services is popular as it is because almost every platform in the world supports it. But it certainly isn't the best way to do things. My guess is that 10 years from now people are going to look back and say "oh my God I can't believe we were using SOAP, it's like the worst thing in the world". Because I'm sure there's going to be something better that comes along. A lot of people are now pushing back at the idea, that WS-* is getting way to hard and to complex.

One of the ongoing debates here at InfoQ is about REST vs SOAP. Do you have an opinion on REST?

REST shouldn't be compared with SOAP because REST is an architecture, in the way that SOA is an architecture. And REST is actually a more constraint architecture, you can build services that are RESTful, you can also build RESTful resources that are not necessarily services, but essentially when you're doing something with REST, you are creating a resource and it has a uniform interface. And you're using HTTP as your interface to it, it supports a very simple method invocation, it has got: GET, PUT, POST, DELETE, that's it and none of these special methods that you define on all your different services. The advantages of REST - and that was proven by the web, REST is the architecture of the Web - is that it's extremely scalable, it supports caching, because you know what a GET will do, and the GET will not make any changes to the backend system, you know what a POST, DELETE will do, and because you know exactly what that thing is going to do it allows you to build extremely high scalable systems like the Web. For a lot of people it's a simpler way of doing it. But at the same time, because you don't actually define a lot of semantics into the interface that means that the semantics are actually left to the application and you have to negotiate the semantics of the interaction out of band, and the question is "which is more valuable to you?". There are certain applications where REST is absolutely the only way to go.

For example?

Extremely highly scalable systems. It's absolutely the only way to go. If I'm dealing with a much smaller scalable system I might find that it's simpler and easier if I have a more direct type of programming interface available that actually it tells me that I'm submitting an order as opposed to just PUT, or POST, I want to be able to say "submit order" because that gives me a little more semantics in the environment.

When you say that REST is the architecture of the Web, is that actually true? Do all systems on the Web follow REST principles?

Well they should, if you want your Web application to support the scalability features that the Web is supposed to support. Then you should design your systems such that they are RESTful. And that means that every resource defined by a URI and that it exposes this common uniform interface, and that when you do a GET it will not actually make a change in the backend and that you have the ability to cache the resource, the representation of the resource. As long as you are following those principles, you get that kind of scalability. But there are many systems out there that are not RESTful, they are on the Web, but what they are doing is tunneling R PCs through POST or something like that. That's a really common thing, that's a lot of the dynamic web out there, a place where you post something, and you send in some information, it's going to go off and call through some web application primary call something to create dynamic stuff. And a lot of those systems are actually not RESTful. They are in fact just tunneling RPCs through the HTTP protocol. There are also a lot of applications out there which claim to be REST but they are really not, they are POX, they are plain XML over HTTP, but they are not actually RESTul in nature.

What do you think about the support for REST that is popping up in different frameworks and different technologies?

That hurts me so much because they are not REST they are POX. For example Axis2, which is one of the more popular SOAP engines out there now has REST support. It will take your Java method and expose it as a RESTful method which is in fact simply just pumping a method call through HTTP so it's not REST it's POX and that's actually what most of these tools are doing.

Many people claim that what's missing in REST is a formal way of describing a contract, a formal way to describe an interface. What do you think about that?

I think that if you want to develop a RESTful application then you want people to be able to consume it, you will need to tell them what are the expected inputs and outputs in this RESTful service. And in that case, you need to describe it in some way. The RESTful proponents out there keep talking about the fact that REST is self describing because it returns a MIME type, but I don't think MIME types are really going to tell me what information I should be sending and what information I'm sending out. I think the way most people build applications; they are not really designed to be completely arbitrary in terms of what goes in and what goes out. That's the beauty of REST that I can tell you when I send something in "here's what I'm sending you" and then you can tell me when you are sending it back out "here's what I'm sending you". But applications don't work very well that way, they like to know what data they should be sending in and out. And in that case XML Schema is perfect type of mechanism to represent "here's the message format that's expected and here's the message that I'm going to send back out". I don't see a reason why you can't provide that kind of interface description to go along with the REST environment. You could define a WSDL document that explains a RESTful interface, although the RESTafarians will say WSDL is not something that you want. I personally believe in a more middle of the road approach. If you want your application to be able to dynamically convert XML into your favorite programming language, like Java or .Net then you want to have a standard definition of what the messages look like so that you can have a tool that will automatically create the appropriate bindings for you.

If we go back to the governance side of things, you mentioned that much of the potential value of introducing SOA is looking at the existing systems instead of the new ones that are being created. Doesn't that mean that a governance solution has to be able to support the governance of old assets as well as new assets? Can I assume that everything is using Web Services or REST? Obviously I cannot, but how do existing tools address this?

That's actually one of the biggest challenges I think that most people have with the tooling out there because most of the tooling is really focused on new development although a lot of the EBSs have tools to take existing applications and expose them as services. My approach to SOA is "don't focus on what the tools can do for you, focus on what it is to trying to accomplish". And once you figure out what you are trying to accomplish look at the available tools that you have at your disposal and figure out which one of those tools is actually going to help you do what you need to do. In an application rationalization process during that you go through and look at each application determine its value and its cost, the most popular tool for that process is a spreadsheet. It's not like you've got a whole bunch of automated tools helping you do this, this is human analysis. And when you are starting to examine your existing applications and figuring out where you want to refactor certain functionalities, that currently exist in a bunch of applications and turn it into a service, at that point you need to do modeling, you need to understand "what is this core capability that I am going to pull out, what's the data that needs to go in and out, how should I actually implement this functionality in such a way that it will support the various application constituencies that need to use it". There's not a lot of tooling out there that necessarily helps you do this, it's very much a roll-your-own kind of thing. But at the same time you have lots of data descriptions that already exist in databases, in applications, in schema documentation. And you suck that information into your repository so now you at least have this information.

One of the things that I find most distressing when I look at people when they are talking about their SOA projects they are always talking about sharing services but they are not really talking about sharing the core artifacts which is a type. The fundamental artifacts that needs to be shared in order to enable true SOA, is a type. It drives me crazy when I see people on the Axis list talking about the fact that they use a code-first approach. I can understand the reasoning behind it, because you can actually automate the development process a lot more if you use a code-first approach, but what you end up doing is creating tons of additional types. So I have the xyz:customer and the abc:customer; are they anything at all the same? Is there a way for me to consolidate this information? No, because you are creating all these new namespaces with all these additional types, you really need to focus on what are the core types, and make sure that your applications are actually sharing the core types.

Any parting words for us? What can we expect in the future of SOA within the next few years?

Lots of hard work, we're talking about 15-20 years worth of work to go through and refactor current systems, I suspect that there are lots of organizations which will never refactor all their systems, it doesn't make sense to do that, you need to have a solid business case to do it, focus on sharing of core artifacts not just the services themselves, don't start your SOA project by saying "ok, we're going to adopt a composite application development environment" because that's not going to help you create these core components that you actually want to share, recognize that SOA is about reducing redundancy and keep that in mind as you go through the process.