Interview: Pete Lacey Criticizes Web Services

In this interview, Pete Lacey, who recently became well-known in the SOA community because of a series of blog posts starting with a very funny one entitled S stands for Simple (which was covered at InfoQ as well), talks to InfoQ's Stefan Tilkov about the problems he sees with Web services, including basic technologies such as SOAP, WSDL and UDDI, as well as advanced standards from the WS-* family.

InfoQ: Your blog post "S stands for Simple" has been linked to by literally every blogger in the SOA/WS community. Can you give us a little background on yourself? How much of what one can read in that post is based on personal opinion, in other words: have you been "SOAP guy" in the past?

Pete Lacey (PL): Yes, I have been SOAP Guy in the past, and am in fact SOA (note, no ‘P’) Guy now. I was introduced to web services late, when I started working for Systinet back in August 2002. I started there as an SE, but given its startup nature, the job was really all things technical and outward facing; that is, sales, marketing, and consulting. In the beginning the work was pretty low-level, down in the dirt with SOAP, WSDL, and UDDI. In that sense, I was SOAP Guy. Interestingly, my customers never played the role of the disenchanted developer, they never pushed back on the complexity of SOAP/WSDL or the abuse of HTTP or anything that tends to rile certain people’s feathers. Though they did gnash their teeth at the interop issues.

Today I work as a consultant for Burton Group in the Application Platform Strategies service. Though SOA is not all we do, it is one of the things we’re famous for, so I spend a lot of my time explaining to people what SOA is and how they might go about managing a SOA initiative. It may sound hypocritical, since I’m not a SOA proponent, to turn around and tell my customers how to go about it. But by the time I’m involved the SOA decision has often been made, and Burton Group does have a more sober perspective on the technology than most vendors do.

InfoQ: Can you briefly explain your position about SOAP and the WS-* specifications?

PL: Taking the SOAP 1.1 specification in isolation, my position on it is that it went too far. Had SOAP simply defined an envelope for XML message passing it would have been a small but interesting step forward. But the SOAP spec also defines an—admittedly optional—serialization mechanism; goes out of its way to be transport neutral, but then defines an HTTP binding that ignores the basic tenets of HTTP; and goes on to define a practice for using SOAP as an RPC mechanism. However, if one ignores the optional bits, SOAP itself isn’t that bad. The envelope design pattern can be useful—HTML uses it after all.

In contrast, I have nothing good to say about the WSDL 1.1 specification (I’m sticking with the 1.1 specs as those are still the WS-I recommended versions). It is overly complex, often ambiguous, and occasionally inconsistent. In practice, tool-generated WSDL documents are nightmarish to read and the source of half of all interoperability issues (and I’m not referring to any XML Schema components). It’s also my position that WSDL is being used as a crutch by web service vendors and developers, even though the functionality it provides should be wholly unnecessary.

InfoQ: Can you elaborate? Isn’t the “contract” described in WSDL the most important aspect of a service? Why do you deem this unnecessary?

PL: Well, I would say that the most important aspect of a service is the service itself. Also, today’s tools encourage—and some mandate—developing the service first and then generating the WSDL, and thus the WSDL isn’t a contract at all. But I understand the question, so here goes.

SOAP enthusiasts are aghast at the claim that WSDL is unnecessary, but it’s true. They claim that the server isn’t returning HTML for humans to read and interpret, it’s returning XML for machine to machine processing, and how can this work if the client machine doesn’t know what the server is sending. But I claim the client doesn’t “know” anything. It doesn’t know that the message is an invoice, only people know what an invoice is. It doesn’t know that the FirstName element isn’t likely to have the value “Tea Kettle” (barring very strange parents). All we can say it “knows” is the structure of the document and what datatypes the elements and attributes can hold.

But, WSDL proponents go on to say, what about code generation and data serialization? A crutch, I say, and not a very beneficial one. I’ve never cared for wizard driven development and code generators. They tend to demo well, but ultimately don’t produce maintainable code. Furthermore, this style of SOAP development reinforces the SOAP as RPC/distributed-component/serialized-object perspective and relegates XML to just another serialization framework. The generated code tends to be especially brittle, and breaks when the message returned changes in any way. For instance, if your client was happily consuming a message that looks like this:

<Movie>
   <Title>Star Wars</Title>
   <ReleaseDate> 1977</ReleaseDate>
 </Movie> 

Why should it choke if the message changes to this:

<Movie>
   <Title>Star Wars</Title>
   <ReleaseDate> 1977</ReleaseDate>
   <Runtime>121</Runtime>
 </Movie> 

For similar reasons, the fact that SOAP messages are strongly typed also introduces a great deal of tight coupling between endpoints and inhibits the independent evolution of services and clients. Serialization also exposes the impedance mismatch between XSD and your programming language of choice (and what about dynamically typed languages?) and is the source of the other half of all interoperability problems. Finally, serialization techniques preclude the use of powerful, XML-centric tools like XPath and XSLT. You would be much better off processing a message manually and ignoring any notion of typed data.

Proponents also say that WSDL and XML Schema are useful for message validation, but if an XSD validator is even in the message path, no one turns it on, because it’s too computationally expensive, not completely implemented, and unable to perform all the validation required (e.g., date X should be no more than 30 days divergent from date Y).

All of this is not to say that a rigorous, machine-readable description of an XML message isn’t useful at development time and even runtime, it is. But the notion of assigning type to elements and attributes is unnecessary. After all, how many HTML forms have been processed successfully, and those are submitted as just name/value pairs. And even when a message description is available, using it to generate code that treats remote resources as local objects and messages as their serialization is counterproductive. XML is for representing structured information not serializing objects.

Ultimately, WSDL proponents will say, we simply need it; CORBA has IDL, RMI has Java interfaces, surely we need something. Well, no you don’t. Think about coding to a local API in a language like C/C++ that doesn’t allow for introspection. How do you make that work? You read the documentation! And this is true for ordinary Java development and web service development too. Real live human programmers are involved; reading the documentation, crafting messages as specified, and interpreting them as needed. It’s not like a programmer uses the first method that her IDE suggests, and simply drops properly typed nonsense into the argument list. There’s no doubt that having a machine readable description of a service can make your IDE do interesting tricks, but that’s a far cry from absolutely essential.

A final argument for not needing WSDL requires that we abandon the SOAP model of development in favor of REST styled development. We’ve already dispensed with the desire for message description, with REST we can also dispense with the portType, because all of the operations are well known. Similarly we don’t need the binding section anymore; the transport is always HTTP and messages are described by the Content-Type header. The service element is unneeded too, because the URL of a resource should be relatively static and well known, and if it changes you’ll get a redirect.

To dwell a bit more on XML Schema, I have to say that it is a deeply flawed specification. Not only is it notoriously complex and inconsistently implemented, it is fundamentally incapable of representing textual XML documents, as opposed to XML documents representing typed data. And even for representing data it leaves much to be desired. Burton Group is soon to publish a best practices document I wrote for creating interoperable, data-oriented schemas, but I can summarize it here: don’t use anonymous types, don’t use element groups, don’t use attribute groups, don’t use redefine, don’t use “any” elements, don’t use anyAttribute, don’t use anyType, don’t use lists, don’t use unions, don’t use substitution groups, and so on. So, not only is the ability to create typed instance documents of dubious value, XML Schema isn’t particularly good at doing it. If you want to use a schema language, use RelaxNG or Schematron instead.

InfoQ: What do you think of the XML Schema Patterns for Databinding spec? Does this address the same concerns as your report?

PL: Not really. I think the Patterns for Databinding is a noble effort and it takes an interesting approach. And down the road it may even help vendors converge on a usable subset of XML Schema. My document, though, is aimed at the user of today’s XSD-aware tooling. It goes much further than warning you away from the poorly supported aspects of XSD, but talks about namespace usage, versioning, RDDL, and much more. But even if the Patterns for Databinding effort ultimately results in XSD documents being 100% interoperable, you must remember that the problem—for web services, anyway—isn’t the inconsistent implementations of XML Schema, but the use of XML Schema at all.

InfoQ: What’s your opinion on UDDI, the third of the core web service specs?

PL: I find it grossly over-engineered, what with tModels and bindingTemplates and categoryBags. Succinctly, anything that requires a design pattern to achieve the predominate use case for which it’s deployed has failed. By that I’m referring to the best practice surrounding mapping a WSDL into UDDI. It’s the single most obvious task to which UDDI can be put, but UDDI can’t do it natively or easily. Even the relative success that Systinet (now HP) is enjoying with its registry is largely due to hiding as much of UDDI as possible.

InfoQ: And the rest of the WS stack: WS-Addressing, WS-Security, etc.?

PL: I won’t address all of the WS-* specs individually, and individually they are not all that bad. Collectively, though, wow! I mean, WOW!

The complexity of the web service framework (WSF) is mind numbing. The odds that even a small portion of it can be made to work in an interoperable fashion across multiple vendors is very slim. (Two very large software vendors recently told me that while they will eventually interoperate with the forthcoming Microsoft implementation of a number of the WSF specs, it will not be by coding to the standard, but to the Microsoft-specific implementation.) And the claim that your tooling will shield you from the underlying complexity is laughable; the complexity will leak through. This is not to mention the (now dwindling) vendor wars, the ungodly amount of time its taking to move these specifications forward, the dearth of implementations, and the “SOA fatigue” experienced by enterprises as they try and make a go of it.

To try and put a positive spin on the WSF, one can look on it as the XML-ification of a number of existing technologies. For instance:

Now, of course, that mapping is not exact, but it’s interesting. If, in the past, someone had said, “I’m tired of having proprietary implementations of message-oriented middleware.” Or if they had said, “you know, using XML instead of opaque, binary protocols opens up some interesting possibilities,” then we would, quite possibly, have been on to something. I don’t think it would look like WS-Anything, since the WSF composes different messaging styles in the SOAP Header, rather than in a layer underneath the message. Nor would a single system try to be all things to all people. In this model, XML-based protocols would be as distinct as their current binary counterparts. Mind you, I’m not advocating such systems, only that I find the idea interesting. But, as Mark Pilgrim said, “Lots of luck with that.”

It's important to understand that my imaginary infrastructure and the current web service framework, suffer from the same problem: they can’t scale. And by that I don’t mean transactional scaling, e.g., how many messages can it process, but rather, to use Dan Pritchett’s terms, I mean operational scalability, deployability, developer productivity, and time to market. So, while web services and the components of the WSF can and have been made to work, these successes (in my experience) are narrowly scoped deployments with tightly bound endpoints that took an inordinate amount of time to develop while promising little in the way of interoperability and raising the barrier to entry way too high.

InfoQ: If I understand you correctly, you’re referring to scaling to Internet dimensions. But not everybody needs that, do they?

PL: It’s true that the WSF prohibits scaling to this level, but, no, that’s not what I’m talking about. Again, when I say “scaling” I’m referring to things like developer productivity; system management, maintenance, and evolvability; access to data and resources, etc. As I see it the WSF is so large and complex, and the participants so tightly coupled, that scaling to even enterprise levels is out of the question. I firmly believe that one can be every bit as successful today with web services as they have been in the past with CORBA. That is, you will be able to build working systems with some distributed components (although not without undue hardship). But you will not be able to use this technology to build a fully distributed enterprise architecture.

InfoQ: Do you think the philosophies of the different "camps" - SOAP/WS-* folks on one side, REST folks on the other - can be aligned somehow?

PL: No, but that’s okay. From a technology point of view SOAP and REST couldn’t be farther away from each other. In fact, the only relationship I can see between the two is that REST is embodied in HTTP and SOAP frequently tunnels over it. I might add that SOAP supports XML message passing (in principle) and REST can pass XML messages too, but as someone said recently (I can’t remember who), message passing is intrinsically uninteresting.

But if the question was “can the SOA and—what I guess we’re now calling—ROA camps be aligned?” that would be a more interesting question. The answer is still no, but it’s a more nuanced no.

The SOA camp claims that one can achieve enterprise (and some claim global) scalability using the WSF. I think this fails for two reasons: one is simply the complexity of the framework as already discussed. It has proven hard enough to get SOAP and WSDL based services to work and interoperate. It’s only going to get harder as the other pieces of the WSF come into play. Introduce any number of intermediaries into this mix, and you’re building on a house of cards. Also, no one gets to play unless they are a proficient, web service aware developer.

InfoQ: Tool vendors would disagree and point to their IDEs, which are claimed to make Web service development ridiculously easy.

PL: I think we’ve already covered how effective this development tooling actually is, but even accepting it for what it is, you’re still limited to just developers. Given that the value of web services is in the consumption, why aren’t they consumable by anyone? Why aren’t services made available to the sales guy with Excel, or the proto-geek in finance with a report writer, or the manager with a news aggregator, or the simply curious with a browser? Because SOAP, WSDL, and the WSF are way beyond any but the technically elite; the barrier to entry is too high. In contrast, anyone can retrieve the contents of a URL.

InfoQ: Let’s get back to the second reason for the alignment problem.

PL: The second reason is that web services are too tightly coupled. Each participant needs to know a great deal about each of the others: the service and operation names, the messages they exchange, the datatypes of the message elements, the security context, the messaging semantics, etc. Thusly are non-scalable (and again I don’t mean transactional scalability) systems designed. One could argue that WSDL and the WS-Policy family of specifications addresses all this, but that’s not what I mean. Whether the tool knows or the programmer knows, the point is that one side of the conversation possesses intimate knowledge of the other, and should either side change, everything breaks.

This is not to dismiss SOA altogether, just to take it down a peg. I don’t believe it can scale and it’s not a silver bullet, but, as I’ve said, it can be made to work if you introduce enough constraints. And such a system is likely to offer functionality that exceeds what can be had using traditional technologies. For instance, if reliable messaging is an absolute requirement of your design, and there are multiple actors, and you’re only exchanging XML messages, then using SOAP+WSA+WSRM, instead of, say, TIBCO Rendezvous, makes sense (once WSRM is finalized and enough interoperable implementations exist—see what I mean), as it allows for the introduction of generic intermediaries that can process and manipulate the messages in transit. Also, much of what is SOA is not technology focused. SOA proponents have a lot to say about process and governance, and much of that is transferable to any team building distributed architectures.

Furthermore, REST/ROA is not all encompassing (and to the best of my knowledge, no one claims it is). There are a number of legitimate enterprise use cases where REST styled systems don’t play, but these are fewer than one would imagine. It all boils down to choosing the best tool for the job.

InfoQ: What would be your advice to someone looking to implement an SOA today?

Read this.