BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News APP vs. Web3S: the Quest for a RESTful Protocol

APP vs. Web3S: the Quest for a RESTful Protocol

Bookmarks
Standardizing on a protocol for resource publishing and editing can bring a lot of benefits since it  increases the chances for achieving interoperability between parties and universal understanding. While XML as a base format creates brings some of those benefits, it's a little too general to be useful without also setting some widely accepted ground rules for describing more specific attributes like collections and entries. Combining such a format with the generic HTTP protocol in a RESTful promises a common ground. Indeed there are several protocols available that aim to solve this problem. The first and probably most mature is the ATOM Publishing Protocol (APP) which is a draft IETF standard close to finalization. APP is defined in the latest draft as follows:

The Atom Publishing Protocol is an application-level protocol for publishing and editing Web resources using HTTP [RFC2616] and XML 1.0 [W3C.REC-xml]. The protocol supports the creation of Web resources and provides facilities for:

  • Collections: Sets of resources, which can be retrieved in whole or in part.
  • Services: Discovery and description of Collections.
  • Editing: Creating, updating and deleting resources.
The Atom Publishing Protocol is different from many contemporary protocols in that the server is given wide latitude in processing requests from clients
The APP also defines a standard extension model to make it more usable and indeed Google, for instance, built on that when it developed the Google Data API (GData) as a common protocol for reading and writing to many of their services like Google Base, Google Calendar etc.

However it seems Microsoft doesn't like APP and decided to release their own protocol which called Web3S ("Web Structured, Schema’d & Searchable"). Microsoft's Yaron Goland recently introduced it in a blog entry :

Most of the services in Live land follow a very similar design pattern, what I originally called S3C which stood for Structured data, with some kind of Schema (in the general sense, I don't mean XML Schema), with some kind of Search and usually manipulated with operations that look rather CRUD like. So it seemed fairly natural to figure out how to unify access to those services with a single protocol. The first place we went was APP. This was a business decision. Live is in the services business, not the protocol or development tool business. I am not paid to give a damn what browser someone talks to Live with, what language our partners/customers develop their software in, what operating system they run, etc. The only thing I am paid to care about is that we get as many people as possible writing as much software as possible that interacts with Live services.

So for us the whole protocol issue is just a barrier to entry. We don't make any money from that barrier. And no, we don't care about lock in. At least the people who give me orders (hey George!) have a good understanding that the days of lock in are long over. The future clearly belongs to connecting services together. That is, nobody is going to have all of their data, services, etc. at Live. It won't happen. We could be the absolute best at everything we do and we still won't own all of a user's data and services. So for us to succeed we have to convince users to keep some of their data/services with us and then make it brain dead easy to connect the data/services they keep with us to all the other data/services they keep in lots of other places.

In other words, it's all about interoperability and the easier it is to interoperate the more successful we will be. So in our dream world there would exist a protocol that would meet the S3C pattern. A popular protocol. A widely support supported. A protocol we could just adopt and get on to the business that can make us money - building cool services.

So with this in mind we first went to APP. It's the hottest thing around. Yahoo, Google, etc. everyone loves it. And as Dare pointed out in his last article Microsoft has adopted it and will continue to adopt it where it makes sense. There was only one problem - we couldn't make APP work in any sane way for our scenarios. In fact, after looking around for a bit, we couldn't find any protocol that really did what we needed.

Because my boss hated the name S3C we renamed the spec Web3S and that's the name we published it under. The very first section of the spec explains our requirements. I also published a FAQ that explains the design rationale for Web3S. And sure enough, the very first question, 2.1, explains why we didn't use ATOM.

Microsoft's Dare Obasanjo explained some of the problems Microsoft sees with APP which (probably) led to the development of an independent protocol in a post called "Why GData/APP Fails as a General Purpose Editing Protocol for the Web". In the post Dare explains in  length that the APP suffers from several limitations because of the constraints in the protocol, specifically he talks about
Mismatch with data models that aren't microcontent: The Atom data model fits very well for representing authored content or microcontent on the Web such as blog posts, lists of links, podcasts, online photo albums and calendar events. In each of these cases the requirement that each Atom entry has an an id, a title, an updated date, one or more authors and textual content can be met and actually makes a lot of sense. On the other hand, there are other kinds online data that don't really fit this model.
and
Lack of support for granular updates to fields of an item: As mentioned in the previous section editing an entry requires replacing the old entry with a new one. The expected client interaction with the server is described in section 5.4 of the current APP draft and is excerpted below.
and lastly
Poor support for hierarchy: The Atom data model is that it doesn't directly support nesting or hierarchies. You can have a collection of media resources or entry resources but the entry resources cannot themselves contain entry resources. This means if you want to represent an item that has children they must be referenced via a link instead of included inline. This makes sense when you consider the blog syndication and blog editing background of Atom since it isn't a good idea to include all comments to a post directly children of an item in the feed or when editing the post. On the other hand, when you have a direct parent<->child hierarchical relationship, where the child is an addressable resource in its own right, it is cumbersome for clients to always have to make two or more calls to get all the data they need.
Bill de hÓra who is involved with IETF APP working group reacted to Dare's claims and said
...All that said, I'm very happy to see real implementors provide some pushback on the Atom Protocol for their needs. However going on to claim GData/APP has failed is random enough conclusion, especially for the problems mentioned, which in one case ,is a deliberate design exclusion (for now). If these are the most serious problems encountered inside MSFT, it strikes me that APP's overall design is in good shape. Given the level of thought and discussion he indicates seems to have gone on inside MSFT, I'm surprised Dare didn't mention these two issues, which strike me as much more substantial: 1. Update resumption: some clients need the ability to be able to upload data in segments. Aside from a poor user experience and general bandwidth costs, this is important for certain billing models; otherwise consumers have to pay on every failed attempt to upload a fote. APP doesn't state support for this at all; it might be doable using HTTP more generally, but to get decent client support you'd want it documented in an RFC at least. 2. Batch and multi-part uploads: This feature was considered and let go by the atom-syntax working group. The reason was that processing around batching (aka "boxcarring") can get surprisingly complicated. That is, it's deceptively simple to just say "send a bunch of entries". Still, it would be good to look at this at some point in th future.
Joe Gregorio (another major APP contributor) asked, in a comment to Yaron's post the obvious question:
What I am more interested in hearing about is why you didn’t approach the WG with these issues when you ran into them. The spec was obviously still in the works when you were working on WebS3 and if you believed you had found real weaknesses with APP, which the ensuing discussion has shown that you didn’t, then why not bring attention to them before we shipped?
IBM's Sam Ruby, co-author of RESTful Web Services, believes the protocol intended to support "Web Structured, Schema’d & Searchable" support neither the Web, Schema, or Search:
There are two new media types (Application/Web3S+xml and Application/Web3SDelta+xml), two new URI Protocols (Web3S and Web3SBase), and one new HTTP method (UPDATE) defined in this document. I can find no discussion of binary data, in fact everything seems defined in terms of the XML infoset. Given that all data needs to be in a namespace, and that all such namespaces need to use a new URI protocol, one can conclude that no existing XML documents can be directly handled by Web3S. Web3S data is further constrained to be a self enclosed tree. There is no general concept of a hyperlink in Web3S, neither to external data, nor within a tree. To traverse this data, one needs to be aware of the specific schema employed by the application.
XML co-inventor Tim Bray criticizes the process that led to Web3S:
With all due respect to Yaron, it is really unfortunate that the first time the world heard of Web3S it was in a post of Dare’s aimed at explaining APP’s failings, which unfortunately revealed that he hadn’t read the spec and didn’t understand it. The treatment he got, especially from me, was pretty harsh; but then the post was pretty clueless, and in my experience Dare normally isn’t, so Occam’s razor naturally generated suspicion of what lay behind the content.
And guys, if you’re going to work at Microsoft, you’ll just have to accept that you’re in a world where you’re routinely suspected of the worst. It’s too late to go back and change the history that led here; they teach distrust-of-Microsoft as an MBA-level business basic these days. Learn to deal with it or find a new job.
It might have been nice if the Live team had, you know, actually tried interacting with the other group which was noisily and in public designing a general-purpose REST-based protocol, but that’s water under the bridge.
David Ing made several interesting observations on the Microsoft effort and summed the whole issue with
So, interesting times still. Not sure what the Church of APP will make of it, but I imagine there will be calls to see how these needs can be address directly in APP rather than outside of it. It seems unlikely that the world needs another simple protocol based around such few verbs, but then again I expect Microsoft has just drawn a straight line of what they need and the quickest way to get there for all their existing resources. This may turn out to be a branch line on the big web railroad, but it's not without a use for a while.
Indeed, interesting times still.

Rate this Article

Adoption
Style

BT