Describing RESTful Applications

A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace.

A statement such as this can be hard to grapple with. If servers control their own namespace without a fixed resource hierarchy, how do clients, and more importantly client developers, get to learn or discover URIs of resources? After all, for a long time, one of the fundamental assumptions of distributed client server development is that, in order to build, maintain and administer such an application, we need a formal description of the interface to the application a priori. Roy Fielding's statement seems to contradict this assumption.

Discussions about descriptions of RESTful systems is not new. Almost always, such discussions conclude with ideas like the above. See, e.g, the note on Debate: Does REST Need a Description Language? at infoQ last year that summarizes some of the discussions that took place then. The state of affairs is not very different today.

Despite a number of opinions expressed both for and against a formal description language for RESTful applications, description languages like WADL have gained some limited ground. However, since there is not a "standard" language that a machine could interpret, the most common route that server applications take is to document all URIs, HTTP methods supported, and structures of representations (e.g. as XML and JSON) so that client application developers can rely on such documentation to write code.

This approach, however, contradicts some of the fundamental principles of REST, as paraphrased by Roy Fielding above. Even if we ignore this objection, for those attempting to build distributed applications over HTTP RESTfully, the fundamental question remains. How can a server get away without formally defining a contract? Without a contract, how can we make sure that clients and servers are implemented correctly, not only to respective design specifications, but to other business/technical policies in place?

Distributed applications using HTTP as an application protocol, and built RESTfully, do have a contract, but of a different nature and kind. We need to know what to look for and where. Along the same lines, if we were to come up with a description language, it can not be similar to WSDL or WADL. My goal in this article is to answer the following questions.

• Why is not there a standard descriptions language for RESTful applications yet?
• How does a contract for RESTful applications looks like?
• What kind of software do we need to build that can understand and take advantage of such a contract?
• If we decide to come up with a machine-readable description, how might it look like?

Let me begin with an example.

Example

The task is to write a client application that can transfer some amount from one bank account to another account of the same customer in a different bank.

Let me first describe all the interactions between the client and the server, and then look at a possible description of the contract.

Step 0: A user logs into the client. In order to keep this discussion focused, let me ignore all security aspects.

Step 1: The client uses a URI http://bank.org/accounts?findby=7t676323a to search for two accounts using the customer ID. Here 7t676323a is the ID of a customer who has several accounts registered with the bank. In response, the server returns IDs of two accounts, viz., AZA12093 and ADK31242, respective customer IDs and current balances, as below.

200 OK
Content-Type: application/xml;charset=UTF-8

<accounts xmlns="urn:org:bank:accounts">
    <account>
        <id>AZA12093</id>
        <customer-id>7t676323a</customer-id>
        <balance currency="USD">993.95</balance>
    </account>
    <account>
        <id>ADK31242</id>
        <customer-id>7t676323a</customer-id>
        <balance currency="USD">534.62</balance>
    </account>
</accounts>

Let me assume that an XML schema bound to an urn:org:bank:accounts namespace describes the XML documents used in this example.

Step 2: Since the client knows the IDs of both accounts, if necessary, it can submit GET requests to the following URIs to obtain more information of each account.

http://bank.org/account/AZA12093
http://bank.org/account/ADK31242

For this example, as the client has the information it needs to initiate an account transfer, let me skip these requests.

Step 3: The client then issues an account transfer by submitting a POST request as follows.

POST /transfers
Host: bank.org
Content-Type: application/xml;charset=UTF-8

<transfer xmlns="urn:org:bank:accounts">
    <from>account:AZA12093</from>
    <to>account:ADK31242</to>
    <amount currency="USD">100</amount>
    <note>RESTing</note>
</transfer>

The server retrieves routing numbers for the accounts, and submits the transfer to some bank-end system to execute the transfer, and returns the following.

201 Created
Content-Type: application/xml;charset=UTF-8

<transfer xmlns="urn:org:bank:accounts">
    <from>account:AZA12093</from>
    <to>account:ADK31242</to>
    <id>transfer:XTA8763</id>
    <amount currency="USD">100</amount>
    <note>RESTing</note>
</transfer>

The transfer is not done yet. The transfer is going to happen asynchronously over a couple of business days (not unusual for inter-bank transactions), and the client can use the ID of the transaction to find out its status.

Step 4: A day later, the client submits a GET request to find the status.

GET /check/XTA8763
Host: bank.org

200 OK
Content-Type: application/xml;charset=UTF-8

<status xmlns="urn:org:bank:accounts">
    <code>01</code>
    <message xml:lang="en">Pending</message>
</status>

Note that, although this implementation uses resources, URIs, representations, and HTTP's uniform interface, it is not RESTful. As we will see in the next section, this example does not take advantage of one of the key constraints of REST, viz., that hypermedia be the engine of application state.

Before attempting to make this RESTful, let me try to write a possible user documentation to accompany this example.

Bank.Org API - URIs

http://bank.org/accounts?findby=someparams

Submit a GET request to this URI to search bank accounts. This will return an accounts document. See the XML schema for more details.

http://bank.org/account/{account-id}

Submit a GET request to this URI to get account details. Here {account-id} is the ID of the account. This will return an account document. See the XML schema for more details.

http://bank.org/transfers

Submit a POST request to this URI to create an account transfer. In the body of the request, include the transfer document. If the request is successful, the server will return a transfer document. See the XML schema for more details.

http://bank.org/check/{transaction-id}

Submit a GET request to this URI to find the status of the transfer. Here, transaction-id is the ID of the account transfer. This will return a status XML document. See the XML schema for more details.

This style of documentation is the most common today. It includes all the URIs that the client will ever need to use. It describes HTTP methods that clients can use with each URI. It also includes descriptions of representations, viz., the XML documents in this example.

But There are two problems with this style of documentation. First of all, it does not help any one looking for a formal machine-readable description. Without a machine-readable description, we can not build generic software tools that can test or otherwise enforce the contract. The lack of such generic software tools is of considerable impediment for those that need to deploy such tools for managing and governing their software. You may consider using WADL, or even WSDL 2.0 to provide a machine-readable equivalent.

Secondly, and more importantly, describing the server's interface in this manner, whether machine-readable as WADL or WSDL 2.0, or human readable, violates two of the constraints of REST that require that (a) messages be self-describing, and (b) hypermedia be the engine of application state. How so and why does it matter?

Back to the Constraints

The key constraints of REST are (a) identification of resources, (b) manipulation of resources through representations, (c) self- descriptive messages, and (d) hypermedia as the engine of application state.

Messages in a RESTful application using HTTP are self-describing by virtue of two things, firstly, by the use of the stateless uniform interface, and secondly, by the use of HTTP headers that describe the contents of messages in addition to implementing various protocol aspects of HTTP such as content negotiation, conditional requests for caching and optimistic concurrency and so on.

By looking at the HTTP method used, and the request/response headers, an intermediate entity such as a proxy or cache can decipher which part of the protocol (viz., HTTP) is being used and how. Such self-describing messages ensure that client-server interactions are visible (e.g. to caches), reliable (e.g. detecting and recovering from partial failures) and scalable.

The fourth constraint, viz., that hypermedia be the engine of application state serves two purposes. Firstly, it relieves the protocol (viz., HTTP) from being stateful. Secondly, it allows the server to evolve (e.g. by introducing new URIs) and keeps clients loosely coupled to servers.

When a server is providing a description of representations as in previous section, it is not taking advantage of the self-describing nature of HTTP. In HTTP, clients and servers use "media types", or those Content-Type headers that we see in requests and responses to describe contents of messages, and not XML schemas. A media type is a like the class of an object, or the schema type of an XML element.

Further, when a server is describing all URIs to its clients, it will not be able to evolve on its own, and its interface becomes brittle. Any changes to URIs will most likely break existing clients. But then, how can you write a client without knowing the URIs that it needs to connect to?

The answer lies in using links with known relations. A link is an indirection that a client can use to discover URIs at runtime. A link has at least two properties - a URI, and a relation. The URI points to a resource, or a representation of a resource, while the relation describes the type or kind of the link. A true RESTful server application conveys URIs to clients by including links with predefined relations in its representations. So, instead of having to learn all URIs a priori, the client can extract URIs from links at runtime. Consequently, the server is free change URIs or even introduce new URIs on the same or some other server that provide a compatible behavior.

Finally, links that a server returns in representations can be contextual, by directing what the client can do subsequently. In other words, links dynamically describe the contract between the client and the server in the form of a workflow at runtime.

To sum up, for a RESTful application, the contract contains three different pieces: the uniform interface, media types of representations, and contextual links to resources.

Sounds like a fairy tale? Let me rewrite the above example to show the contract in action.

Example - Rewritten

Step 0: Same as before.

Step 1: The client uses the same URI http://bank.org/accounts?findby=someparams to search for the accounts. This time, let the server return a different kind of response.

200 OK
Content-Type: application/vnd.bank.org.account+xml;charset=UTF-8

<accounts xmlns="urn:org:bank:accounts">
    <account>
        <id>AZA12093</id>
        <link href="http://bank.org/account/AZA12093" rel="self"/>
        <link rel="http://bank.org/rel/transfer edit"
              type="application/vnd.bank.org.transfer+xml"
              href="http://bank.org/transfers"/>
        <link rel="http://bank.org/rel/customer"
              type="application/vnd.bank.org.customer+xml"
              href="http://bank.org/customer/7t676323a"/>
        <balance currency="USD">993.95</balance>
    </account>
    <account>
        <id>ADK31242</id>
        <link href="http://bank.org/account/ADK31242" rel="self"/>
        <link rel="http://bank.org/rel/transfer"
              type="application/vnd.bank.org.customer+xml"
              href="http://bank.org/transfers"/>
        <link rel="http://bank.org/rel/customer"
              type="application/vnd.bank.org.customer+xml"
              href="http://bank.org/customer/7t676323a"/>
        <balance currency="USD">534.62</balance>
    </account>
</accounts>

In this response, note the value of the Content-Type header, and the links containing URIs.

Step 2: If the client wants to find out more about each account, it can extract their URIs from the links of "self" relation from the above response, and submit GET requests to those URIs.

Step 3: In order to initiate an account transfer, the client extracts the URI from the link with "http://bank.org/rel/transfer" and "edit" relations from either of the accounts above, and submits a POST request to it.

POST /transfers
Host: bank.org
Content-Type: application/vnd.bank.org.transfer+xml;charset=UTF-8

<transfer xmlns="urn:org:bank:accounts">
    <from>account:AZA12093</from>
    <to>account:ADK31242</to>
    <amount currency="USD">100</amount>
    <note>RESTing</note>
</transfer>

Again, note the value of the Content-Type header.

After initiating the account transfer, the server returns the following.

201 Created
Content-Type: application/vnd.bank.org.transfer+xml;charset=UTF-8

<transfer xmlns="urn:org:bank:accounts">
    <link rel="self"
          href="http://bank.org/transfer/XTA8763"/>
    <link rel="http://bank.org/rel/transfer/from"
          type="application/vnd.bank.org.account+xml"
          href="http://bank.org/account/AZA12093"/>
    <link rel="http://bank.org/rel/transfer/to"
          type="application/vnd.bank.org.account+xml"
          href="http://bank.org/account/ADK31242"/>
    <link rel="http://bank.org/rel/transfer/status"
          type="application/vnd.bank.org.status+xml"
          href="http://bank.org/check/XTA8763"/>
    <id>transfer:XTA8763</id>
    <amount currency="USD">100</amount>
    <note>RESTing</note>
</transfer>

Step 4: To find the status of the account transfer, the client can extract the URI from the link of relation "http://bank.org/check/XTA8763" and submit a GET request to it.

This implementation is RESTful as it uses representations with contextual links to encapsulate the state of interactions, i.e., it takes advantage of the constraint that hypermedia be the engine of application state.

Now, let me take a step back and highlight the information required to implement the new set of interactions. To start with, the client needs to know the URI to search for accounts. It then needs to know the names and semantics of various link relations. It also needs to know the details of each media type. It can figure out the rest of the contract dynamically at run time. We can therefore come up with the following revised documentation.

Bank.Org API

URIs

http://bank.org/accounts?findby=someparams

Submit a GET request to this URI to search bank accounts. You can pass customer ID or customer name or customer's social security number as the value of the findby query parameter. This resource supports application/vnd.bank.org.accounts+xml media type.

Link Relations

self

The URI of a link with this relation refers the containing resource, such as account and transfer resources.

http://bank.org/rel/transfer and edit

The URI of a link with these relations can be used to create a new account transfer resource.

http://bank.org/rel/customer

The URI of a link with this relation can be used to fetch a customer resource.

http://bank.org/rel/transfer/from

The URI of a link with this relation identifies the source account resource for the transfer.

http://bank.org/rel/transfer/to

The URI of a link with this relation identifies the target account resource for the transfer.

http://bank.org/rel/transfer/status

The URI of a link with this relation can be used to fetch the status resource.

Media Types

application/vnd.bank.org.accounts+xml

A representation of this media type contains an accounts document declared in the namespace urn:org:bank:accounts. See the XML schema for more details.

application/vnd.bank.org.transfer+xml

A representation of this media type contains a transfer document declared in the namespace urn:org:bank:accounts. See the XML schema for more details.

application/vnd.bank.org.customer+xml

A representation of this media type contains a customer document described in the namespace urn:org:bank:customer. See the XML schema for more details.

application/vnd.bank.org.status+xml

A representation of this media type contains a status document described in the namespace urn:org:bank:transfer. See the XML schema for more details.

Description of a Different Kind

The approach to describing applications RESTfully that I presented in the previous section has some odd as well as some interesting characteristics.

For those familiar with WSDL and WADL, the description in the previous section may look unconventional. Instead of descriptions of input and output messages for every operation, we see media types. However, since generic media types such as application/xml are too generic to help us distinguish representation of an account resource from a representation of a customer resource or a transfer resource, in this example I used custom media types. Each of these media types ends with "+xml", and as per RFC 3023, XML processors (including XMLHttpRequest) can handle such representations as though it is XML. By looking at such media types, the client knows whether it received a representation of an account resource or a transfer resource. More importantly, it does not have to assume anything about the structure or syntax of the URI it used to fetch that representation.

Moreover, instead of listing all the URIs that the application is using, the documentation included just the one URI that the account transfer client needs to start interacting with. Note that, in a different example, we may need to document more than one URIs. The idea is to pre-publish the smallest number possible. Why is this better? It decouples the client from the actual URIs of resources, and the client does not need to know the rest of the URIs until run-time.

Finally, the documentation above did not include HTTP operations applicable for each URI. Instead, I assumed that the client submits an HTTP OPTIONS request on each URI to discover various operations possible, and then uses HTTP GET to get a representation of a resource, HTTP POST to create a new resource in a collection of resources, HTTP PUT to update an existing resource (or create one if the client can assign a URI for the resource), and HTTP DELETE to delete a resource.

To sum this up, describing the contract RESTfully involves the following:

• Pre-publish a few URIs and document the resources that these URIs correspond to. Keep the length of this list as small as possible. These URIs are the starting points for the application.
• Document all media types. For XML based representations, if required, use an XML schema to document the structure of each representation.
• Document all link relations.
• Let clients use HTTP OPTIONS to discover HTTP operations supported for a given URI at runtime. In some cases, the type of the link relation may be sufficient for the client to determine that the server supports a given operation.

Such a description is neither complete nor fully machine-readable.

It is not complete since it just includes the static aspects of the contract, leaving the server to describe the possible workflow at runtime via links.

For those that are already convinced about the benefits of REST and are actively building RESTful applications using HTTP, lack of a completely machine-readable description may not matter.

But for those that are building distributed applications using RPC-like approaches using SOAP, WSDL, and WS-*, and are considering REST, lack of a completely machine-readable description may appear like a show-stopper. However, the amount of work that can be done using a RESTful machine-readable description, if one exists today, is limited. This is due to the following:

• Dynamic nature of the contract: As I described in the example in the previous section, representations describe the dynamic aspects of the contract by using links in representations. Explaining this contextual contract in some XML-based machine readable description outside representations is redundant.
• Flexibility of media types: Unlike SOAP based applications, RESTful applications are not limited to using XML. They can use other media types that do not have schema languages.

Also note that a completely machine-readable description of a contract for a remote interface is a fallacy. A machine-readable description created by using WSDL or WADL can only describe the structure and syntax, but not semantics. But machine-readable descriptions can sometimes help us reduce the amount of work that we, as programmers, testers and administrators, need to do.

If we leave aside the uniform interface and the dynamic aspects of the contract, we can describe the rest of the contract in a machine-readable manner. Here is an example. Note that my intent in this description is only to help tools and frameworks that want to monitor or test client-server interactions, but certainly not to mimic WSDL or WADL.

<description xmlns:bank="urn:org:bank:accounts">
    <types>
        <!-- Include the schema used for all representations -->
        <include href="bank-schema.rng"/>
    </types>

    <!-- List all media types and the corresponding XML types -->
    <media-types>
        <media-type>
            <name>application/vnd.bank.org.accounts+xml</name>
            <representation>bank:account</representation>
        </media-type>
        <media-type>
            <name>application/vnd.bank.org.transfer+xml</name>
            <representation>bank:transfer</representation>
        </media-type>
        ...
    </media-types>

    <relations>
        <relation>
            <documentation>This relation ...</documentation>
            <name>http://bank.org/rel/transfer</name>
        </relation>
        ...
    </relations>

    <resources>
        <resource>
            <name>accounts</name>
            <media-type-ref>application/vnd.bank.org.accounts+xml</media-type-ref>
            <uri> <!-- This is optional -->
                <base>http://bank.org/accounts</base>
                <params>
                    <param>
                        <documentation>Use this parameter to ...</documentation>
                        <name>findBy</name>
                    </param>
                </params>
            </uri>
        </resource>
        <resource>
            <name>transfer</name>
            <media-type-ref>application/vnd.bank.org.transfer+xml</media-type-ref>
        </resource>
        ...
    </resources>
</description>

This is a machine-readable version of the one I described in the previous section, and is certainly not governed by any standard. This description does not eliminate the need for human-readable description, since we still need that to describe the semantics of the application.

Let me highlight the key sections of this description:

• Schema types: Since I chose to use XML for all representations in this example, I included a reference to the schema used. This section would be meaningless if the chosen representation formats are not describable by a schema.
• Media types, and the corresponding XML documents used.
• List of all link relations.
• Names of resources and their media types. Note that the names are not URIs.
• URIs for those resources that provide starting points for the application.

Is this kind of a description more useful than the human-readable description? In the absence of tools and frameworks that can interpret such a description, the answer may be no.

Is this Practical?

If you are writing the server-side code and the client-side code based on a machine-readable contract such as a WADL document, the coding flow may be as follows:

• Generate resource classes from the description. Each resource classes potentially corresponds to a URI in the description.
• Generate classes that bind to representations. If the representations are XML based, I would be generating classes that map to various XML documents.
• Generate client-side stubs that map to various HTTP operations.
• Start coding.

This model won't work for a RESTful description of the contract, and the steps would be different.

1. Read the descriptions of all media types. If the media types are described using an XML schema, grab that schema and generate classes that can parse or generate XML.
2. Read the description of all link relations.
3. Create resource classes manually.
4. Whenever the client receives a representation, in addition to extracting the data from the representation, look for links. If you find a link with a known relation, extract the href and type (if present) attributes for later use.
5. On the client side, before sending an HTTP request, first send an HTTP OPTIONS request to check if the operation you want to perform is supported by the server. If so, proceed to enable that operation in your client application.

Most software frameworks that I am aware of can deal with some of the steps above, such as common interfaces or conventions for resource classes, and depending on your programming language of choice, be able to generate classes to create or parse XML. But the rest is left up to the developer. More over, most of these frameworks emphasize server-side development, and leave out client-side development under the assumption that existing HTTP client libraries are sufficient. Consequently, you may need to create custom code to deal with items (4) and (5) above.

How about software tools that want to test or enforce the contract? It is possible to create such software that reads the above machine-readable description to do the following at runtime.

• Check that the media types of representations are predefined.
• Check that the representations match the predefined descriptions of media types.
• Check that all links contained in representations have predefined relations and media types, and that the URIs included follow predefined URI patterns.

I am not aware of any software that can validate these in this manner. However opportunities abound. If you have read this article this far, you know what those opportunities are.

Conclusion

One of my goals for this article is to illustrate the fact that traditional descriptions of contract such as WSDL and WADL are not adequate to describe RESTful applications. As I demonstrated with the account transfer example, only a part of the contract can be described statically, and the rest is dynamic and contextual. The client can follow this dynamic part of the contract at runtime by looking at links. You can attempt to describe the former via some machine readable document for design-time or testing purposes, but letting the server describe the rest at runtime reduces the coupling between clients and servers significantly. Attempting to describe the complete contract statically is equivalent to duplicating all the contextual links outside representations.

In contrast, description languages like WSDL and WADL try to describe the contract in a context-free manner, and client developers are left to user documentation to learn how to compose a client application from various message exchange patterns described in those descriptions. In RESTful applications, servers provide this information at runtime in the form of links.

To conclude, RESTful applications do have a contract. We just need to what to look for and where, and remember that the contract is contextual.

About the Author

Subbu works at Yahoo. See his blog for more about him.