From Doodles to Delivery: An API Design Process

This series focuses on three key areas of "meta-language" for Web APIs: API Description, API Discovery, and API Profiles. You’ll see articles covering all three of these important trends as well as interviews with some of the key personalities in this fast-moving space.

This InfoQ article is part of the series “Description, Discovery, and Profiles: The Next Level in Web APIs”. You can subscribe to receive notifications via RSS.

Having a basic understanding of web based applications is a good foundation for designing a working Web API. But, if you want to create a good API you need a lot more than that. Designing a good API is hard work and it’s easy to feel overwhelmed when it’s your job to make one.

But, good design is definitely achievable and this article describes a process that will help you succeed. We will investigate what good design is and how an iterative process can help us achieve it. We will also outline three important phases of design: sketching, prototyping and implementation as well as the tools that will make your job easier.

Good API Design Happens in Iterations

Before we can begin designing an API, we must understand its purpose. You should know why you are building an API before you begin and there are many resources available for those seeking help. But, defining your motivation is only the first step. The real trick is in making good design decisions all the way to implementation.

Succeeding with an API design means designing an interface whose usage fosters its purpose. As API designers, each decision we make has an impact on the success of the product. There are big decisions to be made, such as the transport protocol that the API will use, or the message format that it will support. But, there are also many smaller decisions related to the controls, names, relationships and sequences of an interface. When you put them together, all these decisions drive a pattern of usage. If you’ve made only good decisions, then that pattern will support and foster the purpose of the API perfectly.

If you want to make a correct design decision, you’ll probably need to make the wrong one first and learn from that experience. In fact you’ll probably need to make many mistakes, many times before you can get close to being correct. This is the crux of iteration - no one gets it right the first time, but given enough opportunities you can get closer to perfection.

It’s easy to say we should be iterating our API designs, but it’s hard to do that in the real world. A common challenge we face is the difficulty in changing an API after it has been released. The truth is that changing an API that is in use can be expensive and risky. Or, as Joshua Bloch says: "Public APIs, like diamonds, are forever.".

One way to address this problem is to stop breaking the interface every time we change it. This is an excellent habit to have and is a cardinal rule for good API design. But, sometimes breaking changes are unavoidable and fundamental design changes are especially invasive.

Instead, we need to make these changes well before the interface is released. Ideally, all the usability and design issues should be ironed out before change becomes expensive. Of course, finding those problems and fixing them is only possible if we can iterate early and often before the first release.

An Iterative Design Process

Each iteration gives us a chance to measure our design candidate according to its usage. For example, will a developer be able to accomplish their goals using what we have built? Can this interface actually be implemented? How will people feel if we ask them to use this API?

We should be able to achieve the best API design by designing and implementing many interfaces without releasing them. Reviewing and testing each interface would provide powerful insight into how the end product could be improved.

But, in practice, this type of grand iterative design is impossible to achieve. Few of us have the time, money or inclination to design and implement a working API over and over again. For any non-trivial interface this type of iterative design would just take too much time.

A more reasonable approach is to perform iterations very early in the design process. These early designs should have just enough detail to expose the biggest opportunities for improvement, but not so much that they are hard to build. Over time, we can gradually increase the level of detail (or fidelity) until we finally arrive at a fully realized implementation.

This graduated process is popular in the design world and is typically broken down into three important phases:

1. Sketching
2. Prototyping
3. Implementing

The Power of Sketching

Sketching is a universal act of design. Architect Frank Gehry's sketches are famous enough to have had a film made about them. Many of his building projects have started life as a series of sketches drawn on sheets of paper. Gehry draws hundreds of these sketches, each time moving closer to an ideal design.

Interaction designer Bill Verplank describes sketching as an essential first act in the design process. Bill Buxton has written an entire book on the value of sketching for user experience design, calling out its key characteristics of disposability, exploration low cost amongst others.

Incorporating sketching in the early phases of an API design process gives us an opportunity to experiment with the conceptual model of the interface. This is a chance to not only define the ideas that we have in our head already, but also to explore new avenues and ask the "what if?" questions that can lead to real innovation.

Good sketches should be easy to make and easy to destroy. If they take too long to create or are too difficult to make they become harder to walk away from. This disposability property is important because it affords us with an opportunity to take risks.

We can experiment with different types of interface styles and capture the abstract concepts that are floating around in our heads. Once, these ideas are material we have an opportunity to review and discuss them. We get to decide what we like and dislike about a particular notion and start the process again with new sketch that applies that knowledge.

Sketches are rarely evaluated by users who are not embedded in the design team. Not because there isn’t value in testing assumptions early, but because there is a finite amount of user testing that can be done in a real project. Testing every sketch with real users would be an expensive endeavour with limited return.

Sketching w/ Profiles

Using a profile or meta language for an API is useful when sketching it. Profiles provide a set of constructs from which a sketch can develop. A good profile can be analagous to the boxes and lines that form many sketches of systems diagrams. These are elements that both the designer and evaluators understand and make it easier to develop sketches quickly.

In fact, a good way to start the API sketching process is to define the list of words that will feature prominently in the interface. Which words must a user know? Which of them best capture the goals and tasks of your target audience? Answering these questions and creating an interface vocabulary will help you as you form an early sketch of the interface.

The wonderful thing about profiles is that they provide us with a formalised way of sharing and re-using this type of information. For example, we might start our design by picking words from an XML schema document, a vocabulary retrieved from schema.org, or perhaps an ALPS or RDF document depending on our needs.

In the design phase the goal should not be to create a comprehensive vocabulary. Instead, the early vocabulary forms a starting point from which we can draw out other types of detail. We might find that a list of twenty words captures the essence of our interface and leave it at that.

The vocabulary provides a base from which we can sketch the resources and relationships of the API. This may include the URIs, resource names, relationships between resources, link text and other structural and navigational elements. Again, there is no need to sketch every detail, instead the aim is to express the most important parts of the API.

Above all, it’s important not to go to far with the initial sketches. For example, try to avoid diving into error flow modelling or response message elements at this stage. These ideas can be added later or may even become sketches in their own right.

A single sketch does not have to reflect the entire interface. In fact, it is useful to sketch certain details on their own. For example, we might sketch a basic error flow that will be relevant for the entire API or a response message format that can be applied to all responses. Later, during the prototyping phase we can apply these ideas to a working model.

Prototyping

The prototyping stage is an opportunity to build a higher fidelity model of the interface and test some of the assumptions we have made during sketching. A good API prototype should be invokeable . It should handle real request messages and provide responses when required. Developers should even be able to create a simple applications by using the prototyped API.

But, prototypes should still be cheaper to create than a full-on implementation. One way to keep the cost down is to simulate response messages rather than serving real responses from your backend systems. This is sometimes referred to asmocking the interface and is a great way to build a prototype quickly.

Whichever way the prototype is built, it is important to find a level of investment that makes it possible to build new iterations. We should be able to build two or three different prototypes based on sketches, learning as we go along. We may even go back to the sketching phase to experiment with new directions based on what we learn from the prototyping phase.

Prototypes offer your team an opportunity to gather early user feedback on your design and allow for observation of real usage. If the interface has the right level of fidelity, you can ask potential users to build an application and observe the challenges they face in their endeavours.

Implementers should also participate in the prototype evaluation process. Well designed APIs should not only be easy to use, but also sustainable, reliable, performant and long-lived. While the interface shouldn’t expose the inner details of data models or server architecture, implementers can advise the design team on environment constraints and implementation costs for some of the decisions that have been made.

The design cycle is like a scientific process and the prototype phase is your chance to test any hypotheses before it is too late to make changes.

Implementing

The implementors job is to turn a prototyped interface into something that can be relied upon for serious use. The task of delivering a safe, reliable and scalable implementation is challenging and will usually need a design process of its own.

The final prototoype and supporting sketches form a description of what the interface should look like. They reflect the collective design decisions to form a specification of what needs to be built. In fact, it’s useful to use a formal interface description language (or IDL) to transition from the prototype to the implementation phase.

For example, when you are happy with your prototyped API you might choose to describe it in an API Blueprint document (or Swagger, RAML, WADL, or whichever format best fits your working environment.)

Although implementation is the goal, design shouldn’t stop at this stage. This is an opportunity to further test the assumptions you’ve made throughout the design process with real usage data. Just as protoyped APIs allow us to observe usage, implemented APIs allow us to analyze usage at a macro level.

For example, you might want to validate the design assumptions you’ve made. Are application developers really using that convenience operation you created for them? Are you getting the type of users you expected? Are new users having trouble with specific parts of the interface?

This analysis, may lead you to starting the sketching process again in support of a further improved interface.

Automating the Process with Tools

Tooling and technology can radically improve the design process. Tools that lower the cost of sketch and prototype creation will allow design teams to produce more designs in less time leading to improved design decisions.

Incorporating tooling and automation is an essential part of most design processes. In the building design world, SHoP (a consortium of architects based in New York) are succeeding through innovation, collaboration and tool based design. Their process includes prototyping tools that allow designers to incorporate the physical characteristics of the materials that are being used. This method allows them to create thousands of design iterations, each incorporating implementation details that can be easily evaluated.

There is ample opportunity for this type of tool based optimization in the API design space. Indeed, some excellent web API design tools already exist in the service description domain.

It is now common to find that interface description languages provide supporting tools to ease the task of writing a description. These editors shorten the creation time for IDL based design, making it easier to create more descriptions in a shorter time. Swagger, RAML and Blueprint all offer excellent editing tools to support their languages. Even IDLs like WADL that are released only as specifications benefit from tools like SoapUI.

Apiary provides an editor for the Blueprint language is particularly compelling because it offers a complete workflow of tools to support the design process. By simply writing an API description in Blueprint, the designer can evaluate the documentation, invoke a prototype and even analyse invocations.

(Click on the image to enlarge it)

But, most of this existing tooling is constrained to the particulars of the description language it supports . The designer must understand the syntax of the IDL and design their interface using this language. While this style of design can appeal to users who are familiar with programming languages, it can also limit the abstract and experimental thinking that is valuable in the early sketching phases.

These characteristics make IDLs and their associated tools an ideal fit for the prototyping process, but less helpful for the early experimental activity of sketching.

Sketching with Visual Tools

There is a rising interest in the use of visual tools to help with API design. Rather than supporting the act of editing an IDL directly, these tools allow designers to play with a visual representation of the interface.

Visual modelling tools provide an interface description language that is mostly pictorial or diagram based. While this perspective constrains the fidelity of the interface representation it also allows the designer to see the entirety of the interface at a high level. This opens up opportunities for improvement that may not be apparent in written form.

Visual editors that are easy to use make good candidates for automated sketching. The mix of low fidelity, abstract representation and disposability is exactly the right fit.

I’ve been working on a visual modelling tool called Rápido! to support API sketching. Rápido constrains the user to lower fidelity detail not as a side effect of its form factor, but by design. Users can model profiles, navigational elements and response data, but not logical flows or dynamic responses.

When a designer starts a new Rápido project they are asked to create a vocabulary for the API. After coming up with an initial list of words (or importing an ALPS vocabulary from an external source), the designer can start to form the conceptual model for the API, creating resources, experimenting with URI names and even linking states together in a hypermedia canvas.

Eventually, the designer can increase the fidelity by implementing static response messages for each resource or state they are sketching. Finally, when the sketching phase is complete, the design can be exported in IDL format ready to be plugged into higher fidelity tools for the prototyping phase.

The goal of the Rápido experiment is to deliver the experience of a quickly drawn "cocktail napkin" sketch to the web API design space. If the fidelity is correct and the constraints are not too tight, it can form an important part of an API design tool chain.

Succeed with an Iterative Process

Follow the iterative style of design described in this article and you will give your team a chance to create an effective API. Create and evaluate many low-fidelity designs at the beginning of the process to foster experimentation and ideation. Build higher-fidelity prototypes and mock implementations to evaluate early design ideas. Finally, implement the design for real users and gather data for analysis of real world usage.

The specific details of an iterative design process will depend on your unique environment and project. The exact amount of detail, number of iterations and evaluation techniques are left to the designer. But, as a general rule of thumb, the number of iterations should decrease as you move through the design process, while the evaluation effort will probably increase.

A good API design process gives you an opportunity to produce the best interface possible. The secret to building a great API is not expert guidance or secret knowledge. Instead, it is the application of an iterative process that is optimized by great tools, languages and profiles. Use this formula to deliver an API that will help your organization succeed.

About the Author

Ronnie Mitra is Director of API Design for CA’s API Academy. He helps organizations around the world design and implement APIs successfully. Ronnie has over thirty years of experience failing (and learning) iteratively.

This series focuses on three key areas of "meta-language" for Web APIs: API Description, API Discovery, and API Profiles. You’ll see articles covering all three of these important trends as well as interviews with some of the key personalities in this fast-moving space.

This InfoQ article is part of the series “Description, Discovery, and Profiles: The Next Level in Web APIs”. You can subscribe to receive notifications via RSS.