How to be Confident That Your Microservices Can Still Communicate in Production with Pact and Docker

In the beginning, there were Monoliths. Monoliths are large applications that contain all of the business logic for a single project. Over time, Monoliths tended to grow larger and more unwieldy, and became harder and harder to deploy. And so a new paradigm was born; Microservices.

Most are aware of what Microservices are and what the bring to the table regarding scalability and business domain breakdown. Another major part to Microservices is communication. Whereas with Monoliths, you’d have method calls which would then access a data source, you’re now sending HTTP requests over the network to another service.

This communication requires a lot of management. In Java, for example, if a method doesn’t exist, your code won’t compile. However, if a HTTP endpoint doesn’t exist the first time you might find this out could be once the application has been deployed.

With Microservices being changed and deployed rapidly, how do you bring order to this chaos? How can you ensure that your service can continue to provide the right data to downstream services, in the correct consumable format? Testing becomes very important.

As with most forms of testing, the smaller the item under test the easier it is to test. Microservices are no exception, but do so bring some interesting challenges to the table. For example, how do you test an application flow if fifty Microservices are required to enable the flow?  

You could spin all applications up on your local machine. Of course, the more Microservices you have, the more this approach doesn’t scale. For example, Uber has over a thousand Microservices. There’s no way your 16 GB Macbook Pro is going to handle that many.

You could have a QA or Test environment, where you have your entire application up and running. This is great, as you can see your application working with real data, and have all the moving parts on show. The biggest flaw to this approach is that you now have to release your entire application in one go. Teams can’t independently deploy services, as they have all been tested as one. Any changes made to one Microservice, would mean you’d need to revalidate the entire QA environment to ensure everything is still working.

This is where Consumer Driven Contracts come in.

Consumer Driven Contracts

Consumer driven contracts aren’t a new invention. They’ve been around for a number of years. There’s an excellent blog post by Ian Robinson on Martin Fowlers blog, which perfectly outlines Consumer Driven Contracts.

So, what are consumer driven contracts (CDC)?

A contract between a consuming service and a providing service, stating what the consumer wants from a providing service, in a defined format.

They are a way of describing how a consuming and providing service should interact with one another, but where the consuming service drives the communication.

There are a couple of key benefits to this approach. First; they enable services to be deployed independently. No longer do you need to have a QA environment with all of your services running in one go to test that your application works. You won’t need to do long, brittle, and expensive flow tests to ensure that your services can start up and communicate.

Another key benefit is that contracts can be generated independently of a service. Imagine you need a new endpoint for a providing service. You know what it should look like, and the exact data the new consumer will require.

The generated CDC is then given to the provider’s team for them to implement. You can then rest easy in the knowledge that, as long as the providing team follows the CDC, everything should work. This approach works very well with distributed teams. The CDC can be created, and both the consuming team and providing team can work on the code at the same time while being the entire world apart.  

The final, more nuanced part of CDC, is that they enable you to verify what you’re building. Working from the provider perspective you can’t be 100% sure that the service provided is actually going to meet the consumer needs. By starting from a consumer perspective, you can define what should be provided and in what format.
Also, the consuming service shouldn’t require any extra logic or conversions to use the data, as the contract should already defines the format from the consumer perspective.

CDC are not a silver bullet. There are a number of things CDC do not cover. To start with, they are not a test of business logic. That should be covered by your service’s unit tests.
 
If you’re going to change the underlying business logic of an endpoint, but not change the format that’s being returned, then that should be fine as the consuming service shouldn’t be worried about how the data was created or derived. If a consuming service does care how the data was generated, then the business boundaries aren’t properly defined. However, as with all things, communication between teams is a must when making changes.

CDC aren’t a Service Level Agreement (SLA) between services either. They do not state how long a service should be available for, or how many requests per minute it can handle.

Also note that CDC can’t validate external APIs service responses. CDC can’t validate Google Maps, for example. If an external source decides to change their format, that’s their choice. Always best to keep on top of communication between you and third parties, and be ready to migrate when required to.

How Rightmove does Consumer Driven Contracts

Rightmove is the UK’s largest property portal, with over 50 million requests a day. We help people find their dream home, whether it be a quaint old farm house or a modern top of the range London apartment. At Rightmove we have Microservices that are continuously integrated and deployed. This presents a lot of challenge and orchestration, which led us to using consumer driven contracts.

After applying CDC within Rightmove, we came up with a wish list so it would be easier to share what we had done any why.
 
The CDC wish list:

1. A consistent format to describe requests and responses between provider and consumer
2. An easy way to create contracts
3. A way of storing the created contracts
4. A way of testing our services against the contracts, in an automated way, in our CI/CD pipeline
5. A way of running these tests locally

1. A format for CDC

CDC are a concept so a defined, concreate, format is required to use them. At their very core, CDC are nothing more than a statement of “if I send this request, I expect this result”. This is where the Pact Foundation comes in. They created a consistent, JSON format for describing a request and response. Contacts in Pact are called Pacts and they are simple JSON files that contain the request and the expected response. If your service sends a HTTP GET request to /user/125sb45sd, your service expects a 200 OK status code and a JSON body, along with any expected headers.

The wonderful part about Pacts being written in JSON is that it’s a universal data format. Pretty much every programming language has a JSON parser, and as a result there’s a multitude of libraries around generating pacts. In particular, the consuming and providing services don’t even need to be written in the same language for this to work.

2. Creating CDC

The Pact Foundation also have a wide range of libraries for generating CDC, in many different languages. In Java, you’ll find the Pact JVM library. Creating a pact looks very much like writing a JUnit test.

[Click on the image to enlarge it]

(source – Pact JVM Consumer JUnit Library)

When this “test” runs, it’ll produce a JSON file – the pact. This means it’s really easy for developers to understand what’s going on, and build and extend them. Your pacts are defined in code, and are reproducible. As this runs like a JUnit test it’ll create the pact files whenever you run your tests, such as during a build.

3. Storing CDC

At Rightmove, we our own Pact Broker service that we built in-house (though there’s also one provided by the Pact Foundation), which stores all of our pacts in a central location. This can then be queried using a RESTful API. We store all of our pacts against the service name and the version number, during the services build process.

We also have a Deployed Versions service which we developed in house. The Deployed Version service enables us to know which version of a service was deployed and when. Along with Pact Broker, this enables us to get a lot of visibility around changes and leads to better testing.

4. Running the tests within a CI/CD pipeline

Within our Continuous Delivery pipeline, we have a testing block where we check if the service passes its CDC.
Our CD pipeline is a pretty standard affair, where we run a “commit” job that will generate our deployable production artefact, as well as our pacts. The production artefact will be uploaded to our repository, while the pacts are uploaded to the Pact Broker.

During the “commit” step we also generate a “stub” of our application (if it’s a provider), which contains only the controllers and a stubbed service layer. This “stub” is used instead of the production artefact throughout the CDC testing, as the artifact doesn’t require data sources or upstream dependencies; they’ve been stubbed out.

[Click on the image to enlarge it]

If the CDC step fails, the pipeline will fail. This means the potential production artifact can’t continue, and we have to investigate what went wrong.  

So, what do these tests actually look like?

First, we have to break this process down into two different flows – the Provider flow and the Consumer flow. The tests run differently depending on if the service under test is a consuming service, or a providing service. It should be noted that if a service is both a consumer and a provider, we run both.

Originally, we would only test the provider and its flow. However, we found it beneficial to test both consumer flows and provider flows during their respective pipelines. This ensures a consumer can’t change what it wants from an API without the provider being aware of these changes, and being prepared for them. While this does lead to needing some extra orchestration between teams, it gives us confidence that when we roll back an application in production, it’s consuming services will still work, and it’ll still be able to consume.

Provider testing flow

[Click on the image to enlarge it]

It’s easier to think of this flow when you consider a set of concrete microservices. In this example, we want to test a microservice called Location. It has two consumers services: Location-Frontend and Management.

Here’s what the testing process looks like, broken down:

1. Download the latest Location-Frontend and Management service pacts for the Location service
2. Download the stubbed Location service
3. Start the stubbed Location service
4. Gradle runner reads the Pacts, and sends requests and checks the response
5. Shut down the stubbed Location service
6. Publish the results

To start with, we download the JSON pact files for the Location service. Pact Broker facilities this by finding Locations consumers, and then the latest pacts for these consumers. In this case, it’s the Management service and Location-Frontend.

Next, we download our Location service. However, we don’t download the “real” Location service; the one that’ll actually be deployed. Instead this is a stubbed version, where it’s logical-service and data-layer have been removed. This leaves only the controllers and a start-able web service. This removes the dependency on upstream services and data sources. The stub is just the representation of the API layer and fake data. Remember, CDC are not testing business logic. We achieved this using Spring, to swap out the injected dependencies. Other Dependency Injection frameworks should be able achieve the same.

To run the contents of a pact against the stub, we use a runner made for Gradle. This we created in house using the Pact JVM Gradle plugin. This effectively acts as a harness to starting up the stub, reading the pacts, send requests, and record the results.

Consumer Pact flow

The Consumer pact flow is very similar to the provider pact flow in so far as it downloads stub artefacts and runs pacts against them. However, there are a few key differences.

[Click on the image to enlarge it]

Once again, this is easier to understand when you consider a set concrete of microservices:

For example, let’s consider Location-Frontend. Location-Frontend is our Backend for Frontend (or BFF) service for displaying search results. To do this, it finds a location via our Location service, and then passes it on to our Search service. This means Location-Frontend consumes from two providers.

When the consumer pipeline gets to the CDC testing phase, it’ll download its own (latest) pacts from Pact Broker. If a service consumes from multiple providers, it’ll download a Pact for each of its providers.

Let’s break down the flow into steps:

1. Download latest pacts for Location-Frontend, which include Location and Search (the providers)
2. For each provider
   • a.    Download and start stub
   • b.    Gradle runner reads the pact between Location-Frontend and provider, and sends requests / checks response
   • c.    Shut down stubbed provider
3. Publish results

As you can see, we still download the provider stub, but run it against the single pact that our consumer generates, not multiple pacts for each of the providers’ consumers.
It’s effectively the same flow as the providing pipeline, except we spin up each Provider for the single given consumer.

5. Running Tests Locally

As discussed previously, we create a stubbed counterpart to our “real” microservice, which stubs out the data layer and anything extra. It’s mostly a shell with controllers and fake, static, data.

One issue with microservices is running them all in one go locally. Pact goes some way to solve this, but it would be nice to test your applications pacts locally without needing to go through the delivery pipeline.

This is where Docker comes in.

Docker enables us to run our stub applications on any platform in the same way, including in the pipeline. There are many libraries that enable us to interact with Docker, so Rightmove created a new project called the Pact Runner, which is a standalone jar.

The Pact Runner was created to replace our Gradle runner we created to run the consumer and provider flows. This Gradle runner came as two projects; one for each flow. We found it hard to maintain, and we couldn’t use it to run pact tests locally. The Pact Runner was designed and built to fix these issues. Instead of being built on the Pact JVM Gradle library, we instead wrote it on top of the Pact JVM Provider library. This gave us much greater control, and enabled us to write something that was portable and could act as both the consumer and provider runner. The use of Docker enabled it to be platform independent, and also improved the speed of our pact testing as we could cache the service images locally and re-use them between different runs.

It should be noted that the Pact Runner follows described consumer and provider flow above; it’s just an implementation of this logic. For example, and depending on the testing flow, the Pact Runner will download the stub provider(s), which are in Docker Image form, start the image as a container, run the pacts against the container, then shut it down and publish the results.

Pact Runner enables teams to run the same tests they would run within their delivery pipeline.

Wrap up

While this approach isn’t perfect it does enable our teams to work independently, and be confident that their changes won’t break other services when deploying their own. It also improves communication between teams, and helps to get developers thinking about API design early on. Future improvements include using Pacts as a source of automatic endpoint documentation which would enable easier API versioning.

Since implementing CDC at Rightmove, we’ve learned a lot. Our teams are able to deploy services independently of each other, while being confident that their services can continue to communicate. We’ve gone through several iterations, from testing just the provider, to testing both consumer and provider, and also testing our live and rollback applications. We realised that we needed a way to make it easier to test locally, which then pushed us towards the platform-agnostic features that Docker provides.
We wouldn’t be able to do any of this testing without the Pact Foundation, which has many great tools and documents on how to achieve this style of testing for your projects. Many thanks to the hard-working people who make all of this stuff, free and open source.

About the Author

Harry Winser is the Search Team Tech Lead at Rightmove. Prior to this he was on their Platforms team, where he got to play with lots of interesting internal projects to enable developers to delivery code continuously. When he’s not working, he can usually be found hacking on small projects, blogging, playing guitar, or cruising the British canals on his narrow boat.