How Contract Tests Improve the Quality of Your Distributed Systems

Imagine that you’re a developer working in a large enterprise. You look at the code that you’ve been working on for the last 10 years. You’re proud of it since you’ve used all known design patterns and principles to build its foundations. However, you were not the only one working with the codebase. You’ve decided to take a step back and take another look at what’s been built. What you see is this:

Source

After making an internal audit, it turned out that the situation is even more problematic.  We have an immense number of integration and end-to-end tests and almost no unit tests.

Source 

Over the years we’ve been making our deployment process more complex and now it looks like this:

Source

We could limit the number of end-to-end tests but they catch a lot of bugs from integration tests.  We have a problem related to the fact that we can’t catch exceptions when integration (either HTTP or messaging) is faulty.

Why aren’t we failing fast?

Let us assume that we have the following architecture

Let’s focus on the main, two applications, Legacy service and the Customer Rental History service.

In the integration tests of the Legacy service, where we’re trying to run a test that would send a request to a stub of the Customer Rental History service. We’re writing that stub manually as the legacy app. It means that we use tools like WireMock to simulate a response for the given request. Below you can find an example of such a scenario:

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE)
// start WireMock on a fixed port
@AutoConfigureWireMock(port = 6543)
public class CustomerRentalHistoryClientTests {
  @Test
  public void should_respond_ok_when_foo_endpoint_exists() {
     // the Legacy Service is doing the stubbing so WireMock
     // will act as we told it to
     WireMock.stubFor(WireMock.get(WireMock.urlEqualTo(“/foo”))
           .willReturn(WireMock.aResponse().withBody(“OK”).withStatus(200)));

     ResponseEntity<String> entity = new RestTemplate()
           .getForEntity(“http://localhost:6543/foo“, String.class);

     BDDAssertions.then(entity.getStatusCode().value()).isEqualTo(200);
     BDDAssertions.then(entity.getBody()).isEqualTo(“OK”);
  }
}

So what is problematic with this test? The problem often occurs in production where it turns out that the endpoint doesn’t actually exist.

What does it actually mean? Why do the test pass whereas the production code fails?! That’s happening due to the fact that the stubs created on the consumer side are not tested against the producer’s code.

That means that we have quite a few false positives. That actually also means that we’ve wasted time (thus money) on running integration tests that test nothing beneficial (and should be deleted). What is even worse is that we’ve failed on end-to-end tests and we needed to spend a lot of time to debug the reason for these failures.

Is there any way to fail faster? Maybe on the developer’s machine even?

Shifting to the left

In our deployment pipeline we’d like to shift the failing builds as much to the left as possible. That means that we don’t want to wait until the end of the pipeline to see that we have a bug in our algorithm or we have a faulty integration. Our aim is to fail the build of the application in that case.

In order to fail fast and start getting immediate feedback from our application, we do test driven development and start with unit tests. That’s the best way to start sketching the architecture we’d like to achieve. We can test functionalities in isolation and get immediate response from those fragments. With unit tests, it’s much easier and faster to figure out the reason for a particular bug or malfunctioning.

Are unit tests enough? Not really since nothing works in isolation. We need to integrate the unit-tested components and verify if they can work properly together.  A good example is to assert whether a Spring context can be properly started and all required beans got registered.

Let’s come back to the main problem – integration tests of communication between client and a server. Are we bound to use hand written HTTP / messaging stubs and coordinate any changes with their producers? Or are there better ways to solve this problem. Let’s take a look at a contract test and how they can help us.

What is a contract test and how can it help?

Let’s imagine that before two applications communicate with each other, they formalize the way they send / receive their messages. We’re not talking about schemas here. We’re not interested in all possible request / response fields and accepted methods for HTTP communication. What we would like to define are pairs of actual possible conversations that can take place. Such a definition is called a contract. It’s an agreement between the producer of the API / message and its consumer of how those conversations will look like.

There are numerous contract-testing tools but it seems that the two mainly used ones are Spring Cloud Contract and Pact. In this article, we will be focusing on giving you more detailed explanation of what contract tests are via Spring Cloud Contract.

In Spring Cloud Contract a contract can be defined either in Groovy, YAML or a Pact file. Let’s look at an example the following YAML contract:

description: |
  Represents a scenario of sending request to /foo
request:
  method: GET
  url: /foo
response:
  status: 200
  body: “OK”

Such a contract tells us the following

• If one sends an HTTP request with a GET method to url /foo
• Then the response with status 200 and body “OK” will be sent back

What we’ve managed to achieve is codify the requirement of the consumer test that was written against the WireMock stub.

Just storing such pieces of conversations doesn’t mean much. There is no difference in typing that on a sheet of paper or in a Wiki page if we can’t actually verify if that promise is kept on both sides of the communication. In Spring, we take promises seriously, so if one writes a contract ,we generate a test out of it to verify if the producer meets that contract. 

To achieve that, you have to set up a Spring Cloud Contract Maven / Gradle Plugin on the producer side (Customer History Service application), define the contracts and place them under proper folder structure. The plugin will read the contract definitions, generate tests and WireMock stubs from the contracts.

It’s crucial to remember that contrary to the previous approach, where the stubs were generated on the consumer side (Legacy Service), now, the stubs and tests will be generated on the producer side (Customer History Service).

The following picture shows that flow from the point of view of the Customer History Service.

What do those generated tests look like?

public class RestTest extends RestBase {

   @Test
   public void validate_shouldReturnOKForFoo() throws Exception {
      // given:
         MockMvcRequestSpecification request = given();

      // when:
         ResponseOptions response = given().spec(request)
               .get(“/foo”);

      // then:
         assertThat(response.statusCode()).isEqualTo(200);
      // and:
         String responseBody = response.getBody().asString();
         assertThat(responseBody).isEqualTo(“OK”);
   }

Spring Cloud Contract uses a framework called Rest Assured to send and receive test REST requests.  Rest Assured contains an API that follows the good Behavior Driven Development practices. The test is descriptive and all the request and response entries defined in the contract were successfully referenced.  Why do we need the base class though?

The essence of the contract tests is not to assert the functionality. What we want to achieve is to verify the semantics. If the producer and the consumer will be able to successfully communicate on production.

In the base class we can set up the mock behavior of our application services, so that they return fake data. Let’s imagine that we have a following controller:

@RestController
class CustomerRentalHistoryController {
   private final SomeService someService;

   CustomerRentalHistoryController(SomeService someService) {
      this.someService = someService;
   }

   @GetMapping(“/foo”)
   String response() {
      return this.someService.callTheDatabase();
   }
}

interface SomeService {
   String callTheDatabase();
}

In the contract tests we don’t want to call the database. We want those tests to be fast and verify whether two sides can communicate. So in the base class we would mock the application service like this:

public class BaseClass {
   @Before
   public void setup() {
      RestAssuredMockMvc.standaloneSetup(
            new CustomerRentalHistoryController(new SomeService() {
               @Override public String callTheDatabase() {
                  return “OK”;
               }
            }));
   }
}

After setting up the plugin and running the generated test we can notice that we’ve gotten stubs in the generated-test-resources folder an additional artifact with a -stubs suffix. That artifact contains the contracts and the stubs. The stub is a standard JSON representation of a WireMock stub

{
  "id" : "63389490-864e-483c-9059-c1eba8b46b37",
  "request" : {
    "url" : "/foo",
    "method" : "GET"
  },
  "response" : {
    "status" : 200,
    "body" : "OK",
    "transformers" : [ "response-template" ]
  },
  "uuid" : "63389490-864e-483c-9059-c1eba8b46b37"
}

It represents a pair of request, reply that has been verified to be true (due to the fact of passing of the generated tests). When we run ./mvnw deploy or ./gradlew publish what would happen is that the fat jar of the application together with the stubs would get uploaded to Nexus / Artifactory. That way we get the reusability of the stubs out of the box, since they are generated, asserted and uploaded only once, after being verified against the producer.

Let’s now see how we can change the consumer side tests to reuse those stubs.

Spring Cloud Contract comes with a component called a Stub Runner. As the name suggests it is used to find and run stubs. Stub Runner can fetch the stubs from various locations such as Artifactory / Nexus, classpath, git repository or the Pact broker. Due to the plugabble nature of Spring Cloud Contract you can pass your own implementation too. Whatever stub storage you chose you can change the way stubs are shared between projects. The following diagram represents a situation where, after passing the contract tests, the stubs get uploaded to the stub storage for other projects to reuse.

Spring Cloud Contract doesn’t require you to actually use Spring. As consumers, we can call the StubRunner JUnit Rule to download and start stubs.

public class CustomerRentalApplicationTests {

   @Rule public StubRunnerRule rule = new StubRunnerRule()
         .downloadStub("com.example:customer-rental-history-service")
         .withPort(6543)
         .stubsMode(StubRunnerProperties.StubsMode.REMOTE)
         .repoRoot("https://my.nexus.com/");

   @Test
   public void should_return_OK_from_a_stub() {
      String object = new RestTemplate()
            .getForObject("http://localhost:6543/foo", String.class);

      BDDAssertions.then(object).isEqualTo("OK");
   }

}

Here we can see that the stubs of an application with group id and com.example and artifact id customer-rental-history-service get fetched from a Nexus installation available under https://my.nexus.com. Next, the HTTP server stub gets started at port 6543 and is fed with the downloaded stubs. Your tests can now reference the stub server directly. The following diagram represents that flow

So what’s the outcome of such an approach?

• As consumers, we will fail fast if we are incapable of communicating with the producer
• As producers, we can see if our code changes are not breaking the contracts that we’ve agreed upon with our clients

This approach is called the producer contract approach since the producer defines the contracts and all consumers need to follow the guidelines defined in the contracts.

There’s also another way to work with contract that is called the consumer driven contract approach. Imagine that consumers create their own set of contracts for a given producer. Let’s look at the folder structure defined under a producer’s repository :

└── contracts
    ├── bar-consumer
    │   ├── messaging
    │   │   ├── shouldSendAcceptedVerification.yml
    │   │   └── shouldSendRejectedVerification.yml
    │   └── rest
    │       └── shouldReturnOkForBar.yml
    └── foo-consumer
        ├── messaging
        │   ├── shouldSendAcceptedVerification.yml
        │   └── shouldSendRejectedVerification.yml
        └── rest
            └── shouldReturnOkForFoo.yml

Let’s assume that this is the folder structure representing contracts that the Customer Rental History service needs to meet. From this we know that Customer Rental History has 2 consumers: bar-consumer and foo-consumer. That give us insight on how the API is used by which consumer. Also if we make a breaking change (e.g. modification, removal of a field in the response) we will know exactly which consumer will get broken.

Let’s assume that foo-consumer requires an endpoint /foo to return “OK” body, whereas the bar-consumer requires a /bar endpoint to return “OK”. So the shouldReturnOkForBar.yml would look like this:

description: |
 Represents a scenario of sending request to /bar
request:
 method: GET
 url: /bar
response:
 status: 200
 body: "OK"

Now, assuming that after some refactoring on the Customer Rental History side, we’ve removed the /bar mapping, our generated tests will tell us exactly which consumer got broken. This would be the outcome of running ./mvnw clean install

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   RestTest.validate_shouldReturnOkForBar:67 expected:<[200]> but was:<[404]>
[INFO] 
[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

On the consumer side you would have to set up the Stub Runner to use the stubs per consumer feature. That means that only the stubs corresponding to the given consumer will get loaded. Example of such test:

@RunWith(SpringRunner.class)
//Let’s assume that the client’s name is foo-consumer
@SpringBootTest(webEnvironment = WebEnvironment.MOCK,
     properties = {"spring.application.name=foo-consumer"})
//Load the stubs of com.example:customer-rental-history-service from local .m2
// and run them on a random port. Also set up the stubsPerConsumer feature
@AutoConfigureStubRunner(stubsMode = StubRunnerProperties.StubsMode.LOCAL,
     ids = "com.example:customer-rental-history-service",
     stubsPerConsumer = true)
public class FooControllerTest {

  // Fetch the port on which customer-rental-history-service is running
  @StubRunnerPort("customer-rental-history-service") int producerPort;

  @Test
  public void should_return_foo_for_foo_consumer() {
     String response = new TestRestTemplate()
           .getForObject("http://localhost:" + this.producerPort + "/foo",
                 String.class);

     BDDAssertions.then(response).isEqualTo("OK");
  }

  @Test
  public void should_fail_to_return_bar_for_foo_consumer() {
     ResponseEntity<String> entity = new TestRestTemplate()
           .getForEntity("http://localhost:" + this.producerPort + "/bar",
                 String.class);

     BDDAssertions.then(entity.getStatusCodeValue()).isEqualTo(404);
  }
}

Do you have to always store the contracts with the producer? Not necessarily. You can also store the contracts in a single repository. Whatever you pick, the outcome is such that you can write tests that parse those contracts and generate automatically the documentation of how your API can be used!

Also, since you have the parent – child relationship between services, you can easily sketch a graph of services dependencies.

Having the following folder structure:

The following graph of dependencies could be sketched:

Is that all that contract tests give us?

Together with unit and integration tests, contract tests should have their spot in the testing pyramid.

You can check out Spring Cloud Pipelines where we propose placing contract tests as one of the key steps inside the deployment pipeline (API compatibility check). In our deployment pipeline we also suggest using Stub Runner as a standalone process to easily surround your application with stubs.

Summary

Via contract tests we can achieve a number of goals such as

• Creation of a nice API (if the consumers are driving the change of the API it knows exactly how the API should look like to suit their needs)
• Failing fast when the integration is faulty (if you can’t send the request that stub understands, for sure the production application won’t understand it either)
• Failing fast in case of breaking changes of the API (contract tests will tell you exactly which change of your API is breaking)
• Stub reusability and validity (stubs are published only after the contract tests have passed)

Let’s stay connected! Talk to us on Gitter, read the documentation and send us any feedback to the Spring Cloud Contract project.

About the Author

Marcin Grzejszczak is the author of the "Mockito Instant" and "Mockito Cookbook" books. He's also the co-author of Applied Continuous Delivery Live Lessons. In addition, Marcin is co-founder of the Warsaw Groovy User Group and Warsaw Cloud Native Meetup, and Lead of Spring Cloud Sleuth, Spring Cloud Contract and Spring Cloud Pipelines projects at Pivotal. You can find him on Twitter at https://twitter.com/mgrzejszczak.