Microservices work well with independent teams who have end-to-end responsibility and use an automated CI/CD pipeline. Ensuring software quality through end-to-end testing can conflict with rapidly integrating and releasing software components. If an end-to-end test fails, the CI/CD pipelines of all involved microservices are blocked until the problem causing the test to fail is solved.
Lukas Pradel, a senior consultant, spoke about the challenges of end-to-end testing microservices at Agile Testing Days 2020.
Pradel mentioned that implementing end-to-end tests for microservices-based systems is a rather cumbersome effort. Technically, it will likely not even be that big of a deal to replicate your production environment for testing purposes. But since the services are heterogeneous and maintained by different teams, seemingly innocent tasks such as feeding your services with consistent test data can be quite challenging, especially when real-time data and external or legacy systems come into play, Pradel said.
Pradel mentioned that end-to-end tests are notoriously flaky, and it’s even worse in the case of highly distributed systems. "Your tests will fail often for a myriad of reasons", Pradel said, "There’s just so many things that can go wrong."
There are also the issues of identifying the culprit and the impact of failing test cases, as Pradel explained:
We’re using black box tests and in a distributed system it’s not trivial to locate the cause of an error.
And finally, failing end-to-end tests will block the delivery of all involved microservices.
It’s easy to underestimate the organizational effort of getting all teams on the same page and establishing reliable and effective communication, as Pradel explained:
Who writes, implements and maintains your end-to-end test cases? Responsibilities are suddenly not clear anymore as there are no natural service boundaries anymore. In a real-world scenario, you will be faced with 50 or more services maintained by 15, maybe 20 teams, each with their own backlogs and priorities.
There is no simple answer to the question of whether you should test your microservices system end-to-end or not. Deciding in either way is fine, but the decision should not be made lightheaded and unaware of the alternatives, as Pradel explained:
End-to-end tests will increase the quality of your software and help you find potentially dangerous bugs, but you will also considerably hamper your deployment frequency and lead time and see coupling between services and teams. As the saying goes, you can’t have your cake and eat it too.
InfoQ interviewed Lukas Pradel about end-to-end testing of microservices.
InfoQ: What are the test stages for testing a microservices-based application?
Lukas Pradel: An old friend – the test pyramid – once more proves to be of great value here. Starting at the bottom, we have our trusty unit tests. Since a microservices architecture is chosen with the goal of rapid delivery in mind and test-driven development is therefore pretty much mandatory, there might be hundreds of unit tests and that’s perfectly fine, because reliability, speed and isolation are highest at the bottom of the test pyramid.
Next, we have integration tests. With these we verify if all of our units work in conjunction and with actual databases and messaging systems. There will be considerably less integration tests compared to unit tests and even fewer test cases as we move further up the pyramid.
Then, we sometimes have contract tests which ensure that contractually agreed interfaces between a provider and a consumer are implemented correctly.
Moving to the top of the pyramid we have UI tests in case our microservice has a user interface.
Finally, at the very top of the pyramid we find ourselves with end-to-end tests where a test case roughly maps to a customer journey. For those, we apply a certain input at the very edge of the system where users can interact and inspect the resulting output in a black-box manner.
As you climb up the pyramid, the cost, effort and complexity of your tests will rise significantly, whereas their reliability, speed and isolation will drop.
InfoQ: Given the challenges of end-to-end testing of microservices, what are the alternatives?
Pradel: The big goal of end-to-end testing is to minimize the probability of defective service releases which might impact our users. However, we have additional tools at our disposal to achieve this goal. For example, with blue-green deployments you have two production environments one of which is your stable environment (let’s say blue) that receives all the traffic. When you roll out a new service, you deploy it only to the green environment and then route just a small but continuously increasing fraction of traffic to the green environment. If something goes wrong, you can quickly roll back by routing all traffic to blue again. If all is well, then eventually green becomes your new stable environment and you start over with the next deployment.
Then there is the option of feature toggles, though I believe they should be used moderately. Furthermore, with any distributed system, you must have impeccable Application Performance Management (APM). Ideally, you will know about something being wrong before your users do. And finally, it should be worth noting that end-to-end testing can be done in production. This also solves the issue of failing end-to-end tests blocking the delivery pipelines of your microservices.
InfoQ: You mentioned that ensuring software quality through end-to-end testing can conflict with rapidly integrating and releasing software components. Can you elaborate?
Pradel: A microservices architecture is chosen because there is a strong need for short lead time, rapid and independent deployments of components and if you have the organizational structure to back it up – small, empowered independent teams with end-to-end responsibility. Hence, they are often powered by a fully automated CI/CD pipeline. The drawback is that you end up with a high demand for communication and a complex socio-technical system.
End-to-end tests kind of ruin the party here: as soon as one fails, the CI/CD pipelines of all involved microservices are stalled until the issue is identified and resolved. But this can take time, hours or even days. Suddenly, your teams are nowhere near as independent as you thought anymore. In fact, we were faced with "deployment windows", meaning long stretches of a work day where the end-to-end test environment was red and no one was able to deploy, interrupted by small green windows where everyone would rush to deploy only to find that some end-to-end tests failed again for entirely unrelated reasons. The result was quite frustrating for all involved teams.
InfoQ: How did you deal with this dilemma, what solution did you come up with?
Pradel: With all the headache that end-to-end tests have given us over the years, it’s easy to forget that they do also serve their purpose. Make no mistake: they have helped us identify numerous bugs in our microservices. We would have shipped those to our users.
At the end of the day, you need to carefully weigh your priorities with your stakeholders; if software quality is paramount, then end-to-end tests are mandatory with all their drawbacks.
However, if your need to rapidly ship features and the pros of microservices outweigh your need for quality, for example because your product is situated in a highly competitive market, then you might be well advised to consider the alternatives we have discussed. In our case, our stakeholders decided on the former so that’s what we went with and accepted the downsides.