Stubbing, Mocking and Service Virtualization Differences for Test and Development Teams

Service virtualization is a technique for decoupling a test suite from environmental dependencies that is growing in popularity. It is part of the broader category of "test doubles" that include mocks and stubs. There are many tools that support either service virtualization or two very similar approaches: stubbing and mocking. This article offers a comparison of the three approaches, their relative strengths and explores a number of real world situations to recommend which technique may offer the most benefit. The intent is to provide an understanding of all three techniques that enables to you to choose the best option between a mock, stub, or virtual service depending upon the specifics of your situation.

Where and when do stubbing, mocking and service virtualization apply?

Let us have a look at three examples you might find yourself in.

The first one is a scenario where the testing team is testing the system manually (Figure 1). The system is connected to backend systems that are used by many teams for testing purposes. The tests are very fragile since there are many moving parts. The environments where the backend systems are deployed, are often down because of environment and deployment issues. Setting up test data in the backend systems may take days or weeks and is prone to errors. The testing team is often waiting for the systems or test data to become available. If you are a tester, you are often frustrated because you cannot do your job effectively. If you are a manager, you are frustrated because the team spends much of their time blocked by external dependencies. On top of that, you may spend a great deal of money setting up test data and paying for test transactions to an external third party API.

Figure 1: Traditional manual testing problems example

One way of solving the issues of too many dependencies and too many moving parts is by doing system testing before integration testing with external backend systems (Figure 2). Using virtual services or stubs, allow you to decouple the testing from the real backend systems. The problems mentioned above, either disappear or lose priority. People are happy for a while.

Figure 2: Manual testing with service virtualization example

Eventually, you may notice other bottlenecks in your testing process. If you are a tester, you want to start writing automated tests and automate the repeatable daily activities. This lets you focus on what you are best at, creative and critical thinking to test the system. As a manager, you encourage your testers to explore other ways of testing the system and look into automation. You decide to try one of the BDD frameworks, such as Cucumber and start using “over the wire” (remotely located and run) stubs and mocks

Figure 3: Automated testing approach example - test harness

Unfortunately, the developers and testers have problems communicating, they seem to be using the same words but not getting to consensus. Also, the consultants that have joined recently, advertise service virtualization as a superior way compared to stubbing or mocking. What do developers mean when they use the word mock or stub? Let’s look at the differences between stubs, mocks and virtual services, and what might be causing the communication issues.

What is a test double?

A test double is an object that stands in for a real object in a test. A test double allows us to decouple the application from the dependencies when testing the system under test (SUT). You replace the dependency with an equivalent interface that allows you to exercise a given test case.

Common categories of test double used by developers:

• dummy object (a string “Mike”)
• stub (a StubUserRepository class that always returns user object representing a male named John, age 32, living in US)
• spy (a SpyHttpResponse class that records all invocations of the onPost method)
• fake (a FakeDatabase class which persists to an in memory H2 database instead of the DB2 production ­system)
• mock (a dynamic proxy implementation of OrderObserver interface, implemented by Mockito and used in a unit test)

Common categories of a test double used for testing and quality assurance:

• stub (a servlet in a WAR file created using SoapUI and deployed to a remote Jetty instance at http://testEnv.mycompany.com/getWeatherService)
• virtual service (an artifact created with a service virtualization tool and deployed to a remote shared virtual service environment at http://vsenv.mycompany.com:9034/getWeatherService)

Mocks, stubs and virtual services 

The most commonly discussed categories of test doubles are mocks, stubs and virtual services. 

Stub: a minimal implementation of an interface that normally returns hardcoded data that is tightly coupled to the test suite. It is most useful when the suite of tests is simple and keeping the hardcoded data in the stub is not an issue. Some stubs are handwritten; some can be generated by tools. A stub is normally written by a developer for personal use. It can be shared with testers, but wider sharing is typically limited by interoperability issues related to software platform and deployment infrastructure dependencies that were hardcoded. A common practice is when a stub works in-process directly with classes, methods, and functions for unit, module, and acceptance testing. Some developers will say that a stub can also be primed, but you cannot verify an invocation on a stub. Stubs can also be communicating "over the wire", for example HTTP, but some would argue that they should be called virtual services in that case.

Mock: a programmable interface observer, that verifies outputs against expectations defined by the test. It is frequently created using a third party library, for example in Java that is Mockito, JMock or WireMock. It is most useful when you have a large suite of tests and a stub will not be sufficient because each test needs a different data set up and maintaining them in a stub would be costly. The mock lets us keep the data set-up in the test. A mock is normally written by a developer for personal use but it can be shared with testers. However, wider sharing is typically limited by interoperability issues related to software platform and deployment infrastructure dependencies that were hardcoded. They most often work in-process directly with classes, methods, and functions for unit, module, and acceptance testing. Mock provides responses based on a given request satisfying predefined criteria (also called request or parameter matching). A mock also focuses on interactions rather than state so mocks are usually stateful. For example, you can verify how many times a given method was called or the order of calls made to a given object.

Virtual service: a test double often provided as a Software-as-a-Service (SaaS), is always called remotely, and is never working in-process directly with methods or functions. A virtual service is often created by recording traffic using one of the service virtualization platforms instead of building the interaction pattern from scratch based on interface or API documentation. A virtual service can be used to establish a common ground for teams to communicate and facilitate artifact sharing with other development teams as well as testing teams. A virtual service is called remotely (over HTTP, TCP, etc.) normally supports multiple protocols (e.g. HTTP, MQ, TCP, etc.), while a stub or mock normally supports only one. Sometimes virtual services will require users to authorize, especially when deployed in environments with enterprise-wide visibility. Service virtualization tools used to create virtual services will most often have user interfaces that allow less tech-savvy software testers to hit the ground running, before diving into the details of how specific protocols work. They are sometimes backed by a database. They can also simulate non-functional characteristics of systems such as response times or slow connections. You can sometimes find virtual services that provide a set of stubbed responses for given request criteria, and pass every other request to a live backend system (partial stubbing). Similar to mocks, virtual services can have quite complex request matchers, that allow to have one response returned for many different types of requests. Sometimes, virtual services simulate system behaviors by constructing parts of the response based on request attributes and data.

It is often difficult to definitely say which of the following categories a test double fits into. They should be treated as spectrums rather than strict definitions.

All of the approaches mentioned above come with pros and cons. We will have a look at those later.

What is service virtualization?

It is the practice of creating virtual services and sharing them between developers and testers within a team and across teams. Developers and testers working on the same product can use the same virtual service artifacts or even virtual services. Another example is test teams across a large enterprise using the same virtual service artifacts. It promotes communication between development and test teams across many departments. It also attempts to address the problem of duplicated efforts by creating stubs for the same APIs within a large organisation by many teams simultaneously, by establishing new communication channels between teams. We will look at the trade-offs later.

Please keep in mind that the following comparison includes only general guidelines. Depending on the details of your situation solutions may vary.

Stubs, mocks and service virtualization solve different problems. Some problems can be addressed by both stubs, mocks and virtual services. Some problems should be addressed only with mocks and stubs. To highlight only a few of the most common concerns please have a look at the comparison below.

Examples of mock, stub & virtual service applications

Stubs, mocks, and virtual services are used to solve different problems. Here are a few example scenarios where each kind of approach could be considered. The list here demonstrates example usages and is not exhaustive in any way. 

Here are two example situations where a stub is likely to be a good approach:

• If you are a backend developer working on a small new application that uses a third party library to communicate with an external API, you can create an interface in your code to decouple yourself from that third party library. For unit tests, write a stub that returns hard coded values. For end to end testing, implement the interface using the third party library classes.
• If you are a tester who needs to test an application in isolation from the dependencies, create a stub for the HTTP RESTful APIs that the application depends on. Use in house tools provided by your team’s developers.

Now two examples situations where a mock is likely a good choice:

• If you are a back end developer working on a fairly large application and you need to decouple from any HTTP API dependencies, use a remote mocking framework such as WireMock or mountebank and prime the mocks in the acceptance tests. 
• If you are a back end developer working with a code base that has thousands of class unit tests then Mockito could allow you to mock the dependency classes in any tests.

Finally, four examples where a virtual service should be considered first:

• If you are a developer working on a very large and complex legacy application that has many dependencies and your current test coverage is very low (e.g. less than 5%)--especially if the system dependencies are unavailable more than two-thirds of the time. A service virtualization tool would allow you to record traffic when the dependencies are available, so that later, you can test using virtual services by replaying the traffic. This limits the impact of downtime of the dependencies for both you and your testers when you let them make use of the virtual services. This is a short term or tactical approach to give you time to create a suite of automated tests for your application that would not use recorded but primed data. How well this can be implemented depends on the nature of the system you are working with. It can be very easy or close to impossible to deliver depending on the details.
• If you are a front end developer working with a public SOAP (Simple Object Access Protocol) like Weather API, take advantage of Sandbox (getsandbox.com) or TrafficParrot (trafficparrot.com) to generate the virtual service using a WSDL file. Set up the test data in the virtual service so that all of your test cases are represented. For example, temperatures below ­-60°C (which are never returned from the real Weather API).
• If you are an NFT (Non-Functional Testing) tester responsible for the performance of a large banking application, use tcpdump to record incoming and outgoing traffic in a production environment for a day. Next import the outgoing traffic using a service virtualization tool and create a virtual service. You can import the incoming traffic to JMeter using tcpreplay and JMeter Proxy. Run a test suite in your performance environment using JMeter and use the virtual services to decouple yourself from other dependencies. Just to be safe, repeat the recording process every week. This could prove to be very easy or very difficult to implement depending on the specifics of the system you are working with.
• You are working in a microservices-like architecture. Your department employing 20 developers and 10 testers is responsible for delivering 70 small applications. Together, they form a product. Those applications are tested individually in isolation by any given team working on the application. Your CI (Continuous Integration) system is very well maintained. Most of the teams write Consumer-Driven Contract tests or follow similar practices and you do not encounter many integration issues within the department. Unfortunately, the environment outside the department is not that agile. The APIs you connect to are slow, often unavailable and it is very hard to set up test data there. To solve that problem each of the teams interfacing with third parties creates virtual services. Some teams call them stubs, or even different names they have invented. Those virtual services are used by the acceptance tests of a given application as part of the CI build. They are then used by testers on that team, as part of manual exploratory testing. Then, they get deployed as part of the main integration build that runs all applications within a given bounded context together. In the end, same virtual services are also used in full stack performance tests by the performance test team.

Resources for digging deeper

• A good overview of stubs and mocks can be found in the GooS book http://www.growing-object-oriented-software.com/.
• If you are a software developer, try using Mockito to learn how to do mocking.
• If you are a software tester try doing stubbing using any of the open source tools. A list of the tools can be found here: http://blog.trafficparrot.com/2015/05/service-virtualization-and-stubbing.html
• It's also worth reviewing any of the enterprise vendors like CA, IBM or Parasoft but as ThoughtWorks observed in Jun 2014 most of the innovation comes from practitioners contributing to open source.

About the Author

Wojciech Bulaty is a senior software developer working at WB Software Consultancy Limited. Wojciech brings a decades of hands-on-coding experience to his writing on Agile, automation, XP, TDD, BDD, pair programming and clean coding. WB Software Consultancy has consulted for firms like British Sky Broadcasting and Lloyds as well as several startups. Their most recent offering is Traffic Parrot, a software tool that supports stubbing, mocking and service virtualization.

References

1. http://blog.trafficparrot.com/2015/05/service-virtualization-and-stubbing.html
2. http://download.microsoft.com/download/3/A/7/3A7FA450-1F33-41F7-9E6D-3AA95B5A6AEA/MSDNMagazineSeptember2007en-us.chm
3. http://martinfowler.com/articles/consumerDrivenContracts.html
4. http://martinfowler.com/articles/microservices.html
5. http://martinfowler.com/articles/mocksArentStubs.html
6. http://wiremock.org/
7. http://www-01.ibm.com/software/rational/servicevirtualization/
8. http://www.ca.com/us/devcenter/ca-service-virtualization.aspx
9. http://www.jmock.org/
10. http://www.mbtest.org/
11. http://www.soapui.org/
12. http://www.soapui.org/getting-started/mock-services.html
13. http://xunitpatterns.com/Mock%20Object.html
14. http://xunitpatterns.com/Mocks,%20Fakes,%20Stubs%20and%20Dummies.html
15. http://xunitpatterns.com/SUT.html
16. http://xunitpatterns.com/Test%20Double.html
17. http://xunitpatterns.com/Test%20Stub.html
18. https://en.wikipedia.org/wiki/Service_virtualization
19. https://en.wikipedia.org/wiki/Service_virtualization#Tools_available_in_the_market
20. https://en.wikipedia.org/wiki/Software_as_a_service
21. https://github.com/mockito/mockito
22. https://www.parasoft.com/solution/service-virtualization/
23. https://www.thoughtworks.com/radar/platforms/big-enterprise-solutions