Marco Achtziger shared his experiences with deploying continuous testing in large scale agile project at Siemens Healthcare. At the OOP 2015 conference in Germany he talked about the pitfalls that they encountered and the approaches taken to overcome them.
InfoQ interviewed Achtziger about continuous testing and continuous integration, infrastructural and social challenges with continuous testing, testing processes and tools, and improving continuous testing.
InfoQ: Can you explain how continues testing relates to continuous integration?
Achtziger: Continuous testing is a part of continuous integration. In large projects the testing part of continuous integration can get so complex that it has to be dealt with separately. But the one cannot be without the other in my opinion.
InfoQ: In your session you mentioned some of the infrastructure challenges on continuous testing. Can you elaborate on this?
Achtziger: The tooling you use is very important. Especially when you are in a large project. As in the product software itself you also have to be aware of e.g. scalability or reliability of your infrastructure. So typical things to consider would be:
- Usability of the tooling. In ideal case the developers do not have to do anything additional that the tests are executed.
- Do you have to run tests in parallel? If yes does the infrastructure support it?
- Is your tooling able to scale-out? So can it handle multiple test machines and you can also increase the number of machines easily?
- What are the bottlenecks in your test execution? E.g. in our case if we would have used the default mechanism provided by the tooling not the test execution would be the problem but the deployment of the binaries to be tested would define the test execution duration.
InfoQ: You also talked about social challenges with continuous testing. Which problems have you experienced, and how have you dealt with them?
Achtziger: The main thing you will find here is the good old "works on my machine" developer. The software development processes before agile supported this mindset in my opinion (no continuous integration, tests running only after integration, …). Unfortunately there is no silver bullet how to deal with that. What you have to manage somehow is that the mindset of the developers changes. What helped in our environment is to establish what we call TRA (Tests run anywhere) principle. Makes it possible that you do no talk about a negative thing ("You are doing it wrong because it only works on your machine") by giving them something they can support ("Wouldn't it be great that all tests run anywhere?").
InfoQ: You mentioned that there are multiple stages in the testing process: Teams, assets and system integration. Can you describe the different stages and show how they work together in the testing chain?
Achtziger: It is simply an approach for splitting up your test base. An asset is for us an aggregation of several teams working in the same domain. So if you have such a structure you can break up your tests in that way that the team executes the unit tests and maybe already some integration tests. Next stage then executes mainly integration tests to ensure that the different teams did not influence each other and then as final integration there are all changes from all assets coming together in the so called system integration. There you can then decide if you want to execute tests based on the changes which are coming in or if you want to establish a static set of tests which has the whole system in scope and ensure that it is still working.
InfoQ: Which mechanisms do you use to select the tests that must be executed in different stages and build?
Achtziger: In principle we use simple XML files which are located in source control with every assembly we build. There it is configured what tests should be executed if something is changed in that area. So nothing fancy. It is very simple but it works.
InfoQ: You are using a test quarantine process for test which are failing due to bugs. How does this work?
Achtziger: First of all it should be clear that tests are not only failing due to bugs in the product code. Sometimes of course also the test code itself can be improved. But quarantine works in that way if a test fails sporadically you should check the reason for it. If it is a serious product bug then of course fix the bug. If the bug is not so serious, the problem is in the test code or it would be very hard to fix the test to get it back to a green state again you can decide to switch off this test for a while (e.g. use Explicit attribute of NUnit). That brings your builds back into a green state and they do not get spoiled anymore by a frequently failing test case. The amount of tests in quarantine has to be monitored and should be as close as possible to zero. So the whole quarantine is an exceptional thing to deal with sporadically failing tests to avoid that developers start to ignore the build results what would be even worse than the sporadic test failures.
InfoQ: If people are doing continuous testing and want to improve it, can you give some suggestions?
Achtziger: I think the first thing you have to do is always to talk to the users of the system. So talk to developers and find out what pain points they have and how you could support them in your tooling infrastructure. Other things which are always good to have a look at:
- Monitor execution time of your tests and try constantly to improve the slowest ones
- If a bug was not found, do a thorough root cause analysis and try to close the gap
- Look at your tests from time to time and delete not needed or redundant ones
If you follow these three things I would say the main points for continuously improving the testing are covered. Other things are normally organization specific and have to be covered there.
