Testability must be explicitly designed in the system said Peter Zimmerer from Siemens AG. Test architects should drive testability and collaborate with architects, designers and testers in using good design and engineering practices.
At the QA&Test 2014 conference Peter gave a tutorial about design for testability for embedded software systems.
Peter defines testability as “the degree to which a system can be tested effectively and efficiently”. Effectiveness has to do with increasing the depth and quality of tests, where efficiency deals with reducing cost, effort and time of testing. Testability is the ease of validation, the degree to which software can be effectively tested. It plays a role during the initial development of the software and during maintenance, testability can be considered as a capability that enables modified software to be validated.
According to Peter the main factors that influence testability are:
- Control(lability):The better we can control the system (in isolation), the more and better testing can be done, automated, and optimized.
- Visibility / observability: What you see is what can be tested. Ability to observe the inputs, outputs, states, internals, error conditions, resource utilization, and other side effects of the system under test.
Peter stated that testability is often a better investment than automation. Automation also depends on testability, when a systems is designed to be testable then the effort needed to automate testing will also be lower.
Why is design for testability important and how can you convince managers to invest in it? One of the main benefits is that it reduces the cost, effort and time for debugging, diagnosis and maintenance during the full software development lifecycle. Peter referred to a survey that Stefan Jungmayr did in the German test community which is described in Testbarkeitsfaktoren und Testaufwand: Auswertung dreier Umfragen; one of the conclusions from the presentation testbarkeitsanforderungen an die Software is that testability can save around 10% of the total development budget.
Design for testability has to be done by architects, developers and testers jointly. Architects are open to designing in testability said Peter. Testability is a design criteria and testers must define testability requirements. In agile testability is a responsibility for the whole team but it is good to have a specific person like a test architect who is driving testability.
In his presentation Peter provided a checklist for designing for testability. This checklist can be used to discuss how good a team or project is addressing testability and what can be done to improve it:
- Suitable testing architecture, good design principles
- Interaction with the system under test through well-defined control points and observation points
- Additional (scriptable) interfaces, ports, hooks, mocks, interceptors for testing purposes (setup, configuration, simulation, recovery)
- Coding guidelines, naming conventions
- Internal software quality (architecture, code)
- Built-in self-test (BIST), built-in test (BIT)
- Consistency checks (assertions, design by contract, deviations)
- Logging and tracing (AOP, counters, monitoring, probing, profiling)
- Diagnosis and dump utilities, black box (internal states, resource utilization, anomalies at runtime)
- Think test-first (xTDD): how can I test this?
Designing for testability can be done by using good design practices. That is why the architect plays an important role in this said Peter. Doing good agile means doing agile engineering practices. Peter mentioned the Clean Code Developer Wiki which contains principles and practices for better software.
A strategy for design for testability needs to cover requirements, testing and architecture. Testability needs to be consistently defined and well known by everybody involved, handled in a risk based testing strategy and balanced with non-functional requirements. Testability guidelines can be used to specify testability for and embed testability in design. Milestones and quality gates need to have criteria on testability and testability has to be investigated and explored in both static and dynamic testing.
Peter concluded his tutorial by stating that “neglecting testability means increasing technical debt”.