Advanced driver assistance systems can have a huge number of test cases. Cutting the elephant into smaller pieces can ensure every bit and piece is tested. A good test environment is essential to be efficient, fast and to have the flexibility to cover all required tests to ensure quality.
Achim Noller, Department Head at Robert Bosch, spoke about testing driving assistants at QA Challenge accepted 2022.
Advanced driver assistance systems (ADAS) have 20-30 million lines of code with high security and safety demands. Customers expect a high system availability and low false positive rates, Noller said. The software has to be tested and traceability has to be ensured for all the artifacts:
Our approach is to cut this huge elephant into smaller pieces, keeping the whole elephant in mind. We structure the complete system in different architectural levels up from the whole advanced driver assistance system, down to a unit of software, and align tests with a test-strategy which links back all tests to a possible system release.
There is a common testing approach for all ADAS parts, but it has to be as flexible and modular as possible to enable getting synergies between the different units and components:
For example, reuse is possible with diagnostic features or on bus communication. Test cases can be reused as these are standardised customer requirements. On the other side, different features needs different tools for simulation
Noller mentioned that it is very important that testers are involved in the project right from the beginning, in the acquisition phase, and in customer discussions. This ensures that requirements will be testable without endless discussions for corrections. Testers can really make a difference, announce problems at an early stage, and hence avoid task-forces, quality- or delivery problems, Noller stated:
It´s not always easy to announce all these things to your counterparts in the project. But I am sure everybody will see the benefit of testing and the big advantage if testing is linked very closely to project management and engineering.
InfoQ interviewed Achim Noller about testing driving assistants.
InfoQ: How do advanced driver assistance systems look?
Achim Noller: Advanced driver assistance systems (ADAS) include a lot of sensors like Radar, Video, Lidar or Ultrasonics, which enables the system to generate a 360° surrounding model. This are the "eyes" of the car. The central computing unit is the "brain" of the car, where the different driving functions are calculated, sensor inputs are analysed and actuator outputs generated. The actuators enable to steer, break and accelerate the car to offer functionality. There need to be redundancy in all levels to ensure safety requirements for higher automation levels.
InfoQ: What are the challenges of testing advanced driver assistance systems?
Noller: We have to follow up on thousands of requirements on the one side, on the other side we have to stress corner cases of the design, e.g. test in different temperature and voltage levels to see if there are bugs that occur sporadically. Because if there is a bug in the system that can occur, it will occur in millions of vehicles later on.
InfoQ: How do you test such systems?
Noller: The test strategy covers all requirement-based testing. We also perform tests to find errors that are occurring sporadically and cover special test cases for safety and security, such as penetration Tests.
We look on a system level that our functionality is working in the right way, and also try to ensure to have as less as possible "false positive" events. This means that a function is triggered by default. Especially in case of an automated emergency break this can for example lead to very critical cases in traffic and even to accidents, which we have to avoid.
InfoQ: How does the test environment look?
Noller: We have to be flexible with different simulation environments, as some of them have strengths and weaknesses in different areas. There are for example different environments for parking features and driving features. Quite different speeds, dynamics and corner cases where we have to use different simulation tools. Our framework allows using them in a modular way. It is important to establish a continuous testing pipeline to get fast feedback and fast results.
InfoQ: What’s your advice to testing managers and testers?
Noller: Have close collaboration with engineering and project management. Open the mind of your counterparts about what big asset is there with testing. There is no one else than testers having a better overview of the system and the project status. Starting from requirements down to bugs in the software, testers know usually where are weaknesses and problems in the project.
