Bosch is doing automated regression testing and user testing using a real car instead of a simulated one. Their aim is to test the software as quickly as possible, both from the test engineer’s and user’s perspectives. The car can be accessed remotely, and team members can work without being in the car.
Pavel Hubenov, senior test engineer at Bosch Bulgaria, shared experiences from remote verification and validation of self-driving vehicles at QA Challenge Accepted 2021.
The continuous integration testing process is based on an automated system that builds software every night and runs tests to monitor regressions, as Hubenov explained:
We have partial validation every night based on the last implemented changes. In detail, it means that we are testing during the night affected features for regressions based on predefined priorities. It is possible every night configuration of the test set to be different. The results are dozens of reports for analyzing. Another fully automated system with quite complicated criteria which reads, filters, and analyzes, finally makes a conclusion for the whole test execution and all team members are informed in the morning.
Bosch is following the traditional V cycle with the exception that they have included a real car in the last two waterfall levels of testing:
To ensure that our device will work the same way in a real car and on the Hardware-In-The-Loop bench, we realized that we must include a real vehicle in our continuous testing process.
In addition, they organize additional testing sessions from the user’s point of view during the day. Hubenov mentioned that they use vehicles with different equipment levels from a commercial point of view to ensure all model modifications work the same way. One of them is their "hero of the day" which is included in the automated process, Hubenov said.
The car is a special prototype car. It is not allowed to be driven on the road but only used in specially approved parking places, Hubenov mentioned. It has all the necessary terminals to monitor whole communication.
Having a vehicle modified as a test bench, they were able to face challenges which global pandemic of Covid-19 brought to them, as Hubenov pointed out:
Having this unique bench gave us the flexibility not to stop working and to continue delivering the excellent quality our customers deserve. To mention: access is only from within the Bosch internal network and is given only to well-trained people from the project team.
InfoQ interviewed Pavel Hubenov about testing with a remote self-driving vehicle.
InfoQ: What challenges did you face in verifying and validating self-driving vehicles?
Pavel Hubenov: The challenges which we face are not only related to self-driving vehicles. Automotive is a very sensitive sphere, as avoiding possible mistakes may save a human life. With the increasing complexity of the functionality and in-vehicle software itself, test engineers can ensure excellent quality and safety of the product.
In Bosch Engineering Center Sofia, we realize that having highly skilled test engineers is a must, but it is not enough. We also work on the mindset of the test engineers – to give their best to achieve premium quality on each level of v-cycle following the standards and the company processes and not to forget that quality in every aspect is a must.
InfoQ: How did you automate and upgrade your testing with an implemented real car instead of a simulated one?
Hubenov: Based on simulation testing, we decided to reuse tests for the vehicle. We redesigned and adapted tests to run in the car expecting well-known results. For example, setup on the Hardware-In-The-Loop simulation is configured with the real gateway device and surrounded by simulated virtual nodes representing real ECUs from the vehicle. The testing concept on HIL is to send requests to the device and wait for responses in a specified time interval. But, in the real situation in the car, real devices are sending these same requests and waiting for related responses. So, the test is monitoring based on those real requests, and waiting for the triggered responses, without real events from our side which may toggle changes into the system.
In conclusion, for this kind of testing, the test itself is not a direct action maker, but just an events handler, synchronized with the transmitted events and measuring expected time interval for responses. To control the whole system or to affect the system communication, we are only using physical buttons of the car and the keyless remote control with specially-developed and implemented hardware equipment.
InfoQ: How is remote testing done?
Hubenov: In the automotive domain, due to the enormous volumes of tests that we execute for every delivery, we highly rely on automation testing. Manual testing is used mainly for defects reproduction and as an additional activity to deliver better quality. We rely on an automated system for nightly testing to ensure stability and non-regressions. Testing in Automotive is a 24/7 activity, and people might get tired; they need rest, but test benches can work instead of people all the time.
Remote access is allowed only for team members, no matter the location. Every team member has permission to book a slot for car usage and to run manual tests or automated ones. Sometimes, when a critical issue is to be reproduced and fixed as soon as possible, the agenda is paused, and team leaders are the decision-makers. Cases like this are processed with collaboration between developers and validation members. They can control the vehicle with a specially-dedicated pc and hardware designed for the robotics industry, and adapted for our prototype vehicle.
InfoQ: What have you learned on your test automation journey?
Hubenov: I understand that when automation is a team effort, everything is possible. I am happy to share that I had my team behind my back and colleagues from other departments during my journey. We are about 450 highly-skilled engineers in Bosch Engineering Center Sofia, and I know that I can rely on each of them when I need help or advice.
Our management highly supports this friendly and positive environment and knows that we all are one team. I can share that I see one of my dreams come true. I can start looking for new challenges which might look quite impossible or unrealistic. I believe that we, as a team, can find a technical solution for every idea, even if it seems crazy at first glance. Every proactive suggestion is discussed, estimated and accepted or postponed by the management based on project priorities and needs. Since the process is dynamic, we need quick feedback to give an idea of ours new life, and they know it perfectly. Communications between members and leads on all levels are inspiring and positive. The whole family has the same futuristic vision in achieving our goals.