Waymo developed a testing framework called Collision Avoidance Test (CAT) to evaluate the ability to avoid crush or potential hazard situations of its Waymo Driver, compared to a human driver. CAT is a scenario-based testing framework, and the methodology is disclosed in this paper.
To understand how good the Waymo Driver is at avoiding crashes, its behavior is compared with the conduct of a reference model that is called non-impaired, with eyes always on the conflict (NIEON) human driver. This model is essentially a model of a human driver without distractions and fatigue. NIEON reaches a level of performance that doesn’t exist in the human population so it is a high benchmark with which to compare the Waymo Driver's performances.
Waymo develops and builds testing scenarios for CAT using information that came from police databases with data on crashes, public datasets on higher severity collisions, and experts' knowledge about geographic data, areas, driving consciousness, and road types. In addition, the Waymo scenarios are also provided by the miles driven by the Waymo test cars that continuously drive in public streets and allow the database expansions. All of this information is added to the Waymo Crash database, which covers almost all kinds of crashes, and allows the Waymo expert to develop new test scenarios. This database can be used as a baseline in all the cities because all the potential crash situations are the same independent of the cities.
The test scenarios are built first in a closed test track, such as the facility test truck of Waymo, and it is recorded in order to reproduce them in simulation. The tests on public streets are recorded too so, in case of a potentially hazardous situation, they can be added to the database. All these scenarios are then tested in simulation modifying speeds and other parameters to understand the Waymo Driver behavior without potential incidents with the "agents"; in the scenario. For potentially highly dangerous situations, for example for high speeds, a fully synthetic simulation is created. The simulation is fundamental for Waymo because it allows them to test all the software releases in thousands of scenarios with hours and not with months or years of testing.
Example of potentially dangerous situation
Waymo Driver collectively demonstrates better performances than NIEON for collisions, particularly for those involving potential serious injury. This benchmark shows that Waymo Driver avoids more crashes than NIEON in the simulated real-world fatal crashes in the Waymo test track.
Waymo CAT is one of the methods used to test the safety of Waymo Driver. It is designed to be continuously developed and the data are continuously added in order to have better testing scenarios. This method can be applied across of wide range of vehicle platforms, from passenger cars to trucks, and different driving environments.