In 2014, the State of California passed a law requiring makers of autonomous vehicles to submit “disengagement reports” to its Department of Motor Vehicles. A “disengagement” is any time when a human driver has to manually take control of the car from its computer driver for safety reasons.
A disengagement can be initiated by the self-driving car’s software itself (i.e., the car tells the driver it needs to take over) or it can be initiated by the driver, such as when they feel compelled to hit the brake or move the steering wheel. The deadline for the first reporting requirements was January 1 and covers a 15-month period from September 2014 to November 2015.
Google, along with a host of other autonomous-car makers, including Bosch, Delphi, Nissan, Mercedes-Benz, Tesla, and Volkswagen, have now filed those reports. The results probably won’t do much to assuage the fears of those who are unsure about the safety of autonomous vehicles.
Of the seven self-driving vehicles makers that filed disengagement reports, there were a total of 2,894 disengagements logged during the period. The most disengagements occurred with Google’s cars; a total of 341 over 424,000 miles of driving. Delphi reported 405 disengagement over 16,662 miles, while Nissan reported 106 over 1485 miles. Tesla reported no disengagements from the time it was issued a permit to test autonomous vehicles on California roads.
It’s important to note that conclusions shouldn’t be drawn about the reliability of each company’s vehicles by the stats above. These disengagement reports only cover self-driving cars that were driven on California roads and do not include disengagements on cars driven in other states. The reports also don’t take into account improvements in disengagement over time. For example, in January 2015, Google’s autonomous vehicles only drove 18,000 miles and had 48 disengagements. In October 2015, Google drove more 47,000 miles, yet had only 11 disengagements.
The reports can also be a little misleading in the automaker’s favor. Of Google’s 341 disengagements, 272 of those occurred when its cars detected an error, such as sensor problems or steering or braking issues, and handed control back to the driver. The other 69 times the driver took control of the vehicle, it was without the car’s computer thinking it was necessary; for example, when the driver tapped the break because they thought the car was going too fast (maybe it wasn’t).
However, Google says that its drivers actually took over the cars of their own accord “many thousands of times,” yet the company is not including those times in its numbers. That’s because Google argues that the California DMV only requires manufacturers to report when the driver was justified in taking control of the car. The company itself decides which manual disengagements were justified and which were not by replaying each disengagement in an online simulator.
In a statement accompanying its report, Google said:
Disengagements are a critical part of the testing process that allows our engineers to expand the software’s capabilities and identify areas of improvement. Our objective is not to minimize disengagements; rather, it is to gather, while operating safely, as much data as possible to enable us to improve our self-driving system. Therefore, we set disengagement thresholds conservatively, and each is carefully recorded.
For their part, other autonomous-vehicle manufacturers listed detailed examples of scenarios in which disengagements occurred. Delphi, for example, said disengagements occurred when drivers made way for emergency vehicles, and as “precautionary intervention to give extra space for a cyclist.” Nissan cited times “a route could not be generated due to a localization error” and when the car “did not recognize stopped vehicle in front of it. The driver overrode the system with manual brake input, causing the control to disengage.”