Autonomous vehicles (AVs) have made headlines in recent months, though often for all the wrong reasons. Cruise, Waymo, and Tesla are all under U.S. federal investigation for a variety of accidents, some of which caused serious injury or death.

A new paper published in Nature puts numbers to the problem. Its authors analyzed over 37,000 accidents involving autonomous and human-driven vehicles to gauge risk across several accident scenarios. The paper reports AVs were generally less prone to accidents than those driven by humans, but significantly underperformed humans in some situations.

“The conclusion may not be surprising given the technological context,” said Shengxuan Ding, an author on the paper. “However, challenges remain under specific conditions, necessitating advanced algorithms and sensors and updates to infrastructure to effectively support AV technology.”

The paper, authored by two researchers at the University of Central Florida, analyzed data from 2,100 accidents involving advanced driving systems (SAE Level 4) and advanced driver-assistance systems (SAE Level 2) alongside 35,113 accidents involving human-driven vehicles. The study pulled from publicly available data on human-driven vehicle accidents in the state of California and the AVOID autonomous vehicle operation incident dataset, which the authors made public last year.

While the breadth of the paper’s data is significant, the paper’s “matched case-control analysis” is what sets it apart. Autonomous and human-driven vehicles tend to encounter different roads in different conditions, which can skew accident data. The paper categorizes risks by the variables surrounding the accident, such as whether the vehicle was moving straight or turning, and the conditions of the road and weather.

Level 4 self-driving vehicles were roughly 36 percent less likely to be involved in moderate injury accidents and 90 percent less likely to be involved in a fatal accident.

SAE Level 4 self-driving vehicles (those capable of full self-driving without a human at the wheel) performed especially well by several metrics. They were roughly 36 percent less likely to be involved in moderate injury accidents and 90 percent less likely to be involved in a fatal accident. Compared to human-driven vehicles, the risk of rear-end collision was roughly halved, and the risk of a broadside collision was roughly one-fifth. Level 4 AVs were close to one-fifthtieth as likely to run off the road.

The paper’s findings are generally favorable for level 4 AVs, but they perform worse in turns, and at dawn and dusk.Nature

These figures look good for AVs. However, Missy Cummings, director of George Mason University’s Autonomy and Robotics Center and former safety advisor for the National Highway Traffic Safety Administration, was skeptical of the findings.

“The ground rules should be that when you analyze AV accidents, you cannot combine accidents with self-driving cars [SAE Level 4] with the accidents of Teslas [SAE Level 2],” said Cummings. She took issue with discussing them in tandem and points out these categories of vehicles operate differently—so much so that Level 4 AVs aren’t legal in every state, while Level 2 AVs are.

Mohamed Abdel-Aty, an author on the paper and director of the Smart & Safe Transportation Lab at the University of Central Florida, said that while the paper touches on both levels of autonomy, the focus was on Level 4 autonomy. “The model which is the main contribution to this research compared only level 4 to human-driven vehicles,” he said.

And while many findings were generally positive, the authors highlighted two significant negative outcomes for level 4 AVs. It found they were over five times more likely to be involved in an accident at dawn and dusk. They were relatively bad at navigating turns as well, with the odds of an accident during a turn almost doubled compared to those for human-driven vehicles.

More data required for AVs to be “reassuring”

The study’s finding of higher accident rates during turns and in unusual lighting conditions highlight two major categories of challenges facing self-driving vehicles: intelligence and data.

J. Christian Gerdes, codirector of the Center for Automotive Research at Stanford University, said turning through traffic is among the most demanding situations for an AV’s artificial intelligence. “That decision is based a lot on the actions of other road users around you, and you’re going to make the choice based on what you predict.”

Cummings agreed with Gerdes. “Any time uncertainty increases [for an AV], you’re going to see an increased risk of accident. Just by the fact you’re turning, that increases uncertainty, and increases risk.”

AVs’ dramatically higher risk of accidents at dawn and dusk, on the other hand, points towards issues with the data captured by a vehicle’s sensors. Most AVs use a combination of radar and visual sensor systems, and the latter is prone to error in difficult lighting.

It’s not all bad news for sensors, though. Level 4 AVs were drastically better in rain and fog, which suggests that the presence of radar and lidar systems gives AVs an advantage in weather conditions that reduce visibility. Gerdes also said AVs, unlike humans, don’t tire or become distracted when driving through weather that requires more vigilance.

While the paper found AVs have a lower risk of accident overall, that doesn’t mean they’ve passed the checkered flag. Gerdes said poor performance in specific scenarios is meaningful and should rightfully make human passengers uncomfortable.

“It’s hard to make the argument that [AVs] are so much safer driving straight, but if [they] get into other situations, they don’t do as well. People will not find that reassuring,” said Gerdes.

The relative lack of data for Level 4 systems is another barrier. Level 4 AVs make up a tiny fraction of all vehicles on the road and only operate in specific areas. AVs are also packed with sensors and driven by an AI system that may make decisions for a variety of reasons that remain opaque in accident data.

While the paper accounts for the low total number of accidents in its statistical analysis, the authors acknowledge more data is necessary to determine the precise cause of accidents, and hope their findings will encourage others to assist. “I believe one of the benefits of this study is to draw the attention of authorities to the need for better data,” said Ding.

On that, Cummings agreed. “We do not have enough information to make sweeping statements,” she said.