California releases autonomous mileage reports car makers love to hate

Alphabet’sWaymo and General Motors-backed Cruise saw their autonomous vehicles require less help from human drivers on California's public roads in 2019.

Waymo drove 13,219 miles on average before a human safety driver had to take control of the vehicle and Cruise saw its autonomous cars drive 12,221 miles before the autonomous system was disengaged,according to data released today by the California Department of Motor Vehicles.

The autonomous vehicle disengagement reports arepublishedannually to monitorthe performance of self-driving systemsby many of the biggest companies pursuing this technology.But in recent years, it has drawn criticism from those very companies, who question the value of the data.

Two key statistics are published: total miles driven autonomously with no input from a human driver,and disengagements, meaning deactivation of the autonomous modeand/orintervention by a safety driver.Companies holding a permit totest self-driving technology on California roadsare required to self-report the figures to the DMV. It is a selective snapshot of how far companies’ autonomous ambitions have progressed, but it is the only standardized measure of this technology available to the public in the U.S..

In recent years, Waymo,Cruise,and startup Zoox, Inc.have led the way in autonomous testing, based on miles driven and miles per disengagement.Zoox reported doubling its public road testing to more than 60,000 miles last year, while its disengagement rate stayed roughly the same at about one every 1,700 miles.Apple Inc.drove around 118 miles per disengagement on average, up from just one mile per disengagement a year earlier. However, the company also significantly cut the number of autonomous testing miles. All told, the 62 companies reporting drove nearly 2.9 million autonomous miles during the reporting period.

Industry leaders, academicsand automotive analysts have questionedthe utility and reliability of the data,because the companiesoperate under varying circumstances. Some say itshould not be used to benchmark one company against another.

“We appreciate what the California DMV was trying to do when creating this requirement," Waymo posted on Twitter after the report’s release, “but the disengagement metric does not provide relevant insights into the capabilities of the Waymo Driver or distinguish its performance from others in the self-driving space."Kyle Vogt, the founder and chief technology officer of Cruise, in a January blog post described the report as“woefully inadequate for most uses beyond those of the DMV." He wrote:“The idea that disengagements give a meaningful signal about whether an AV is ready for commercial deployment is a myth." Cruise's miles per disengagement rose 134% in 2019; the company almost doubled the number of test miles.

The reports “are not intended to compare one company with another or reach broad conclusions on technological capabilities," said DMV spokesman Marty Greenstein. “Permit holders all have different goals and business models, and are testing in different ways, locations and conditions with different amounts of vehicles."What counts as a disengagement varies. Different testing environments yield different results, withmanydisengagementsoccurring in congested urban areas. Human intervention typically occursless frequently on highways and less populated areas.

“It comes down to standardization," saidPete Kelly, the managing director of researcher LMC Automotive. He believes policymakers should spell out more clearly the circumstances in which a safety driver should or should not takecontrol from the autonomous system. A better measure of the technology’s evolution will come only when legislators require more detailed reports, with fuller descriptions of the vehicle’sbehavior during disengagement.

Industry executives, including Cruise’s Vogt, don’t like that the DMV’s disengagement metric isusedto infer whether self-drivingtechnology is ready for commercial deployment. Vogtsuggests that more qualitative analysis of autonomous testing, such asreviewsof large volumes of raw video material, could better inform companies and regulators of the reasons fordisengagement.

Aurora Innovation Inc. chief executive officer Chris Urmson, a former headofGoogle's self-driving team, says that using the disengagement rates as a benchmark of progress ismisguidedand that what counts isthe quality of autonomous miles driven.

Aurora is moving away from collecting the data generated from testing on public roads. The company drove the majority of its 39,729 test miles on public roads in 2019, with a driver in control of the car; only 13,429 of those miles were fullyautonomous. Instead, Aurora has focused on simulated testing, runningmore than 750,000 tests in a virtual lab.

In a recentblog post, CEO Urmson wrote that abetter approach for the DMVwould be to join other state and federal agencies and regularly communicatewith the companies developing autonomous technology.“If the company or the public is measuring by disengagement rate, then you create this kind of odd incentive," Urmson said in an interview. “The drivers in the vehicles will be like,‘Oh geez, I don’t know if I want to just engageat that moment, because it’s a pressure.’"

Jim McPherson, a transportation lawyer specializing in autonomy, said some of the objections are valid, “but some of them also come across as disingenuous."McPherson believes companies are not adequately explaining the circumstances of each disengagement. California’s regulations do require companies to add written details—such as weather conditions, traffic conditions or surrounding construction—in enough detail to explain what triggered the disengagement.

At stake is billions of dollars in potential revenuetied to a change intraffic systems globally.BloombergNEF expects 27 million robotaxis on the road globally by 2040, while CruiseCEO Dan Ammann has claimed there will be a $1 trillion addressable market in the U.S. alone for autonomous ride hailing. Waymo, seen as a front-runner to deploy a commercial service first, has been valued by some analystsat more than $100 billion.

Cruise does all of its California testing in San Francisco, a city of rolling hills, complex road systemsand multiple vehicle typessuch as cable cars. Other companies like Waymo, Zoox and Aurora are testing both in San Francisco and in more suburban settings such as Mountain View or on the freeway, as well as in additional states, where regulations are different. For example, Arizona and Florida have become hubs for testing, in part because they don’t have disclosure requirements. Waymo operates a small autonomous ride service in suburban Phoenix for around 1,500 paying users. The vehicles usually have human safety drivers behind the wheel. In California, the California Public Utilities commission is running a more limited pilot,with five companies giving rides to the public but not being able to charge fares.

One notable companymissing from the pack of leaders in the DMV reports is Tesla. The automaker is working towarddeploying fully autonomous vehicles by the end of this year, havingsaid it plans to issue a fully self-driving version of its Autopilot feature. The deadline initially was set for the end of 2019. CEOElon Muskalso said Tesla customers would be able to put their cars into a shared robotaxi network this year, but clarified recently that this will depend on regulatory approval.

Tesla, whose valuation has swelled on optimism of its technological lead, holds a California DMV testing permit but does not actively run autonomous systems on California’s public roads. The company says it drove a single 12-mile autonomous journey around the perimeter of its Palo Alto headquarters in April 2019; it did not report any disengagements. It is developing full self-driving by using data gathered from its vehicles globally and using the data to build and train neural networks.

“The winner in an AI project has the most data and the highest-quality data,"Cathie Wood, the CEO of Ark Investment,said recently on Bloomberg Television.