Do regulators — and the public — have reason to trust the safety and reliability of the AVs that are already on the public streets and roads in various U.S. cities? Truth is, we don't know enough to start trusting.
Autonomous vehicles (AV) are already on the public streets and roads in various U.S. cities. Do regulators — and the public — have reason to trust the safety and reliability of these vehicles? We aren’t being given enough information to start trusting any company developing AV technology or AVs themselves, largely because the agency responsible for auto safety is giving tech companies and AV automakers a free pass from scrutiny.
The only “safety” information available to regulators and other interested observers are reports submitted by the companies developing the vehicles to the National Highway Transportation Safety Board. They can be found on NHTSA’s web page as “Voluntary Safety Self-Assessment” (VSSA) documents.
Bowing to the mantra of “innovation,” NHTSA places no obligation on companies to report any particular data, let alone data in standardized formats. (NHTSA, however, does offer the Voluntary Guidance and the VSSA Template.) A VSSA, as indicated by the name, is a voluntary self-assessment. Companies aren’t even obligated to file a VSSA. Many don’t bother.
This marks a stark contrast compared to drug suppliers and aircraft manufacturers who are required by the Federal Drug Administration (FDA) and Federal Aviation Administration (FAA) to prove conformance to exacting industry standards before they roll out their products.
To that point, Colin Barnden, lead analyst at Semicast Research, told us: “The FDA does not permit new drugs to be developed on random passers-by, neither does the FAA permit experimental aircraft designs to be tested over densely populated areas.” He stressed, “Endlessly driving the same routes with nothing bad happening and no one getting killed doesn’t prove the safety case of a new technology. NHTSA (and state regulators) have a responsibility to ensure the safety of the public, not to act as enablers of experimental technology being tested on non-consenting subjects.”
In a bubble
By all counts, the AV industry in the United States lives in a bubble.
First, AV manufacturers effectively can self-certify their autonomous vehicles using their own standards and procedures. Second, they are under no obligation to file a VSSA. Third, some AV developers even fail to mention in their report any safety-relevant industry standards, including ISO 26262 (functional safety), ISO 21448 (Safety of The Intended Functionality) and ANSI/UL 4600. Even if they happen to cite these standards, they tend to do so only in passing.
A small percentage of companies opt to file the reports. As of April 28, 2021, 55 companies had obtained permits to test autonomous vehicles (monitored by a driver) from California’s Department of Motor Vehicles. Of those 55, only 15 companies have thus far filed VSSAs with NHTSA.
There have been a total of 24 VSSA documents filed. There are two reasons for the difference in numbers: not all companies that are testing in the U.S. are testing in California, and there are two instances in which a single company has filed two VSSA reports.
EE Times examined every VSSA filed by AV developers on the NHTSA website. Each company talks about “saving lives.” They state that “winning the public trust” is important. Each company says “transparency” is its goal in issuing the safety report.
Since this sort of self-congratulation lacks rigor, we decided to create a scorecard to discern, if possible, what each company that took the time to file VSSA really means by “taking safety seriously.” Our examination placed heavy emphasis on companies’ approach to industry standards such as ISO 26262, ISO 21448 or ANSI/UL 4600 — because each standard has a role to play and all are applicable to any autonomous vehicle. (Editor’s note: See the scorecard in the table below on this page)
On one extreme of the stream of 24 safety reports, three didn’t mention any industry standards at all. These were Zoox (now a part of Amazon), Local Motors and Apple.
On the opposite end of the spectrum were European carmakers BMW, Mercedes-Benz and Bosch. They are heavy promoters of industry standards.
Among the many variations between these extremes, numerous VSSA filers mentioned ISO 26262 but avoid claiming that their vehicles conform to the standard.
As Philip Koopman, co-founder and CTO of Edge Case Research and professor at Carnegie Mellon University, noted, “Many of the VSSAs do a sort of standards roll call without really committing to do anything specific in a mentioned standard. Vague statements that amount to picking and choosing pieces of standards don’t really tell us what companies are doing about safety.” He calls it “#ISOwashing.”
Waymo, for example, made no standards reference in its September 2020 Safety Report. Boosting its System Safety Program, Waymo stated: “As the first company to complete a fully self-driving trip on public roads in 2015, we have written our own playbook at Waymo.”
In an accompanying document, “Waymo’s Safety Methodologies and Safety Readiness Determinations (30 pages)” published in October 2020, the company cited a host of standards and explained why they are not entirely relying on them.
On ISO 26262:
ISO 26262 has provided significant insights for Waymo’s hazard analysis processes. However, Waymo does not rely strictly or exclusively on ISO 26262’s principles, which are not a perfect fit for a Level 4 ADS (Automated Driving Systems).
Asked to unpack this, Koopman said, “It’s important when interpreting statements like this to consider what it says, and what it doesn’t say.”
Koopman translated: “Waymo appears to say they are mindful of the section of ISO 26262 part 3 that covers Hazard and Risk Analysis. However, because that’s not a perfect fit they do not rely on it. Unsaid is whether they have considered many other parts of ISO 26262.”
He added, “Also unsaid is what principles they actually rely on, since they don’t seem happy with those in the well-established ISO 26262 functional safety standard.”
He summed up: “The reality is that ISO 26262 should apply to many aspects of their vehicle, as it does to any other vehicle. Since ISO 26262 is tailorable, it’s hard to understand why they don’t adjust it to fit their needs.”
Then, there’s the case of Argo AI. In the company’s recently issued Safety Report, it wrote: “Our approach to systems engineering is built around two key ISO standards: ISO 26262 and ISO 21448.”
Koopman said, “While the sentiment seems well intentioned, a reader has no way of knowing what this means in concrete terms.” He acknowledged, “It certainly gives the impression that they conform to both standards, but why wouldn’t they say ‘conform’ to it if that were true? I’d think they would make the strongest true claim they can about safety, and that would be either ‘conform’ or the less buzzwordy ‘follow.’ If they make a weaker claim than either of those two words, they shouldn’t expect to get credit for a stronger claim.”
EE Times contacted Argo AI to clarify the company’s statement. A spokesman told us, “Our intention is to conform although our safety case is not complete yet and that’s why we wrote the statement in the way that we did.”
Is there something about the existing standards that makes AV industry players believe that they are not relevant to the “autonomy” they seek in their L4 and L5 cars?
Koopman made it clear: “ISO 26262 addresses functional safety and deserves a place in an L4/L5 vehicle. What you need for L4/L5 is additional coverage for Safety of the Intended Function (SOTIF / ISO 21448), and system level safety (ANSI/UL 4600). You need all these pieces to be covered.”
With no oversight, no verification and no validation currently required in the testing of autonomous vehicles, a fundamental question — how to ensure the safety of autonomous vehicles before they proliferate on the road — becomes a riddle inside a conundrum.
Despite the existence of competing and misaligned AV standards, “A solution to this problem exists: embrace the approach already taken by experts — scholars, engineers, automakers and industry representatives — who came together to develop a technology-neutral safety standard,” recently wrote Jack Weast, Intel Fellow and chair of the forthcoming IEEE P2846 standard (Assumptions for Models in Safety-Related Automated Vehicle Behavior).
Weast isn’t alone.
Given that the industry itself spent significant engineering resources to create these standards, Koopman asked: “Why wouldn’t they follow them?”
Among the all the AV companies filing VSSAs, Nvidia’s presence on the list struck as a mild surprise. This suggests Nvidia’s ambition to develop an AV platform (complete with its powerful SoC and AV software stack) that automakers can simply pick up and drop into vehicles for an AV launch.
In its safety report, Nvidia wrote: “The Nvidia Drive architecture enables vehicle manufacturers to build and deploy self-driving cars and trucks that are functionally safe and can be demonstrated compliant to international safety standards such as ISO 26262 and ISO/DIS 21448, NHTSA recommendations, and global NCAP requirements.”
We found the GPU giant’s statement to be a little problematic. Nvidia’s bow to ISO 26262 does not mean that an AV containing Nvidia’s chip will necessarily conform to ISO 26262.
Koopman confirmed, “Yes, ‘enables’ means it’s up to the customer to actually conform.” He explained that for example, that if Nvidia sells an approach that lets a client conform to 21448 “in simulation,” begs the question of whether the final product meets 21448 in the real world.
Koopman added, “Beyond Nvidia, we’ve seen scattered claims of ISO 26262 conformance, but they are really only talking about the chips, and not even all the hardware.”
Safety Reports are marketing brochures
Reading 24 safety reports conveys the inevitable impression that these are, essentially, marketing brochures.
Barnden noted that they are “completely interchangeable. If I was sent the raw text from, say, the Motional, Waymo and Zoox brochures with the company name removed, I couldn’t tell them apart.”
The issue isn’t just about the reports’ surfeit of glossy pictures. There is a lack of technical substance. Barnden said, “It wasn’t clear to me what metric could measure progress made in the last 12 months, nor what information would materially change in a new safety report issued next year. Endlessly repeating the word ‘safe’ doesn’t, in itself, make a product or process safe.”
Take Waymo. Barnden said, “In 2020, Waymo reported to the California DMV 21 disengagements over 628,839 miles driven, a rate of about one disengagement every 30,000 miles. But as we’ve seen in a recent video, a ‘fully driverless’ Waymo vehicle was completely bamboozled by a single static traffic cone. At one point, the [AV] reversed on the highway and entirely blocked a lane, forcing human drivers to steer around it. This took place in Waymo’s primary testing area around Chandler, Arizona. In fact, disengagement reports are meaningless. These vehicles are clearly not yet safe enough to be operated on public roads without a trained human safety driver.”
Barnden was referring to footage of the Waymo Driver, by YouTuber JJRick, who rode as a customer in a Waymo driverless taxi in Chandler.
A case of Waymo
Considering that Waymo has been on the road since 2015, it is surprising that its robotaxi is still confused by traffic cones – which hardly seems to qualify as an “edge case.”
In recent months, Waymo, once perceived as a clear AV industry leader, has lost six high-level executives, its CEO, CFO, treasurer, head of manufacturing, chief safety officer and system safety chief. Such personnel exodus has, understandably, stirred concerns and energized the Silicon Valley gossip mill.
Which bring us back our original point. How much do we really know about the progress the Waymo Driver has made? Waymo’s Safety Reports are notably unrevealing, leaving observers to rely on a YouTuber as a key source to the Driver state-of-the-art. Not exactly reassuring.
Time is also overdue for many AV developers to come clean about how long it will be before they can launch fully autonomous vehicles. The safety reports should include not just their safety claims but challenges they are facing and how they plan to deal with them.
Kodiak is one company professing that higher test mileage does not necessarily equal more safety. In its safety report, the company wrote: “Of course, this disciplined approach means, we will probably never log as many test miles as some of our competitors. We see our lower mileage count not as a risk, but as a sign of our commitment to safety.”
Aurora agrees. “…we treat real-world testing as a mechanism for validating and improving the fidelity of more rapid offline testing. This strategy has allowed us to contain the size of our on-road testing fleet. We limit the distance our test vehicles travel by pursuing mileage quality over quantity; that is, we seek out interesting miles rather than just pursuing large quantities of miles.”
On the importance of edge-case research, Koopman noted that the problem isn’t just about the frequency of surprises — unsafe events — that need a fix. What matters even more is what the fix population looks like. Surprises are all different. If there’s a huge population of problems, it will take a very long time to make progress fixing them all, said Koopman. Most likely you’d never get there.
Koopman wrote in his paper:
Creating safe autonomous vehicles will require not only extensive training and testing against realistic operational scenarios, but also dealing with uncertainty. The real world can present many rare but dangerous events, suggesting that these systems will need to be robust when encountering novel, unforeseen situations.
Generalizing from observed road data to hypothesize various classes of unusual situations will help. However, a heavy tail distribution of surprises from the real world could make it impossible to use a simplistic drive/fail/fix development process to achieve acceptable safety.
Autonomous vehicles will need to be robust in handling novelty, and will additionally need a way to detect that they are encountering a surprise so that they can remain safe in the face of uncertainty.
NHTSA’s laissez-faire approach to the auto industry is notorious. For decades, the agency’s interest has squarely aligned with automakers and tech companies.
The Biden Administration has given no clue about its approach to safety issues in autonomous vehicles. Will new secretary of transportation Pete Buttigieg steer NHTSA into a new direction and, for example, make tech companies and carmakers’ AV safety claims more transparent and accountable?
The first clue might come when NHTSA responds to public comments it collected in Advance Notice of Proposed Rulemaking (ANPRM) on autonomous driving systems issued by the previous administration. The comment period ended on April 1st.
Asked about its timeline, NHTSA told EE Times that it might answer after “reviewing public comments received in response to the ANPRM regarding Automated Driving System (ADS) Safety Principles.”
The agency will then determine its next steps. “NHTSA’s regulatory plans for the next 12 months will be published later this spring at the Semi-Annual Unified Agenda of Regulatory Actions,” she said. That has yet to be released.
One catastrophic event away
Many AV industry observers are aware that the fledging industry is only one news event away from the government finally deciding to enforce stricter rules.
As Barnden noted, “If an AV test-level vehicle kills a single pedestrian or a child, that leads straight to President Biden.” He noted that the entire Boeing 737 Max fleet was grounded by the FAA when fundamental flaws in the design of the Maneuvering Characteristics Augmentation System (MCAS) were exposed. “The safety questions here are clear, so why the free pass for AV development? It is untenable for an industry to wail ‘saving lives’ while actually endangering them. The political risks are enormous and the Administration needs to wake up and show decisive leadership right now.”
There is already a precedent. When Elaine Herzberg was killed by an Uber test AV, the accident prompted the SAE to develop SAE J3018, guidelines for safe on-road testing of SAE Level 3, 4, and 5 prototype Automated Driving Systems.
A year after the Uber’s fatal accident, an invitation-only auto industry group, the Automated Vehicle Safety Consortium (AVSC), started to “document and make publicly available best practices associated with in-vehicle fallback test drivers based on the type of measures and processes the members use,” according to Ed Straub, director of the SAE Office of Automation. He described the AVSC’s mission as “generating public trust in SAE L4 and L5 autonomous vehicles.” All AVSC members are doing on-road ADS testing for various applications.
But here’s the thing: when AV companies started seeking from cities and states permission to test drive AVs on public roads, why didn’t local regulators demand that AV operators conform to the J3018 standard?
Straub said, “I can’t speak to the minds of different states and cities.” He speculated that awareness of the standard might not have trickled down to “right people who could potentially use it as a reference.”
But he also defended the auto industry’s position: “SAE International standards are voluntary and have been for over a hundred years, unless explicitly cited in regulations such as the FMVSS.” He pointed out, “The technology and testing associated with automated driving changes at a pace we haven’t seen before. Because of this, it can be very difficult for open industry standards to keep pace… Because voluntary industry standards invite all interested stakeholders to develop them, they can take a longer time to develop. Regulations that would ‘require’ compliance (like FMVSS) can take even longer.”
Seriously, though, what sort of industry spends time and resources developing a safety standard for all its members but shows no interest in enforcement? How is it that rules of the road apply to everybody but the people who make the cars?
The table below was compiled by EE Times. The Glossy Pix Ratio (GPR) is the ratio of the number of pages in the report devoted to promotional photography against the number of pages devoted to text. A report with a higher number of pages and a high GPR might therefore contain less information than a report with fewer pages but a lower GPR.
The next four columns count the number of times any given report includes the word “standards” or mentions a particular standard. The VSSA Score is an evaluation of the VSSA.
(This table is presented in sections; double-clicking on a section will render a larger view of that section.)
This article was originally published on EE Times.
Former beat reporter, bureau chief, and editor in chief of EE Times, Junko Yoshida now spends a lot of her time covering the global electronics industry with a particular focus on China. Her beat has always been emerging technologies and business models that enable a new generation of consumer electronics. She is now adding the coverage of China’s semiconductor manufacturers, writing about machinations of fabs and fabless manufacturers. In addition, she covers automotive, Internet of Things, and wireless/networking for EE Times’ Designlines. She has been writing for EE Times since 1990.