Credible claims of safety require proof. It won't cut it for companies such as Waymo, Uber or GM to simply declare their vehicles are safe
Arguing credibly about the safety of any autonomous vehicle (AV) requires somewhat more proof than a company like Waymo, Uber or GM declaring that its robocars are safe for commercial deployment. The automaker must be able to demonstrate that its AI-driven vehicles meet specific and rigorous standards.
The next questions then become:
- Which safety standards?
- Who’s developing the standards?
- How are vehicles being tested?
- Who’s going to “assess” the safety of the AVs?
- Do we trust the assessors?
There are no easy answers. Further, safety standards, especially for AI-driven vehicles, come with additional twists and numerous caveats.
Put simply, AI-based autonomous vehicles are running on machine-learning algorithms housed in black boxes. The inner workings are probabilistic in nature, and it’s almost impossible to determine why they make whatever decisions they make.
This being so, what strategies do tech companies and car OEMs use to verify their AVs’ safety?
- Will the public accept that the traditionally self-certifying automotive industry certifies the safety of its AVs all by itself? Someone might remember the bang-up certification job done by the FAA (which is apparently less than independent than previously thought) on Boeing's 737 Max MCAS system.
- Can any safety standards keep up with rapidly evolving algorithms and software code deployed by self-driving cars?
- Isn’t it prudent to think that AV safety standards might be short-lived in usefulness and destined to become obsolete too soon, too often?
UL 4600 draft standard
Against this backdrop, Underwriters Labs, currently developing a “Standard for Safety for the Evaluation of Autonomous Products” — UL 4600 — said the members on its Standards Technical Panel (STP) met in person for the first time on June 12 and 13, to review and discuss the initial draft standard.
EE Times last week caught up with Phil Koopman, co-founder & CTO at Edge Case Research, a principal technical contributor to the draft.
The minutes of the first meeting are yet to be made public. Koopman, however, described the first meeting as “very positive and constructive. “ He said the members hit all the main issues of UL 4600 that must find solutions.
Who are on the roster?
According to the UL website, the UL 4600 group lists some 30 or so STP members with voting rights. They include four chip vendors — Nvidia, Renesas, Intel and Infineon — and commercial AV users and developers, among them General Motors, Uber, Nio, Bosch, Argo AI and Aurora. Both the U.S. Department of Transportation (DoT) and Pennsylvania DoT are sending representatives.
Interestingly, the UL 4600 STP also includes three insurance companies: AXA, Liberty Mutual and Munich Re America.
In pursuit of transparency and broader inclusion, the UL 4600 group is seeking any responsible party willing to be registered as a “stakeholder.” Once registered and approved, stakeholders can request to review and comment on the draft standard. While stakeholders have no voting rights, adding them is important, explained Koopman, to ensure “a very open procedure” and to signal that AV safety “is a matter of public policy.”
UL 4600 vs. ISO 26262 and ISO/PAS 21448 (SOTIF)
UL 4600 is a relative newcomer to automotive safety standards development. ISO 26262 already exists, while ISO/PAS 21448 (safety of the intended functionality or “SOTIF”) are well into development. The most frequently asked question about UL 4600 is why the automotive world needs yet another standard?
Stressing that the UL 4600 group is closely in touch with leaders in ISO 26262 and ISO/PAS 21488, Koopman made it clear that “resolving potential overlap is an ongoing activity.”
Developing safety standards based on the assumption that systems will have no responsible human driver is what separates UL 4600 from other standards.
In contrast, “existing standards such as ISO 26262 and ISO/PAS 21448 were envisioned for vehicles that ultimately have a human driver responsible for safe operation of the vehicle,” Koopman noted. In his opinion, the technology in robocars and other autonomous systems “exceeds the scope of these and other traditional safety standards. Those standards are necessary, but not sufficient.”
Did you think of that?
In other words, in developing self-driving cars, Koopman believes automotive design engineers will soon discover numerous issues they hadn’t even thought about before. Speaking of “the pervasive implications of vehicles not having a responsible human driver,” Koopman put those items under the rubric of “Did you think of that?”
Expect the UL 4600 to be much more prescriptive compared to other standards.
While ISO 26262 and SOTIF provide safety as a “target” to shoot for, UL 4600 offers a “bullseye,” said Koopman. For instance, UL 4600 will expect from automotive designers a lot of details, such as, “if you are doing X, don’t forget to do Y.” Other standards show “how to get to safety,” but UL 4600 prescribes “where you end up with your system.”
Make the safety case
UL 4600 is technology-agnostic. It doesn’t specify which sensor technologies should be used or how many miles of road testing must be accumulated before commercial deployment. Instead, UL 4600 concentrates on ensuring that “a valid safety case is created.” A safety case, in UL 4600’s view, includes three elements: goals, argumentation, and evidence.
So, if Tesla doesn’t want to use lidar in its AVs, for example, no problem. Rather, UL 4600 will ask Tesla to make the safety case, Tesla must credibly argue that “relevant objects will be successfully detected and classified with whatever sensors are installed within the limits of the intended operational design domain.”
In other words, “You can use any process or technology you want,” Koopman said. “But if you are not testing them, we’re not trusting you.”
Similarly, instead of asking for X miles of road testing, UL 4600 asks vendors to argue that “an acceptably robust combination of analysis, simulation, closed course testing, and safe public road testing have been performed to ensure an appropriate level of system safety for the initial vehicle and each software update.”
UL 4600 does not try to invent yet another V-cycle-based engineering process for computer-based system safety, Koopman stated. Rather, it “serves to take the information produced by existing and potentially new design and validation processes and standardizes how to organize the results into a coherent safety case.”
Making safety arguments isn’t particularly easy. Koopman said, “Think of it this way. You already have a lot of work products developed to ensure safety. You have a file cabinet full of papers. UL 4600 will help you organize those papers, structure them, and make it obvious all the items that must tested.”
The point is “to have a way to make sure that the design team didn’t miss something that was reasonably foreseeable,” he added.
Independence of assessors
The credibility of any standard hinges on who assessing the system’s compliance. In the automotive world where self-certification has long been SOP, any attempt at self-certification could be deemed self-serving.
The UL 4600 group, however, believes self-certification might work if the assessor is sufficiently independent and credentialed. “The key here is that you need to disclose the sufficient independence of your assessor,” said Koopman. “Use of credentialed external assessors is recommended, but not required so long as assessor independence and capabilities are credible and documented.”
Improving Safety Standards for AV
Managing unknown unknowns
One of the weak points of traditional IEEE standards is that they are not designed for frequent update. Presumably, conventional standards could update frequently “online.” But such updates thus far have no built-in mechanisms to validate their safety.
In contrast, Koopman claims that UL 4600 uses feedback loops to manage the risk of “unknowns.” Instead of “pretending that the engineers have thought of everything (they haven’t, and they won’t),” the standard accommodates responsible management of uncertainty, he explained.
Especially in the era of machine learning, it’s crucial for standards to have a build-in mechanism to fold in changes. “We call it continuous maintenance. That’s how the UL 4600 process works.”
That means those who participate in UL 4600 are asked, “If you find something, send it up” to the UL 4600 group. Users must report not only catastrophic failures but also “near misses,” so that the UL 4600 group can study the glitch and fix it.
“This is how we plan to keep the members well informed.” The group is promising to make urgent updates available in a few months and periodic updates perhaps yearly.
The UL 4600 group is developing its standard at a very aggressive speed. Comments by a Standards Technical Panel will be reviewed this summer, so the group can create a revised draft, which the group calls a “preliminary review version.”
Once STP comments have been substantially addressed, this version will be released to stakeholders. That is scheduled to happen "August or early September,” said Koopman. Once distributed to stakeholders that version of draft UL 4600 will be a public draft standard.
Registered stakeholders have the right to comment, but no voting rights. After one or more rounds of considering STP and stakeholder comments, the STP will enter a balloting phase. If all goes well, the result will be a public standard in late 2019 or early 2020.