In the wake of the Uber self-driving fatality in Arizona in March, safety experts are urgently advising tech and automotive companies to push the reset button.

Headlines such as “How Safe Is Driverless Car Technology, Really?”, “Autonomous Cars: How Safe Is Safe Enough?” and “How safe should we expect self-driving cars to be?” have peppered the media in the past few months.

None of this is good news for robocars. The early public euphoria over a future of “driverless cars” has fizzled. The simplified arguments that assume that “AV technologies will save people’s lives” and “AVs are much safer than human driving” are under scrutiny.

The industry discussion about the “safety” of automated cars and, inevitably, safety standards is overdue.

Undoubtedly, “the self-driving car industry’s reputation has suffered a setback,” as Philip Koopman, associate professor at Carnegie Mellon University, wrote in a blog for EE Times after the Uber accident.

The question is how to fix it.

Rather than requesting that the industry stop AV development, safety experts are asking technology developers, third-party testers, and policymakers to step up with innovative ways to demonstrate safety.

As Koopman noted, the self-driving car industry needs to “make a safety goal, build a strategy for meeting the goal, and show evidence that the strategy actually works” for their autonomous vehicles. This must come, he emphasized, before they start testing their robocars on public roads.

The day when companies like Uber simply resorted to brute-force road testing in order to rack up AV test miles is over. The auto industry wants alternatives that include virtual testing, mathematical analysis, computer simulation, and methodologies to deal with corner cases.

A table with three numbered rows show sample results for each of Rand Corp.'s three statistical questions about the miles needed to demonstrate safety. They correspond to human-driven (A) fatality rates, (B) reported injury rates, and (C) reported crash rates.
Source: Driving to Safety (Rand Corp.)

In the table above, three numbered rows show sample results for each of Rand Corp.’s three statistical questions about the miles needed to demonstrate safety. They correspond to human-driven (A) fatality rates, (B) reported injury rates, and (C) reported crash rates.

The Rand Corp.’s analysis above shows that autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their reliability in terms of fatalities and injuries. The firm’s analysts summed up: “Our results confirm and quantify that developers of this technology and third-party testers cannot drive their way to safety.”

The question, then, is what alternative methods there are to augment real-world testing to assess the safety of AVs.

Lessons we’ve learned
In late May and early June, the National Transportation Safety Board (NTSB) successively issued two reports: one on Uber’s self-driving vehicle fatality in March in Arizona and the other on a Tesla Model X fatality in March in California.

These eagerly anticipated reports remain “preliminary.” The final analysis is not due until 2019, according to NTSB. In the preliminaries, investigators offered neither probable cause nor recommended fixes for the two accidents. Nevertheless, they did provide some insights.

It’s important to note that the separate NTSB reports deal with accidents involving two very different vehicles. Uber’s fatality occurred in the company’s self-driving car with a safety driver inside. Tesla’s Model X, in contrast, is a Level 2 car that drove into trouble while its autopilot was active.

Regardless of the divergent technologies involved in these accidents, the reports exposed several alarming elements about autonomy in general. Automotive industry experts, after reading the NTSB’s preliminary reports, cited the limitations inherent in sensors and the importance of monitoring drivers (whether a “safety driver” testing an AV or a Tesla owner using autopilot). Another critical factor noted is the fundamental difficulty of handing the wheel over, expeditiously, from machine to human.

Sensor aren’t always perfect
Tesla’s accidents, presumably caused by driver misuse of autopilot, clearly exposed the fallibility of sensors. Equipped with sensors that are not almighty, a Level 2 car can fail to recognize a catastrophe even as it drives into one.

We’ve already seen some jarring accident scenes that involved Teslas.

In Florida — Tesla’s first fatal accident on U.S. roads — neither autopilot nor the driver noticed the white side of a tractor-trailer against a brightly lit sky. The vehicle just kept going.

Tesla’s most recent fatality in California proved that the Model X steered toward a barrier and sped up in the moments before impact. “At [three] seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increased from 62 to 70.8 mph, with no pre-crash braking or evasive steering movement detected,” according to the NTSB’s preliminary report.

In short, sensors used for ADAS — radars and cameras — do not function at a level of full automation that average consumers expect. Drivers must keep their hands on the wheel while using autopilot.

Speaking of the Tesla fatality, Phil Magney, founder of VSI Labs, noted, “Radar does a poor job on static objects. It has to filter out most of them because if it did not, there would be too many false positives. This creates hazards.” He added, “Camera-based lane-keep algorithms really don’t know the difference between right and wrong. No ground truth to go by.”

Monitoring is critical
Recent Uber and Tesla crashes have exposed a hard reality. It’s really tough for drivers, who have gotten comfortable with “smart” technology, to re-focus on driving when a crisis looms.

Colin Barnden, principal analyst at Semicast Research, believes that “the moment automation comes into a vehicle,” regardless of the autonomy level, “it’s time to introduce [a] Driver Monitoring System (DMS).”

Magney agreed. “Recent accidents have substantially heightened the importance of DMS in partially automated Level 2 systems.” Citing his firm’s experience with Tesla’s Autopilot over the past six months, Magney now sees some weaknesses in Tesla’s approach, which he said “leads to system misuse.”

(Source: Seeing Machines)
(Source: Seeing Machines)

The actions of Uber’s safety driver during the accident are under scrutiny. Regarding the responsibilities of companies that beta-test prototype AVs on public roads, Koopman pointed out that “we have a safety driver” is a cop-out. At a minimum, he explained that safety authorities must be informed of the safety drivers’ training and how AV companies plan to ensure that drivers are alert and awake. Each driver’s on-road performance must be closely monitored.

Among DMS solutions designed to monitor drivers inside AVs, there are emerging companies that offer a “teleoperation-as-a-service” safety solution. Phantom Auto, for example, claims that it enables “a remote human operator to operate an AV when it encounters a scenario which it cannot handle on its own, allowing for optimally safe testing and deployment of these lifesaving vehicles.”

Magney sees Phantom Auto’s role “as more of a monitoring firm with the ability to teleoperate for when an AV is in a bind.” He explained, “Just like today’s TSPs [Telematics Service Providers; i.e., OnStar] who perform services for human-driven cars, there would be service providers that would serve the driverless robotaxi fleets in the future, and teleoperation would be one of the services they would offer.”

The potential concerns for networks that offer these services include coverage, bandwidth, and latencies. A monitoring center could also be overwhelmed by a natural disaster that requires fleetwide intervention. But in principle, Magney said, “I would imagine they are figuring out a way for one teleoperator to simultaneously teleoperate multiple vehicles at once.”

Simulation, simulation, simulation
In examining the Uber fatality, it is important to concede that a victim was killed not by a fully autonomous vehicle but by “a test vehicle” still in development.

When such unproven vehicles hit the road and then hit the populace, the responsible regulators — in this case, the state of Arizona — should share the blame. Absent this state’s permissive policy, the fatality was unlikely. Arizona required no official third-party oversight of safety verification, at any level, by the vehicle operator — in this case, Uber. This is why Koopman stresses that operators “should be required to explain why their on-road testing approach is adequately safe beyond simply saying that a safety driver is sitting in the vehicle.”

This leads to another question: Because it is currently impossible for an operator to prove that an autonomous technology is perfect, how do carmakers and tech companies assure themselves and tell others that the technologies they developed are “reasonably safe”?

Some experts think that the answer lies in simulations.

VSI Lab’s Magney told EE Times, “Simulation technology is increasingly valuable to developers of automated vehicles as they face mounting pressures to speed development, validation, and performance of their AV solutions.” He added, however, that simulation is hardly new to the auto industry. There are many tools to simulate a car’s powertrain, transmission, engine, thermal management, system dynamics, energy, electromechanical systems, etc.

What’s new is the combination of traditional simulation tools with a new generation of real-time raw data fusion platforms. Siemens, for example, has leveraged simulation products developed by Tass International, which Siemens acquired last year. They have collaborated to create a direct interface with a real-time raw data fusion platform, called DRS360, developed by another Siemens acquisition, Mentor. In this offering, Tass’s PreScan simulation environment produces highly realistic, physics-based simulated raw sensor data for an unlimited number of potential driving scenarios, traffic situations, and other parameters. The data from PreScan’s simulated lidar, radar, and camera sensors feeds into Mentor’s DRS360 platform, where the data fuses in real time to create a high-resolution model of the vehicle’s environment and driving conditions.

Varying fidelity-level sensor modelling. (Source: Siemens)

Varying fidelity-level sensor modelling. (Source: Siemens)

As Amin Kashi, ADAS/AD director at Mentor, told EE Times, on the combined platform, autonomous vehicle developers can do “development and verification of autonomous driving systems in an embedded centralized platform” consisting of an FPGA, SoC, and MCU.

Siemens isn’t alone. Similarly, at the company’s GPU Technology Conference, Nvidia discussed its new simulation system. With its Constellation Simulation System, as Magney explained, Nvidia is becoming “like the AWS [Amazon Web Service] of automated vehicle simulation.” This holds promise for AV developers who might not have had adequate access to computing power or simulation tools.

What about corner cases?
Compared to public road testing or closed-course testing, AV simulation is far “more scalable” and probably “less expensive,” according to Koopman. Simulations are critical. But equally important is: What exactly are we simulating?

Michael Wagner, co-founder and CEO at Edge Case Research, told us that the bad news for AV developers is that billions of simulation miles won’t necessarily expose so-called “corner cases” or “edge cases” that could befuddle robotic vehicles.

Click here for larger image
(Source: Edge Case Research LLC)

(Source: Edge Case Research LLC)

In the last few years, deep-learning chip vendors have spent vast resources advocating for the deep-learning algorithms that will make hands-off autonomous driving systems possible. Such algorithms hold the promise for robocars to develop the human-like ability to recognize patterns without requiring exposure to every possible situation.

The flipside of the argument, however, is that when a machine- or deep-learning system encounters something never seen before, the machine-learning-based machine might not register the anomaly, ignore it, and just keep on going.

Faced by a really unusual situation, human drivers — at the very least — realize that it’s weird. A normal human response: “I have no idea what I am seeing here. I’d better do something right now.”

Edge Case Research is focused on creating such anomalies for inclusion in a simulation platform, said Wagner. Acknowledging that it is still “early in the development phase,” he explained that his firm’s platform, codenamed “Hologram,” is being designed to “turn every mile driven by a real vehicle into millions of potential scenarios to root out the ‘unknown unknowns’ as quickly and safely as possible.”

Explainable AI
While the technology and investment communities tend to see AI as the sexiest part of the AV story, safety experts see AI as the thorniest, particularly for validating safety. The automotive industry does, indeed, have ISO 26262, a functional safety standard required for AVs. But machine learning is beyond the scope of ISO 26262. “Mapping machine-learning-based systems to traditional safety standards is challenging because the training data set does not conform to traditional expectations of software requirements and design,” said Koopman.

Not only is machine learning known to be brittle, it can also “game the system,” explained Koopman, by “memorizing” training data rather than “learning” to generalize from trends.

Machine learning does its own learning. But none of us non-machines know what exactly the machine has learned and how it has come to a certain conclusion.

Learning happens in a black box.

To validate the safety of an AI-based system, “we need an AI that can give us an introspective narrative … to tell us what it has learned,” said Koopman. In the AI world, it’s called “explainable AI.” While this is not ready yet, it’s on top of the to-do list among AI researchers.

Rigorous engineering needed
When an AV is driving in conditions outside of the intended operational design domains, the vehicle must recognize that it is in an unfamiliar territory.

Koopman said, “I’m not saying that the autonomous vehicle operating outside its envelope is NOT safe, but being ‘unsafe,’ and ‘not sure’ aren’t the same thing.” Noting that there are only three situations — ‘safe,’ ‘unsafe’ and ‘not sure,” he said, “Don’t tell me your autonomous vehicle is safe when you haven’t done the engineering legwork to determine whether it is safe or if you aren’t sure.”

AVs need rigorous engineering to prove safety.

If a robocar ventures beyond its operational design domain, “First, I want to make sure that your vehicle knows about it. Second, I want to know what the strategy is when that happens. Will it shut down the system in an orderly manner or will it do something else?”

That’s where the “big red button” comes in. By this, Koopman doesn’t mean just any big red button but a “properly designed” disengagement mechanism that “must be compliant to ISO 26262.”