Is anybody in the tech industry identifying technologies for watching driver behavior sufficiently effective to keep the driver’s attention from wandering dangerously?
The automotive and tech industries have worked feverishly on safety technologies that can sense, detect and steer clear of objects on the road. But is anybody working as hard on similar technologies that would watch drivers and be effective at preventing their attention from wandering dangerously?
Yes, but hardly anybody talks about it.
Driver monitoring systems (DMS) are often deemed either ineffective or obsolete by tech companies trying to add more automated functions to vehicles. Their answer to prevent driver distraction and save lives is more autonomy — “taking humans out of the loop,” a concept that was ominously explored in the film “WarGames.”
Humans are going to remain in the loop for many years to come, however, and partial automation creates a false sense of security that too easily leads to dangerous levels of disengagement with the task of driving.
A slew of studies, test results and investigations have proved the lesson of the movie. In this article, EE Times asks two key questions:
If we agree that Mobileye is dominant in advanced driver-assistance systems (ADAS) today, who might be, we wonder, the Mobileye of the DMS market?
Last week, the Insurance Institute for Highway Safety (IIHS) released research results showing the adverse impact of more partial automation deployed in Level 1 and Level 2 cars.
The research team, with members from IIHS and the Massachusetts Institute of Technology, studied for four weeks the behavior of 20 volunteers. Ten drove a Land Rover Range Rover Evoque with adaptive cruise control (ACC). The other half drove a Volvo S90 with ACC and Pilot Assist. The evidence showed: “The longer drivers used partial automation, the more likely they were to become disengaged by taking their hands off the wheel, using a cellphone, or interacting with in-vehicle electronics.”
IIHS is hardly alone sounding an alarm about partial automation.
U.K.-based Thatcham Research last month unveiled a scoring system for “assisted driving,” jointly developed with Euro NCAP. Its first test results suggest that more autonomy is not a safety panacea. A case in point is Tesla’s Model 3. The vehicle beat everyone in the assistance and backup categories with scores above 90 — but scored dead last for engagement with a 36 on a 100-point scale. Thatcham research director Matthew Avery explained: “Tesla lost ground for over-selling what its ‘Autopilot’ system is capable of, while actively discouraging drivers from engaging when behind the wheel.”
Assisted Drivig Grading — October 2020
Missy Cummings, director of Duke University’s Humans and Autonomy Laboratory, last month discussed what she calls “the conundrum of partial autonomy.” By sharing her recent research on “Tesla Model 3’s Autopilot interactions with the driver monitoring system,” Cummings emphasized that a so-called “shared responsibility” between computers and humans is problematic.
Earlier this year, the National Transportation Safety Board (NTSB), which investigated the two fatal Tesla crashes, published its final report. The NTSB chairman’s remarks on Tesla’s Autopilot Systems, were scathing: “The driver’s overreliance on Tesla’s ‘Autopilot’ and the operational design of Tesla’s ‘Autopilot’ have led to tragic consequences.”
Given the evidence piling up on driver disengagements triggered by partial autonomy, Colin Barnden, Semicast Research’s lead analyst, told EE Times, “It’s just a mystery to me why we’ve been blind to the absolute necessity for driver monitoring systems for too long.”
Barnden, often a lone voice advocating DMS within the media/analyst community, has his own theory on why DMS has languished. He contends:
The “self-driving” narrative remains deeply embedded the mainstream media’s focus. The press tends to follow the money. For example,Tesla’s market cap is $460 billion. Mobileye sold for $15 billion. Today Seeing Machines (DMS developer ) is worth about $275 million.
Despite all this money in the market, NHTSA has no legislation in place for DMS. Meanwhile, Europe has two DMS tracks (Euro NCAP and EU’s General Safety Regulation). Even China has a DMS roadmap! It would seem that the cards were stacked in favor of full-stack AV suppliers, who just so happen to be concentrated in the U.S.
Looking back, Barnden said, “I think a lot of people thought ADAS was a waste of time, DMS obsolete and AV just around the corner. The screeching U-turn was the death of Elaine Herzberg [caused by Uber] in March 2018.”
Skeptics on DMS
In May 2018, a tweet from CEO Elon Musk defended Tesla’s decision to forgo vision-based DMS. It read: “Eyetracking rejected for being ineffective.”
To be clear, Tesla’s Autopilot comes with a sensor to capture small movements of the steering wheel to gauge whether drivers are holding on. But the system is notoriously lax, delaying any warning until the driver’s hands have been off the wheel for as long as two minutes.
Over the years, automakers have seen many DMS technology companies emerge. Among them are Seeing Machines (Canberra, Australia), Smart Eye AB (Gothenburg, Sweden), Eyesight Technologies (Herzeliya, Israel), Jungo Connectivity (Netanya, Israel), Xperi which acquired FotoNation (San Jose, Calif.) and Affectiva (Boston).
If Musk still thinks that DMS is all about eye-tracking, he needs an update. Many leading systems no longer measure just one element, such as head position, eye gaze, face, or eyelid closure. Rather, they track multiple elements to generate a more holistic view for data and analysis.
“Some people think that driver monitoring is only eye-tracking. It’s not,” stressed Mike Lenné, senior vice president for human factors & aftermarket solutions at Seeing Machines, during a recent interview with EE Times.
He explained that eye-tracking must be done “exceptionally well to have a chance at doing driver monitoring.” But when people ask, “Can you detect visual distraction, cognitive distraction, drowsiness, intoxication, etc.,” Lenné noted that the issue comes down to “how good are you at harnessing the value, applying AI to those raw signals that come from eye-tracking, and harnessing that into useful, higher level drivers’ state feature?”
Nick DiFore, senior vice president for automotive at Seeing Machines, concurred. “If you can’t do a good job of tracking head, face and eyes, you are not going to have a lot of the requisite data” that enables you to translate it into high-level drivers’ state analysis, he explained.
Taking a few steps back, DiFore explained that Seeing Machines’ early lead in DMS derived not just from its data accuracy, but also a system design that made reliable signals “available.”
Critical is “the optical path,” said DiFore, including camera, image sensor, lens and illumination technology. Combined with software algorithms, “that really determines how robust the system will be,” said DiFore.
DiFore remembers that at the lab stage, Seeing Machines might have seemed little different from its many competitors. But once the DMS went into a car, driving in darkness and sunlight, car OEMs had to factor in that people do “weird things” like putting on sunglasses, glasses and contact lenses. DiFore said, “These are not even corner cases. It’s the real world.”
The differences in offerings by Seeing Machines and by its rivals have even widened when DMS companies began applying human-factor science to the collected data, seeking to understand what’s really going on inside the driver’s head.
If low level signal data was not accurate and repeatable, the result of machine learning analysis couldn’t be accurate. “Garbage in means garbage out,” said DiFore. Seeing Machines have been able to stay a step ahead of the competition, said DiFiore, with “good, stable, reliable signals.”
From face measurement to mind reading
So, what’s the next frontier for DMS? “State-of-the-art driver monitoring is now looking at about 60 frames per second vision analytics,” said Barnden. “There’s tremendous throughput, tremendous processes going on in these systems.”
Barnden pointed out, “What I am seeing now is a direction that looks very much like where Mobileye has gone with a front camera.” With DMS, he predicted, “You end up with a very tightly coupled chip and software solution.”
Obviously, a lot of innovations are happening in the optical path. “A transition from 850 nanometers infrared into 940 nanometers infrared” is important, because at 940nm, there is much less interference from direct sunlight, Barnden noted.
But the next phase of DMS is using probabilistic AI to apply human-factor and behavioral research, said Barnden, to assess levels of driver engagement.
Chip and software solutions
Until a few years ago, many tech companies thought they didn’t need DMS at all, or they limited the discussion to human-machine hand-over issues in Level 3 cars. Many reasoned that because they plan to skip Level 3 vehicles and jump to L4, why study DMS?
Now that even partial autonomy at Level 1, 2 and Level 2+ has proven to affect driver disengagement, DMS has rebounded. Ross Jatou, On Semiconductor’s senior VP and general manager, called DMS in a recent interview with EE Times a “tick-the-box” item.
Asked about OEMs who embrace DMS, Barnden noted that GM was early, in 2013, with a DMS program that became Super Cruise. BMW, Subaru and Nissan followed quickly, then Mercedes S-Class in 2017 and installing DMS in its F-150 in 2018.
Curiously, Audi, Porsche and Jaguar Land Rover all seem to have abandoned their first stabs at DMS. Barnden observed. “DMS is really unbelievably easy to do badly.”
Barnden puts stock in Seeing Machines – partly because of its experience and a series of design-wins. Seeing Machines takes pride in the voluminous data collected by monitoring more than 23,000 real-world truck drivers, combined with data from several thousand vehicles gathered through a licensing agreement with Caterpillar.
Seeing Machines’ spokesperson noted that it already has nine individual design-win programs with six OEMs.
The Australian company isn’t the only one racking up design wins, however. Earlier this year, Smart Eye announced 24 new design wins with four different OEMs. Smart Eye, at that time, said, “Of the four OEMs, two are new customers; one is an American high-volume car maker and one is a European premium manufacturer, and two are existing European premium customers.”
DMS in hardware vs. software
As DiFore explained, roughly half of Seeing Machines’ DMS business comes from the solutions made available in software, the rest is via chips. Xilinx has long been Seeing Machines’ silicon partner for DMS. With its field-programmable ability, Xilinx’s Fovio chip has gained traction from OEMs.
Seeing Machines also offers OEMs a second option of software only DMS solutions, creating an optimization path for specific chips OEMs prefer.
A good example is Qualcomm. Qualcomm has pushed into infotainment and ADAS, leveraging its Snapdragon platform. Seeing Machines, with Qualcomm, has worked “on a very optimized software pipeline that makes use of not only the standard Arm cores on that device, but the proprietary Qualcomm accelerators,” explained DiFore.
Aware that DMS isn’t the only application those ADAS processors will run, DiFore called optimization “very important.”
Without naming OEMs as to who will using whose chips, DiFore noted that Seeing Machines has already demonstrated that its DMS software runs on Nvidia, Renesas, Texas Instruments and others “on a wide array of platforms that are popular among OEMs.”
In September, penetrating its own DMS further into the market, Seeing Machines announced a new generation AI engine called Occula. Available in both SW only and Fovio Chip form, the Occula neural processing unit enables the next generation of Driver and Interior Occupant monitoring systems, according to the company. Seeing Machines is also making Occula available for license in ASIC form to chip suppliers, with a goal to have Occula “fit efficiently into semiconductor companies’ own automotive compute platform,” the company added.
According to Seeing Machines, General Motors is the only car OEM publicly disclosed to adopt Seeing Machines’ DMS — in the Cadillac CT6 Super Cruise system. Seeing Machines added that more models are in progress. Under the agreement, Seeing Machines supplied GM its DMS solution in software.
Seeing Machines won contracts to supply DMS via its Fovio chip solution into two U.S. OEMs and one Chinese carmaker.