Seeing Machines Might Be the Next Arm

Article By : Colin Barnden

Little attention has been paid to technology which better understands humans and that not only makes us safer drivers.

With CES 2022 finally over, let’s sidestep entirely the feelings of déjà vu surrounding “consumer AVs,” and “personal AVs.” In this article, I focus on details for driver monitoring systems (DMS) and announcements related to Seeing Machines, which is beginning to look a lot like the next Arm.

Far away from the smoke and mirrors of the Las Vegas Strip was a technical white paper published by Ojo-Yoshida Report (that’s EE Times’ old friends Bolaji Ojo and Junko Yoshida) entitled “The DMS Embedding Challenge” written by Seeing Machines. In it the authors describe how industry-standard CPUs and SoCs are typically poorly matched to the specialist processing and pipeline structure necessary for efficient DMS processing:

“DMS products are inherently high performance, real-time computing systems. From a processing perspective, video is pushed into a pipeline which operates on the continuous stream of pixels, in a series of stages, where hierarchies of interconnected algorithms (or a DMS “engine”) squeeze out and extract desired information from the image data, passing higher-order information from one stage to the next, until what is produced is a low-bandwidth set of high-level (and high-value) results, such as where the driver is looking and if they are distracted or impaired.”

The authors continue:

“In automotive embedded systems that contain DMS functionality, it is fair to say that there is almost always the need to accelerate stages of the pipeline using specialist processor designs. This isn’t because regular CPUs aren’t capable of executing the pipeline, but simply because the CPU resources made available are almost never sufficient and because there are usually far more efficient ways of executing specialized types of functions, including image pre-processing, computer-vision operators, signal processing, and neural networks.”

According to Seeing Machines, nearly all of the issues that it has seen when mapping DMS algorithms to a given chip end up being associated with inefficiencies around dataflow and the suboptimal use of both on-chip and off-chip memory. The authors observe:

“Accelerators will often achieve lightning-fast numerical operations only to let us down badly in the act of moving the data to the next stage of the pipeline and being forced to buffer data in external DDR memory. This is again symptomatic of the chip simply not being designed with the DMS-oriented pipeline configuration in mind.”

The parallels with Arm stem from the decision made by Seeing Machines to develop a proprietary architecture to perform DMS-oriented processing as efficiently as possible. The authors describe the advantages of a software-hardware co-design approach  and then summarize the benefits it brings to its Occula neural processing unit (NPU):

“Despite being designed and built from the ground up for DMS solutions, the Occula NPU design when married with Seeing Machines DMS algorithm stack, may offer performance advantages to a far wider range of products — any product that is price or power sensitive, and can yield an advantage from understanding contextual information about humans.”

The authors conclude:

“Occula has been developed to address the limited application scope of not only DMS, but more generally of understanding humans.”

Further details of the Occula NPU are presented in this video.

Humble beginnings

The seeds of Arm’s victory in embedded processing were sown in the early ’90s, with a vision of massive demand for power-efficient processor cores tied to a business model of licensed intellectual property. Commercial success came first in cellphones, starting with a GSM baseband processor for Nokia, designed in partnership with Texas Instruments.

From humble beginnings in a converted barn just outside Cambridge, UK, Arm rose to global dominance in cellphones (and later smartphones), and expanded its presence steadily into consumer electronics, automotive, industrial, medical and communications infrastructure.

Over the course of three decades, Arm built a broad ecosystem of partners using a licensing model spanning CPU cores ranging from the original ARM1 to the most recent Cortex-A710 and Cortex-X2.

From headquarters in Fyshwick, a little-known suburb of Canberra, Australia, Seeing Machines has for several years pursued a broadly similar licensing model to Arm. In September 2020 it announced its next-generation automotive strategy, which encompasses three pillars:

  1. The Fovio chip, a Xilinx Zynq-7000 FPGA optimized for driver and occupant monitoring.
  2. The embedded driver monitoring engine (e-DME), optimized both for accelerated (e.g., Qualcomm, Texas Instruments) and non-accelerated (e.g., Nvidia) processing.
  3. The Occula NPU, licensable in ASIC form.

As CES 2022 demonstrated, Seeing Machines’ licensing-based automotive strategy is rapidly forming into a broad ecosystem of partnerships. Let’s look at some of the announcements.

Omnivision OAX4600

Omnivision was announced as the first silicon licensee for the Occula NPU at CES in 2021. At CES 2022 Omnivision introduced the OAX4600 ASIC, combining an image signal processor (ISP) with the Occula NPU.  Omnivision also announced the OX05B1S, a 5 megapixel (MP) RGB-IR image sensor for in-cabin monitoring.

Taken together this two-chip solution can be regarded as an example of a highly optimized “imaging signal chain,” comprising an ISP tuned both for the 5MP OX05B1S image sensor on the front-end and the Occula NPU on the back-end. This solution looks positioned to be very successful in occupant monitoring mirrors, for example from Magna, which are both highly space and power constrained for integration of the optical components and image processor, as shown below.

As the only current licensee of the Occula NPU, Omnivision holds a clear competitive advantage over other automotive image sensor suppliers, including Onsemi, Sony and STMicroelectronics, heading into the crucial mass-market automotive OEM nominations for driver and occupant monitoring to be decided throughout this year and next.

Ambarella CV2x

Ambarella is rarely thought of as a processor supplier to the OEM automotive sector. However, as this slide from its Capital Markets Day presentation shows, that is beginning to change.

Ambarella recently announced its CV2x-based reference design platform, which performs the vision processing and fusion of Seeing Machines’ driver monitoring software with forward-facing ADAS features to provide a complete, integrated DMS and ADAS solution.

Ambarella looks to now be competing against Mobileye in ADAS on the basis of an “open platform” strategy (the same argument is used by Qualcomm), while targeting software-defined platforms at a lower performance and price-point than Nvidia.

Based on comments made in its CES keynote by Ali Kani (VP/GM of Automotive), Nvidia appears increasingly apathetic towards ADAS and altogether disinterested in DMS. As highlighted recently by Egil Juliussen, Nvidia dominates the supply of processors in robotaxi platforms, perhaps explaining its apparent focus on the highest levels of driving automation.

This strategic shift looks to have been seized upon by suppliers such as Ambarella which, as shown below, is working to establish an automotive ecosystem of its own, including with Seeing Machines.

Qualcomm Snapdragon vision

Qualcomm used CES 2022 to again challenge the narrative of leadership by Mobileye and Nvidia in high-performance automotive processors, as shown in its keynote video below.

Over the last three years, Nvidia has announced partnerships with three global, established, automakers: Hyundai Motor, Mercedes-Benz, and Volvo Cars. Demonstrating its potency, Qualcomm promptly announced Honda, Renault, and Volvo Cars (again) just in this keynote.

Qualcomm’s big CES reveal was the Snapdragon Ride Vision System, a 4nm SoC designed for an optimized implementation of front and surround cameras for ADAS and automated driving. As this slide shows, Seeing Machines is now the only specialist DMS software provider named on the Qualcomm Snapdragon Ride platform, optimized to run on the Spectra ISP.

CES 2022 also saw the announcement of the expansion of the Seeing Machines software stack to include occupant monitoring integrated onto the Qualcomm Snapdragon Cockpit Platform, almost certainly optimized to use the 5MP Omnivision OX05B1S image sensor mentioned earlier.

Ford-Volkswagen ADAS collaboration?

The collaboration between Ford and Volkswagen for autonomous driving is well-documented. However, buried in the details of the “Under the Hood” presentation from Mobileye’s Amnon Shashua was this slide.

This implies the technology collaboration between the two automakers could now also extend to ADAS and a next-generation hands-free “highway assist” system, powered by Mobileye’s EyeQ processors and using Mobileye’s REM maps.

Research suggests both Ford and Volkswagen will integrate Seeing Machines’ DMS, with Ford’s BlueCruise using a system developed by Veoneer powered by Seeing Machines’ Fovio chip, and Volkswagen using a system developed by Magna powered by the Seeing Machines software running on Texas Instruments’ TDA4VM Jacinto processor.

Arms, heads, and eyes too

Mark Twain is often credited with saying “History doesn’t repeat itself, but it often rhymes.” Arm may have started out with a simple mission to develop a power-efficient processing architecture, but it ended up revolutionizing both communications and consumer electronics, and enabled development of the IoT.

68000, MIPS, and Power Architecture were three other established embedded processing architectures in the ’90s, but all were swept away by the inexorable rise of the Arm ecosystem, as success simply bred more success.

Seeing Machines’ business model is not an exact replica of Arm’s, but it rhymes. What started life as mundane software to monitor mining truck drivers for distraction and drowsiness has now morphed into state-of-the-art neural net-based technology to “understand humans,” including not just arms but heads, faces, and eyes too.

Compared with the tens of billions of dollars spent seeking to replace humans as drivers, little attention has been paid to technology which better understands humans and that not only makes us safer drivers but personalizes the in-cabin immersive experience too.

Qualcomm is merely the first of the technology majors to fully comprehend the revolution ahead in the digital cockpit and chassis, a trend which has barely begun and yet will run for decades. Apple is likely monitoring technology developments for use in a future iCar,  so too Amazon, Google (Alphabet) and others.

As the above slide shows, analysis of the key competency indicators (KCIs) suggests Seeing Machines is heading for the leading position in the DMS market. Intriguingly, no partnership announcements were made by either Cipia or Smart Eye at CES.

As this article explains, an automotive ecosystem is rapidly forming around Seeing Machines that includes Omnivision, Ambarella and Qualcomm. Bolstered by a recent $10 million investment from Magna, design-wins with automakers such as Ford and Volkswagen, and partnerships with sixteen tier one suppliers (as of August 2021), its rise now has both traction and momentum.

We are witnessing the emergence not just of a new supplier, but of an entirely new class of technology. For all the talk of “consumer AVs,” and “personal AVs” at CES, the next big trend is seeing machines that understand humans.

This article was originally published on EE Times.

Colin Barnden is principal analyst at Semicast Research and has over 25 years of experience as an industry analyst. He is considered a world expert on market trends for automotive vision-based driver monitoring systems (DMS). He holds a B.Eng. (Hons) in Electrical & Electronic Engineering from Aston University in England and has covered the automotive electronics market since 1999.


Subscribe to Newsletter

Leave a comment