Owl to Join the Thermal-Imaging Fray

Article By : Junko Yoshida

Thermal imaging is on the rise. Owl is introducing a monocular thermal imager producing High Definition images with 3D ranging information...

Teledyne’s announcement Monday to acquire Flir Systems, a thermal imaging sensor company, in a $8 billion cash and stock deal, signaled a clear message to automotive technology suppliers: Thermal imaging is on the rise.

Although somewhat unexpected, Teledyne’s move shouldn’t come as a surprise.

The year 2020 taught us that neither autonomous vehicles (currently in road tests) nor consumer vehicles with advanced driver-assistance systems possess a level of “eyesight” that works in all weather conditions on all terrain all day long and in the dark.

On one hand, AV developers have logged hours of driverless vehicles picking up and dropping off passengers in sunny Arizona, test-driving in Florida or tooling around downtown San Francisco (often limited to certain areas of the city at certain hours). On the other hand, conspicuously absent from the show are video clips of robocars braving fog, rain, snow, blizzards, low light and no light.

Herein then lies the opportunity for new sensing modalities, including thermal imaging.

Thermal image proponents, including Flir, Adasky and Foresight, have been proposing their technology as a “must-have” in the ADAS and AV sensor suites. Some thermal imagers have been designed as an option into commercially available luxury cars, although not as a standard feature.

Phil Magney, founder and president of VSI Labs, told us, “The biggest use case for thermal cameras will be for pedestrian detection in low light conditions. Right now, the safety agencies are not testing for this yet, but once they do, then you will see a rapid adoption of thermal cameras for ADAS applications.”

In a recent interview with EE Times, Chuck Gershman, CEO at Owl Autonomous Imaging, a startup based in Rochester, NY, said his company is joining the thermal imaging fray.

Owl is introducing a “monocular thermal imager producing high definition (HD) images with 3D ranging information,” he explained.

Asked how it differs, Gershman pointed out that suppliers such as Flir, Adasky and Foresight offer VGA images, not HD. Flir and Adasky create 2D, not 3D, images. Foresight offers 3D with a four-camera stereo system. Owl claims that its 3D system is monocular.

None of the developers (including Owl) touts its thermal imager as an end-all, be-all sensor for ADAS. However, they are challenging conventional wisdom on traditional sensor suites — for which many believe that visible-light cameras fused with radar will suffice, perhaps with a lidar, if you can afford it.

Thermal imaging promoters note that the beauty part of thermal is its ability to boost performance for ADAS (and AV) platforms in weather that includes dust, smog, fog, rain and snow.

Who is Owl?
Owl is a new entrant to the thermal imaging market for automotive applications. With 15 engineers, including several from Kodak who worked on the development of digital photography, Owl is preparing now for Series A funding.

Citing the startup’s extensive “homework on imaging,” Gershman said, “We didn’t take it lightly to go into this market.”

At the long- and mid-wavelength infrared range (LWIR and MWIR) spectrum on which Owl’s thermal imager operates, its sensor is completely passive. As with other thermal imagers, it leverages the principle that all objects emit thermal energy. As Owl would explain it, “Everything has a unique thermal signature.”

In contrast, other sensors need active light/energy sources. Visible-light cameras, for example, depend on light from the sun, streetlights, or headlights. Lidars emit laser light and use the reflected energy to measure time of flight. Radar sends radio signals and processes their bounceback.

Not having to rely on a light source to detect objects is the killer edge for thermal imagers. However, the question becomes how well a particular thermal imager can classify objects, and capture their associated range and velocity.

Owl claims it can do it all: detection, classification, range/velocity, day or night whatever the weather.

Basic building blocks of Owl’s thermal imager
Owl’s thermal imager comes in three blocks: 1) a monocular thermal imager, 2) an SoC that takes photonic information, converts photons to electrons and digitizes, and 3) a multi-aperture optical lens combined with CNN to create 3D ranging.

Click the image above to enlarge. (Source: Owl)

Owl claims its thermal imager can create depth/ranging information simply by using a monocular camera, rather than relying on stereovision.

Thermal imagers with 3D depth map output
To create ranging information, Owl has done a few things.  First, the multi-aperture lens in Owl’s monocular camera offers different perspectives of the same image. Using computational photography, Owl synthesizes a single image on every frame.

Second, Owl applies CNN to determine distance. Rather than training CNN by using new data sets based on Owl’s thermal imaging sensor (collecting numerous data sets is time-consuming), Owl devised a way to train CNN using stereo video through a process called “transfer learning.”

As Gershman explained, “At run time, we use both our multi-aperture optics coupled with our previously trained CNN to extract pixel disparity (which is the inverse of range) from a single image frame.  Additionally, we also generate pixel disparity, again the inverse of range, via a motion-based technique known as optical flow, which is a produced via intra-frame disparity measurements.” In the end, “It is the combination of all of these techniques that enables Owl to generate range from a mono thermal camera.”

Owl’s video linked here demonstrates visually how the grayscale thermal video stream generated by a single passive sensor is overlaid by a colorized dense range map from Owl’s CNN software.

(Source: Owl)

The video on the left shows the industry standard for 2D visual imaging with classification boxes. The image on the right shows Owl’s passive 3D thermal ranger with classification boxes. By overlaying Owl’s AI software algorithms, Owl’s grayscale thermal image is now colorized, and it generates range information from the camera. Both images use the standard Yolo v5 classification engine.

Gershman stressed that the grayscale thermal video stream and 3D range map are “simultaneously generated,” then optically fused on the imager. In his opinion, applying ranging techniques to thermal “has never been done before.”

Of course, as Gershman acknowledged, “Range estimation with multiple cameras such as stereo has been around for a long time. Further, range estimation with a mono camera using a ‘know reference’ is not really new.” By ‘know reference’, he means the pre-determined size of an object in the scene.  “For example, if I put a mono camera on the front of a train, and I know how wide the tracks are, I can calculate range by counting the number of pixels between the tracks, [because] the absolute width of the tracks never changes,” explained Gershman.

However, Owl’s 3D thermal imager in contrast uses its CNNs to determine distance without reference to a known-size object. “That is relatively new,” he noted. This was first demonstrated in academic literature in a September 2016 article entitled “Unsupervised Monocular Depth Estimation with Left-Right Consistency.”

Owl has been continuously enhancing its CNN through iterations, according to Gershman.

(Source: Owl)

Owl’s video shows that the thermal stream measures distance and classifies cyclists, vehicles, pedestrians, cars parked far to the side of the road. In contrast, the visual video fails to classify nearly all of these objects. The image above is a closer look at a couple of the critical objects of interest from the video. Note a frame with the two bicyclists (the image on top left) and a frame with the SUV and woman (the image on the bottom left) captured by the Owl’s thermal imager (left), compared to the frames captured by the visual video camera (top and bottom right).

While Owl has no intention to replace RGB cameras with its thermal imager, Gershman pointed out that his company’s thermal imager offers a much wider field of view.

“Your RGB camera at night is limited in distance by the distance of your headlights – it should have maybe 40, 50 meters total — and by the width of your headlights. Our intention is to build 105 to 110 degree wide field of view in a single camera.”

Owl’s thermal imager vs. radar
Many radar companies lately are talking up their 3D, 4D imaging sensors.

While acknowledging some improvements, Gershman believes that radars still produce very low-resolution images.

In debating resolution, there is a simple litmus test for 4D imaging radar. Assume a large car enters its field of view. Then, put a person next to the car.  The car is obviously bigger and taller.

The problem with radar is that it sees a car, but not a person. As radar emits radio signals and processes the return signal, it favors the car because the bigger the object, the more energy comes back. Likewise, more metal means more energy bouncing back. The lower energy response coming back from a smaller, softer object gets lost. This is why radar is often described as “noisy,” said Gershman.

Compared to radar, Owl’s thermal imager produces higher resolution images with less noise.

Lidar vs. Owl’s thermal imager
Lidar and Owl’s thermal imager are both looking for depth information.

Magney is still skeptical about the quality of Owl’s 3D information compared to lidars. He called lidar ranging “dead accurate,” because it uses time of flight to calculate range. In contrast, he suggested, Owl’s thermal imager is doing an “estimate.”

This circles back to the original question: What’s the real advantage of Owl’s thermal imager over lidar?

Owl’s Gershman noted, “Lidar produces a sparse cloud, which makes detection and classification of smaller objects at longer range unreliable.”

He added, “As a rule of thumb a LiDAR point is generally accurate to about 10cm which is good. However, that is based on a point being returned.  As LiDAR produces a sparse cloud objects at longer range may not be hit and thereby go undetected.”

In contrast, Owls’ dense point cloud, said Gershman, provides range information for objects large and small even at long range, providing enough response for Automatic Emergency Breaking to prevent contact with pedestrians, cyclists and animals.

However, as Gershman stressed, the point of offering a 3D monocular thermal imaging camera is not to replace cameras or radars. “The goal is to complement them,” he said.

Why OEMs are reluctant?
Despite the obvious advantage of thermal imaging (all weather, low and no light conditions), “OEMS have been reluctant,” Magney pointed out.

“First of all, it is expensive.” The cost argument also applies to carmakers reluctant to add lidars to ADAS.

Owl is talking turkey with some forty companies, including tier ones and OEMs who are already piloting with Owl’s prototypes, according to Gershman. He said, “We’ll be kicking off a number of additional pilots in the first half of 2021.

Owl’s new, improved version is scheduled to come out in mid-2021. Noting that this new iteration will be cheaper, Gershman said Owl expects to sell a monocular thermal imaging camera “at $200 in volume between 2023 and 2024.”

The second reason why a thermal imager makes OEMs hesitate, Magney noted, is the difficulty of installation. “You cannot mount a thermal camera behind the windscreen.  The optical properties of glass are incompatible with thermal imagers.  So this means you have to find a new place to mount it. This usually means the roof.”

Gershman concurred. “It can’t be behind the windshield. The windshield glass will either be lowered slightly, or the roof line raised around the camera.” He pointed out, however, that alternatively Owl’s thermal imaging camera – which fits in the palm of a hand – can fit inside a headlight. “Headlight plastic can transmit thermal.”

Subscribe to Newsletter

Leave a comment