Thermal imaging sensors are posed to augment the functionality and value of many products in our homes.
There are many home appliances and devices that promise to make your home safer and more comfortable, and to save you time or make your life easier. They automate several trivial tasks or make the controls of such devices available at your fingertips through connectivity to your mobile phone.
This functionality is enabled by a plethora of digital sensors, measuring anything from air temperature and humidity, through light conditions, to gas sensors and motion detectors. From the perspective of the information that such sensors provide, they have one thing in common—they are point-wise and lack spatial resolution.
For example, a conventional temperature or a light sensor would report the temperature or the light intensity only at the point of installation in a room. No awareness of the people in the room, nor their actions, nor their level of comfort can be inferred from such sensors.
In contrast, visual imaging sensors have very fine spatial resolution delivered by the millions of detectors integrated in the sensor. In that way, they generate rich unstructured information, upon which visual image analytics are applied to obtain structured information (for example, are there people in front of the doorbell, how many people, who they are, etc.).
The application of visual image sensors in smart robots and home security system is already very broad. Meanwhile, the evolution of technology is bringing other multipixel sensors to the arena of home automation and smart homes. Thermal imaging sensors are the prime candidate, with the prospect of helping in preventing fire hazards in the kitchen, helping your cooking, privacy-preserving human occupancy monitoring, measuring our comfort level and enhancing the well-being awareness around the elderly and babies. Below, we look at a few specific cases.
Your safer and smarter kitchen
Why thermal imaging in the kitchen? The short answer: for safety, health, and convenience. Indeed, according to the National Fire Protection Association (NPFA) of the United States, cooking is the leading cause of domestic fires, and the reason of 49% of such fires. And it is not that all the affected households had a malfunctioning smoke detector, but rather, by the time the smoke was detected, it was too late to eliminate the conditions leading to the fire. Thermal imaging can provide a clear way to minimizing such costly and devastating accidents, by identifying elevated temperatures that may lead to fire long before the hazard conditions set in.
A natural way of providing thermal imaging in the kitchen is by integrating the sensor within the extractor hood to realize a cooktop monitor. This is shown in Figure 1—the world’s first intelligent cooking assistant product (the copper-colored device mount right under the extractor hood). From there, the thermal imager can continuously ‘see’ the temperature distribution on the entire cooktop or stove.
Many culinary ingredients are sensitive to the temperature at which they are processed. Overheating them may not only deplete their nutritious value, but even turn them into less than friendly compounds for our bodies. A thermal imager above the cooktop can ‘read’ their temperature continuously and indicate it to us, aiding us to produce healthier meals.
“This leads us to the concept of a culinary assistant, helping you to create healthy and delicious meals every time. In its essence, Cooksy enables you to share your culinary recipes or to follow other’s recipes to the minute details of how long and at what temperature the ingredients are being cooked—during the process itself, in real time. Cooksy combines visual and thermal camera sensors with a sophisticated intelligence and connectivity in a device that is easy to mount above the cooktop. The device can livestream your cooktop to your mobile, letting you ‘see’ the temperature of the cooked ingredients and advises you how and when to act upon this information,” said Steven Cartwright, CFO of Cooksy Corp.
Figure 1 COOKSY-PRO, the world’s first intelligent cooking assistant (Source: Cooksy)
The benefits of a cooktop monitor extend beyond culinary art and nutrition into the realm of kitchen safety. For example, it can detect an unattended open flame of a gas cooker or an overheated range plate, overheated pots and pans with or without oil in them, and boiling and overflowing containers, to name a few. This is shown in Figure 2, where the boiling of a pot of water with a lid is monitored over time by a thermal imager.
Although the directly observable temperatures do not ever reach anywhere near 100°C (212°F), which is the temperature of boiling water, the application of thermal image processing and analytics makes it possible to detect when the water starts to boil, when it starts to overflow from under the lid, and when the gas burner is turned off. In the context of smart homes and connected appliances, the identification of such events allows not only to the triggering of an alarm but also the moderation or stopping of the electricity or gas supply if necessary and prevent an incident or fire hazard.
Figure 2 Thermal Imaging Analytic for cooktop applications (Source: Meridian Innovation)
A cooktop monitor is not the only place in the kitchen where one may find a thermal imaging sensor. An early prototype of a thermal-imaging enabled microwave oven—originally presented by Meridian Innovation at CES 2020, is shown in Figure 3 (A video of its operation can be seen here.
Figure 3 Smart Thermal Imaging Microwave Oven (Source: Meridian Innovation)
The unit features the ability to heat the food to the desired temperature, instead of heating it for a fixed time at a fixed power. It is based on an off-the-shelf microwave oven that has been modified to integrate a thermal imager and alternative controls. This allows us to dispense of the guesswork we must frequently apply in selecting the power and the time for re-heating our food. At long last, we can convey our intent directly, by selecting the desired food temperature, and leaving the technicalities to the machine. This intent is conveyed either through a wirelessly connected mobile device, or via the touch screen integrated in the modified microwave over.
Naturally, one can set the desired food temperature as well as to monitor the food while it is being heated up. Moreover, with the advances of TinyML and other initiatives towards the availability of AI capabilities on embedded processors, this set up may allow for the unit to guess what we have put inside the microwave while it is being heated, so that the optimal temperature is automatically selected for us. The imperative for a timely delivery of a thermal imaging cameras TinyML for the detection of elevated temperature and the complexity of building a good solution has led to many partnership efforts to provide a number of reference designs for OEM and ODM.
One good example is an article “Arrow Electronics Launches AI Thermal Sensing Solution Powered by STMicroelectronics’ X-CUBE AI.”
Another new solution is also mentioned with our partner below – When thermal imaging meets AI chipset.
Air conditioning that cares for your comfort
Airconditioning and HVAC systems are another example where integrating thermal imaging can make a positive impact. For the most part, the control of the temperature of a room involves a simple temperature sensor embedded in the box of the air-conditioning unit—this sensor provides the only feedback from the environment. This type of systems can be regarded as generation one and is by far the most common one today.
We may think of generation two being a system which knows whether there are occupants in the room, and possibly identifying in which part of the room they are. This enables more efficient utilization of the air conditioning unit and can be achieved by incorporating a thermal imager. For a small apartment room or a hotel room, a thermal imager of a few hundred to a thousand pixels may be sufficient, and for a larger living compartment, a sensor with a few thousand pixels is needed, coupled with a lens of more than 90 degrees field of view. A good example was a Smart Thermal Imager Air Conditioner concept shown in Figure 4, showcased by Meridian Innovation at Shanghai Smart Home Technology Expo 2019.
Figure 4 Smart Thermal Imager Air Conditioner (Source: Meridian Innovation)
Note that the content of the thermal image gives insight not only to the occupancy of the compartment, but to the overall distribution of temperature across the room. Based on this more detailed information the unit can automatically adjust the temperature, direction, and speed of the air flow to save energy and to improve the degree of comfort.
As for comfort, there are various mathematical models that establish a direct relation between a person’s average skin temperature and the person’s perceived level of comfort. We should keep in mind that the purpose of air-conditioning and HVAC systems is to ensure certain level of comfort and well-being of the occupants, rather than to merely cool or heat the air of a room to a preset temperature. This leads us to the concept of what we call generation three of HVAC systems. At that tier, the heating or cooling unit uses an even higher resolution thermal imaging – of the order of 5,000 to 20,000 pixels and can obtain a more detailed thermal picture, measure the average skin temperature of the occupants directly, and estimate their level of comfort. This information enables ambience control that aims at true thermal comfort, rather than temperature, and puts the human experience in the center of the feedback loop.
Air-conditioning units of generation two are already available on the market, made possible by recent availability of cost-efficient thermal imagers of 1,000 to 5,000 pixels at very large volumes. We project that the rapid evolution of thermal imaging technology will bring generation three functionality to high end models within five years.
Babies and elderlies first!
Baby monitors with video streaming have already been on the market for a while and offer a great way to share moments of the child’s joy with relatives and friends, or to have an eye on the baby while doing something around the house. However, the embedded visual cameras cannot give a direct insight into the physiological state of the baby. A thermal imaging sensor can. It can continuously track the temperature of the baby in a non-intrusive way.
Elevated temperature is a common symptom of babies being unwell. Such idea has been promoted everywhere as shown Figure 5 and new products are in the horizon. Every parent’s desire is to recognize such a symptom as early as possible, even in the middle of the night. Therefore, there is a lot of research and development effort into the creation of a baby monitor that combines both visual and thermal cameras, and to back the two imaging sensors with a rich AI computational engine.
The field offers many possibilities. For example, one can develop an accurate tracking of the baby’s normal temperature patterns—when the baby is healthy and happy, then automatically detect the occurrence of anomalous temperature patterns, and call for the awareness of babysitters or parents early in the development of a symptom. It should be kept in mind that the healthy pattern of babies’ core body temperature exhibits a significant diurnal variation that can be nearly 2° between its lowest and its highest. This pattern appears to be quite unique to each child and slowly evolves, as the child physiology and sweat glands develop, and it requires continuous observations to capture this pattern. Therefore, a baby monitor above the cot seems like a natural place to observe and record such an evolution.
Beyond health status, the thermal and visual light streaming modalities can be further leveraged in a feedback loop with a smart baby mobile to create an interactive play for the developing brain. Both the behavioral pattern and corresponding temperature markers can be analyzed at the same time. So, here you are—safe, smart and healthy babies, guaranteed!
Figure 5 Smart baby monitoring (Source: courtesy of 5GENCARE)
The wellness monitoring by thermal imaging extends to the elderlies, too. This is crucially important given two significant global trends—increased life expectancy and aging of the population and increasing number of elderly people living alone.
Thermal imaging may help in many ways, from simple monitoring of vital signs of temperature and motion, all the way to complex behavior analytics. One common and important theme in elderly care is the detection, and even the prevention of falls. While this can in principle be achieved with the installation of visual cameras in the living room, one would not do it in the bathroom.
But bathrooms pose the highest risk. With this in mind, a team from LSCM Hong Kong pioneered a thermal camera solution, built upon a relatively low resolution LWIR sensor with just under 5,000 pixels. This resolution is high enough to allow for pose estimation, but small enough to allow for continuous, real-time thermal image processing with modest computational power, and most importantly—to preserve privacy! The unit locally runs a sophisticated algorithm able to detect the occurrence of an accident, a fall, and alert a responsible nurse or a family member about the incident. The product won a gold medal on the 47th International Exhibition of Inventions in Geneva in 2019. Other companies develop camera solutions for the bedrooms of nursing homes, aiming at preventing common accidents – when elderly individuals get out of bed, by detecting that the person is awake and promptly alerting a caretaker to attend.
When thermal imaging meets AI chipset
Generalplus Technology is a global supplier of integrated circuits from Taiwan known for its consumer products and home appliances solutions in millions of the industry’s top brands products.
“We are excited to partner with Meridian Innovation. Their thermal imaging module and Generalplus’s AI chip are a perfect solution for smart homes. When Meridian’s engineer mounts its thermal imaging module on our AI SoC platform shown in Figure 6, it is a masterpiece in making,” said Jacky Chen, sales manager at Generalplus. “A cost-effective and standalone solution is born to integrate temperatures to audio, video, and graphics with Convolutional Neural Network (CNN) functions for smart and AI applications. When presenting such solution, we have seen great interest from customers in this cutting-edge technology to create an innovative product for better tomorrow.”
Figure 6 SmartForm Kit (Source: courtesy of Generalplus)
Thermal imaging sensors are posed to augment the functionality and value of many products in our homes, solving long standing challenges from preventing fire hazards in the kitchens and accidents with the elderly, through assisting over one’s cooking or raising both energy efficiency and comfort, to watching over babies and infants. This is enabled by the availability of a new generation of cost-efficient thermal imagers with a resolution of thousands and even tens of thousands of pixels, that are now manufactured in large volumes.
At the same time, we see a proliferation of accessible reference hardware designs and software solutions, which greatly facilitate the adoption of the technology and enable fast development cycle of innovative products incorporating thermal imaging cameras. Indeed, this is pivotal. Like the eyes of a person, thermal imaging sensors need a sophisticated intelligence behind in order to make sense of all the thermal data that is coming through.
Thus, the stage is set for the unfolding of yet another compelling story where tech innovation fosters safer and better living.
About the Authors
Hasan Gadjali is the co-founder and COO of Meridian Innovation Ltd.
Dr. Markov Stanislav Nikolaev is a scientist at Meridian Innovation Ltd.