IoT Calls for Standard Benchmarks

Article By : Rick Merritt

Benchmarks need support, but it’s true that they can’t tell the whole story of how one IoT chip compares to another. Chip vendors need to run and release meaningful benchmarks for IoT...

In the course of writing an article on 10 cool IoT processors for our sister publication Electronic Products, I realized a few things about the state of IoT silicon.

The good news is that a wealth of chips is available for IoT gateways and end nodes. A growing set of OEMs, system integrators, and even large end users will become customers for them as the IoT gets established over the next decade.

The bad news is that finding the one that’s right for you is like sorting through a sandbox to find a grain that’s a specific shade of brown. Chip vendors could help by increasing their generally poor level of support for existing and emerging benchmarks.

The most basic benchmark for IoT is performance/watt, but very little data is available on it. The EEMBC trade group has created a handful of IoT chip benchmarks to date, but many top vendors have yet to publish data using them.

For example, EEMBC released its ultra-low-power ULPMark in 2014 to measure the energy efficiency of MCUs. Kudos to the several vendors who have posted scores to date, but shame on the many who have not including some of the giants in small processors such as Cypress, Dialog, Infineon, Renesas, Samsung, and Toshiba. Get with it, people!

I am told that some marketing departments don’t want the quality of their chips reduced to a number that may not tell their whole story. Balderdash! EEMBC already supports a variety of metrics including Core and Peripheral profiles for ULPMark, a Bluetooth version of the new IoTMark, and even a SecureMark.

More are on the way. EEMBC hopes to release a standard measure of CoreMarks/joule at Embedded World this year. The group also plans versions of IoTMark for other wireless networks, with Wi-Fi probably the next one.

In addition, the group hopes to release in the next several weeks a machine-learning benchmark. It is expected to initially focus on computer-vision jobs using convolutional neural networks.

Given the heat around AI these days, chip vendors should be racing to be among the first to publish data using this emerging inference benchmark. The same goes for another embedded inference benchmark in the works at the MLPerf group.

I wrote a story for Electronic Products last year about embedded accelerators for machine learning. I came away from working on that story and a related article for EE Times convinced that basically everyone who sells embedded processors is working on an AI accelerator chip, block, or program.

They should all be putting engineers on getting out benchmarks for their work ASAP. Especially in this new style of computing, engineers need to know what’s working poorly, better, and best.

Beyond benchmarks, security and memory are key

Benchmarks need support, but it’s true that they can’t tell the whole story of how one IoT chip compares to another. Hugo Fiennes, founder and chief executive of Electric Imp, says that security should be a top concern.

Here, Fiennes starts with the datasheet. “If a chip doesn’t have the basic features, we won’t look at it.” His prerequisites include “a proper random number generator and enough key space, support for a secure boot, and an architecture that is not executing out of external memory, even memory in package.”

Determining how the hardware is used involves a more nuanced exploration.

“Security hardware and software have not been well-joined … People often don’t turn on all the security features in the hardware, so you already can build more secure systems than what most people ship,” he said.

Internet-class security can take a million lines of code, a level of complexity outside the typical sandbox of many embedded-system developers. What’s more, “IoT security has to be created to make sure it will still work in 10 years.”

All of that software needs room, and it’s cheaper to buy it upfront.

“It may cost $1,000 to swap out a node in field, so it’s better to pay $1.50 for more flash and RAM at design time — but not all vendors are offering enough memory, and they are missing out,” he said.

Fiennes cautioned about picking IoT chips strictly on their MIPS/watt. The metric “is not interesting because the processor typically sits next to a radio with a big power amp often burning hundreds of milliamps — and no cellular device will run off of coin cells.”

Overall, the outlook for IoT design is “pretty bright,” he said. “There are a lot of building blocks and they are getting more mature … so you can take something off the shelf, customize it, and ship working products with a [cloud] back end from a Microsoft or someone else — there’s been more pre-integration.”

— Rick Merritt, Silicon Valley Bureau Chief, EE Times

Virtual Event - PowerUP Asia 2024 is coming (May 21-23, 2024)

Power Semiconductor Innovations Toward Green Goals, Decarbonization and Sustainability

Day 1: GaN and SiC Semiconductors

Day 2: Power Semiconductors in Low- and High-Power Applications

Day 3: Power Semiconductor Packaging Technologies and Renewable Energy

Register to watch 30+ conference speeches and visit booths, download technical whitepapers.

Subscribe to Newsletter

Leave a comment