Benchmarking Ultra-Low-Power -NPUs

Microcontrollers (MCUs) are widely used in resource-constrained environments due to their form factor and low cost and power consumption. Performing model inference on MCU-scale hardware improves privacy by running locally, lowers operating costs for model vendors, and eliminates dependence on network connectivity.

However, deployments remain constrained by limited memory, throughput, and compute. The growing computational demands of modern models have catalyzed specialized accelerators across the computing spectrum, from data centers to embedded systems.

At this resource-constrained end, MCU-scale neural processing units, or NPUs, have emerged to provide real-time or near-real-time inference within milliwatt-scale power budgets.

See also the full paper at Benchmarking Ultra-Low-Power μNPUs