InstMeter: An Instruction-Level Method to Predict Energy and Latency of DL Model Inference on MCUs

This paper introduces InstMeter, an instruction-level predictor that leverages MCU clock cycles to accurately and efficiently estimate the energy and latency of deep learning model inference, significantly outperforming existing methods in prediction accuracy and data efficiency across diverse hardware, software, and environmental conditions.

The Gist

Imagine you are trying to build a tiny, battery-powered smartwatch that can recognize your voice or identify a cat in a photo. To do this, you need to install a "brain" (a Deep Learning model) inside a very small, low-power chip (a Microcontroller or MCU).

The problem is that these chips are like tiny, frugal campers. They have very little battery and memory. If you pack the wrong "brain" into the camper, it might run out of power in an hour, or it might be too slow to wake you up when you say "Good morning."

The Old Way: Guessing with a Map

Previously, engineers tried to predict how much energy and time a model would take by looking at a high-level map of the model. They counted things like "Multiply-Accumulations" (MACs)—basically, counting how many math problems the model might solve.

The Analogy: Imagine you are trying to guess how long a road trip will take. The old method is like looking at a map and counting the number of cities you plan to visit.

• The Flaw: This is inaccurate. Driving through a city with traffic (compiler optimizations) takes longer than driving through the countryside. The old method didn't account for the traffic, the type of car (hardware), or the road conditions (software settings). It was a rough guess that often led to the camper running out of gas or arriving late.

The New Way: InstMeter (The Instruction-Level GPS)

The authors of this paper, Hao Liu, Qing Wang, and Marco Zuniga, built a new tool called InstMeter.

Instead of looking at the high-level map (the model's structure), InstMeter looks at the actual engine instructions the chip will execute. It counts the exact number of "ticks" (clock cycles) the chip needs to do the work.

The Analogy: InstMeter is like a high-tech GPS that knows exactly how your specific car behaves on every single road.

• It doesn't just count cities; it knows that "turning left at a red light takes 3 seconds" and "accelerating on a hill takes 5 seconds."
• Because it looks at the fundamental building blocks of the chip's work, the relationship between "ticks" and "energy" is perfectly straight and predictable (linear).

The Magic Trick: From Thousands to Ten

Here is the most impressive part. Because the relationship between "ticks" and "energy" is so straight and simple, InstMeter doesn't need to study thousands of examples to learn the rules.

• Old Methods: Needed to test 1,000 to 5,000 different models to learn how to guess correctly. This is like trying to learn to drive by crashing into 5,000 different cars.
• InstMeter: Only needs to test 5 to 10 models to get it right. It's like learning to drive by taking a short test drive and then knowing exactly how the car will perform.

How It Works (The "Source-to-Disasm" Bridge)

The tricky part is that the code engineers write (Source Code) looks very different from the code the chip actually reads (Disassembled Code). The chip code is like a secret language where variable names are replaced with random numbers.

The authors created a clever translator that maps the "loops" (repeating actions) in the human-readable code to the secret code the chip uses.

• Analogy: Imagine you have a recipe written in English (Source Code) and a recipe written in a secret code (Disasm). InstMeter's translator realizes that "Mix for 5 minutes" in English is the same as "Repeat instruction X, 5 times" in the secret code. Once it makes this connection, it can calculate exactly how long the cooking will take.

Why This Matters

1. Super Accuracy: InstMeter is 3 to 6.5 times more accurate than the best existing tools.
2. Super Fast: It needs 100 times less data to learn. This means engineers can design better AI models for tiny devices much faster.
3. Universal: They tested it on different types of chips (ARM and RISC-V), different temperatures, and different software settings, and it worked like a charm every time.

The Bottom Line

InstMeter is like giving engineers a crystal ball that tells them exactly how much battery a smart device will use before they even build it. It stops them from guessing and helps them pack the perfect "brain" into their tiny, battery-powered campers, ensuring they stay powered up and responsive for years.

Technical Summary

1. Problem Statement

Deep learning (DL) models are increasingly deployed on Microcontrollers (MCUs) for edge intelligence. Designing efficient models for MCUs requires balancing accuracy with strict constraints on energy consumption and latency.

• The Challenge: Neural Architecture Search (NAS) is the standard method for finding optimal models, but it requires evaluating thousands of candidates. Directly measuring energy and latency for every candidate is computationally expensive and time-consuming.
• Limitations of State-of-the-Art (SOTA): Existing predictors rely on coarse proxies like Multiply-Accumulations (MACs) or model parameters.
  • Non-Linearity: The relationship between high-level model parameters and final hardware performance is non-linear due to compiler optimizations (e.g., dead code elimination, instruction scheduling, kernel fusion) and specific MCU implementations.
  • Data Hunger: To capture these non-linearities, SOTA methods (like nn-Meter or MAC-based regressors) require thousands of training samples (1,000–5,000), which is impractical for resource-constrained MCU environments.

2. Methodology: InstMeter

The authors propose InstMeter, an instruction-level framework that transforms energy/latency prediction from a non-linear problem into a linear one by leveraging MCU clock cycles as the primary proxy.

Core Insight

While model parameters (e.g., layer types, filter sizes) have a complex, non-linear relationship with energy, the number of clock cycles executed has a strong linear relationship with both energy and latency.

• Linear Model: $E \text{ (or } L) = a \cdot \text{Cycles} + b$
  • Where $a$ is energy/latency per cycle, and $b$ is a fixed overhead.
  • This linearity allows the model to be trained with extremely few samples (<10).

Key Technical Components

1. Source-to-Disassembly Mapping:

   • Challenge: Source code (C++) contains loop variables but lacks instruction details; Disassembly (Assembly) contains instructions but obscures variable names.
   • Solution: A novel loop-level mapping algorithm that bridges the two.
     • Structural Features: Uses Control Flow Graphs (CFGs) to map loop structures (subset and intersection relationships) between source and disassembly.
     • Semantic Features: Uses string literals (function names, variable names) and, crucially, comparison operators (e.g., <, >, ==) which remain consistent across representations.
     • Algorithm: Combines graph matching (VF2 algorithm) with weighted semantic similarity scoring to achieve one-to-one loop mapping.

2. Instruction Profiling & Library Generation:

   • Once loops are mapped, the system extracts the number of times each instruction is executed based on the loop variables from the source code.
   • It multiplies instruction counts by their Cycles Per Instruction (CPI) (derived from documentation or empirical measurement) to calculate total execution cycles.
   • An Instruction Library is pre-generated for specific MCUs (Cortex-M4, M7, M33, ESP32-C3) and TFLM versions, allowing rapid cycle estimation for new models without recompilation.

3. Data Collection:

   • An automated tool compiles models, uploads them to MCUs, and measures energy/latency using a high-precision power meter (OTII ACE PRO).
   • The system only requires a handful of training samples (e.g., 5) to fit the linear regression parameters ($a$ and $b$).

3. Key Contributions

1. Paradigm Shift: Transformed DL energy/latency estimation from a non-linear, data-hungry problem to a linear, low-data problem by using clock cycles as the fundamental metric.
2. Novel Mapping Algorithm: Developed a fine-grained source-to-disassembly mapping technique using structural (CFG) and semantic (operator-based) features, enabling accurate cycle counting without hardware performance counters.
3. Comprehensive Benchmark Dataset: Created a large-scale dataset (2,125 models) covering:
   • Hardware: 3 ARM MCUs (Cortex-M4, M7, M33) and 1 RISC-V MCU (ESP32-C3).
   • Variables: Compiler optimizations (-Os, -O2), GCC versions, temperatures (21°C, 43°C), DVFS settings, and application scenarios (Keyword Spotting, Image Recognition).
4. Instruction Dictionary: Provided pre-built instruction libraries for researchers to predict instruction counts for any DL model on supported MCUs.

4. Results

Extensive evaluations demonstrate InstMeter's superiority over SOTA methods (MAC-based predictors and nn-Meter):

• Accuracy:
  • Energy Prediction: Reduces error by 3× compared to nn-Meter and 6.5× compared to MAC-based methods.
  • Latency Prediction: Reduces error by 6.5× compared to MAC-based methods.
  • Error Rates: InstMeter achieves a relative error <30% for 90% of models with just 5 training samples. In contrast, nn-Meter requires 500 samples to achieve ~100% error, and MAC-based methods require 50 samples to achieve ~200% error.
• Data Efficiency:
  • Requires 100× less training data than nn-Meter and 10× less than MAC-based methods.
• NAS Performance:
  • In Neural Architecture Search scenarios, InstMeter identifies optimal models that stay within energy budgets with <10% error, whereas other predictors often exceed budgets or sacrifice accuracy due to high estimation errors (up to 100%).
• Generalization & Robustness:
  • Maintains high accuracy across different MCUs, compiler versions (-Os, -O2), GCC versions, temperatures, and DVFS settings.
  • Works effectively across different TFLM versions (v2.4 and vCI).

5. Significance

• Enabling Efficient NAS: By drastically reducing the data collection burden (from thousands of samples to <10), InstMeter makes it feasible to perform rigorous NAS on MCUs, leading to better-performing edge AI models.
• Hardware-Aware Optimization: It bridges the gap between high-level model design and low-level hardware execution, accounting for compiler optimizations that previous methods ignored.
• Community Resource: The release of the source code, instruction libraries, and the comprehensive benchmark dataset fills a critical gap in the literature, as no equivalent datasets existed for MCUs previously.
• Scalability: The linear approach is computationally lightweight, making it suitable for integration into automated design flows and edge devices.

In summary, InstMeter solves the critical bottleneck of accurate, low-cost performance prediction for edge AI by leveraging the fundamental linearity of hardware clock cycles, outperforming complex non-linear models with a fraction of the data.