🤖 machine learning

MiCoPro: End-to-End Mixed Precision HW/SW Co-design with HW-aware Proxy Model

The paper presents MiCoPro, an end-to-end hardware-software co-design framework that utilizes a hardware-aware proxy model and a novel optimization algorithm to efficiently search for mixed-precision quantization schemes, enabling rapid deployment of edge AI models with significant latency reduction and minimal accuracy loss.

Original authors: Zijun Jiang, Yangdi Lyu

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zijun Jiang, Yangdi Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fit a massive, high-definition movie into a tiny, old-school MP3 player. The movie is a "Neural Network," a type of computer brain that learns to recognize cats, translate languages, or drive cars. The problem is that these brains are usually built with heavy, floating-point numbers (like 32-bit or 64-bit), which take up a lot of space and require a lot of energy to process. To make them run on small devices like smartwatches or drones, engineers use "Quantization." Think of this as compressing the movie: instead of using millions of colors, you force the image to use only a few shades of gray. This makes the file smaller and faster to play, but if you compress it too much, the picture gets blurry and the AI stops recognizing the cat.

For a long time, engineers tried to compress the entire movie to the same low quality, like turning every frame into a 4-bit sketch. But this is like forcing a slow-motion action scene and a quiet dialogue scene to have the same low resolution; it wastes space on the quiet parts and ruins the action parts. A smarter idea is "Mixed Precision Quantization" (MPQ). This is like a smart compression algorithm that keeps the action scenes in high definition (8-bit) but shrinks the boring dialogue scenes to tiny sketches (2-bit). The challenge, however, is figuring out exactly which scenes to shrink and by how much. If you guess wrong, the movie looks terrible. If you guess right, you get a tiny file that still looks amazing. This is the puzzle the researchers in this paper set out to solve.

The paper introduces a new toolkit called MiCo (and its upgraded version, MiCoPro) to solve this puzzle automatically. Instead of a human trying to manually tweak every layer of a neural network—which is like trying to edit a 100-hour movie frame by frame—the MiCo system acts like a super-smart, fast-forwarding editor. It uses a "proxy model," which is essentially a crystal ball trained to predict how fast a specific compression scheme will run on real hardware without actually running it.

The researchers found that the old way of guessing speed was like counting the number of words in a book to guess how long it takes to read it. It's a rough estimate, but it ignores whether the font is small, if the reader is tired, or if the book is heavy. The paper argues that this old method (called "Bit Operations" or BOPs) often leads to bad choices because it doesn't understand the specific quirks of the hardware. MiCoPro, on the other hand, builds a custom "speedometer" for each specific device. It learns how different hardware handles different types of math, allowing it to find the perfect mix of high and low precision that makes the AI run the fastest without losing its accuracy.

In their experiments, the team tested this system on various AI models, from simple image recognizers to larger language models. They found that by using their new "Hardware-Aware Proxy," they could cut the time it takes for the AI to think (latency) by up to 40% while only losing less than 3% of its accuracy. It's like finding a way to make your phone's battery last 40% longer without making the screen dimmer.

The paper also highlights that this isn't just a theory; they actually built the software to take a model designed in Python (a popular coding language) and turn it into raw, bare-metal C code that can run directly on chips, including specialized accelerators and modified RISC-V processors. They showed that their method works better than previous "search" algorithms, which often get lost in the huge number of possible combinations. By using a technique called "Near-Constraint Sampling," MiCo focuses its search on the "sweet spot" right near the speed limit, rather than wasting time checking options that are too slow or too fast.

Ultimately, the paper suggests that to get the most out of AI on small devices, we need to stop treating all layers of a neural network the same. By using a smart, hardware-aware guide to mix and match precision levels, we can make AI faster and more efficient. The authors demonstrate that this approach is robust, working across different types of chips and models, and they provide an open-source framework for others to use. While the results are based on simulations and specific hardware tests, the consistent success across different scenarios suggests that this "smart compression" strategy is a powerful tool for the future of edge AI.

Technical Summary: MiCoPro: End-to-End Mixed Precision HW/SW Co-design with HW-aware Proxy Model

Problem Statement
Quantized Neural Networks (QNNs) are essential for deploying AI on resource-constrained edge devices. While layer-wise mixed-precision quantization (MPQ) offers a pathway to maximize efficiency without sacrificing accuracy, existing exploration algorithms face significant limitations. Conventional methods struggle with the high dimensionality and complex correlations of the MPQ search space, often relying on inefficient heuristic searches or machine learning techniques that require excessive evaluation budgets. Furthermore, standard hardware cost metrics, such as Bit Operations (BOPs), fail to accurately predict real-world latency on specific hardware targets because they ignore architectural nuances like operand packing, memory bandwidth, and SIMD utilization. Finally, there is a lack of a unified, end-to-end framework that can efficiently explore MPQ schemes based on hardware characteristics and directly deploy the resulting models from PyTorch to bare-metal C code across diverse platforms.

Methodology
The authors propose MiCo, a holistic framework for MPQ exploration, and its extension MiCoPro, which integrates a Hardware-Aware Proxy (HAP) model.

MPQ Exploration Algorithm (MiCo):
- Formulation: The search is treated as a budget-constrained black-box optimization problem.
- Surrogate Modeling: The framework employs Random Forest (RF) ensemble learning to predict model accuracy based on quantization configurations. RF is chosen for its ability to model non-linear interactions in discrete search spaces with limited training data.
- Sampling Strategy:
  - Orthogonal Initial Sampling: Instead of uniform random sampling, the method generates an initial set of samples that include extreme bitwidths (e.g., W8A8, W1A1) and orthogonal combinations where specific layers are quantized while others remain full precision. This ensures the predictor learns the bounds and interactions of the search space efficiently.
  - Near-Constraint Sampling (NCS): A genetic algorithm generates candidate schemes within a Region of Interest (ROI) close to the hardware constraint (e.g., latency or BOPs limit). The ROI starts narrow and expands if no feasible solutions are found, prioritizing schemes that maximize accuracy near the constraint boundary.
- Evaluation Strategy: The framework utilizes a bifurcated evaluation approach. For bitwidths $\ge$ 4 bits, it uses Post-Training Quantization (PTQ) for rapid estimation. For sub-4-bit configurations, it employs a short Quantization-Aware Training (QAT) re-training phase to recover accuracy, balancing search cost with precision.
Hardware-Aware Proxy (MiCoPro):
- Kernel Profiling: The system collects cycle-accurate latency data for mixed-precision kernels (MatMul, Conv2D) across various shapes and precision combinations on target hardware.
- Feature Engineering: To overcome the limitations of BOPs, the authors introduce Composite Bit Operations (CBOPs). These features decompose hardware cost into:
  - BMACs (Bit Multiply-Accumulate): $\max(b_w, b_a) \times MACs$ , capturing the bottleneck of mixed-width operand extension.
  - ALoads/WLoads: Separate terms for activation and weight memory traffic, capturing asymmetric impacts on bandwidth.
- Calibration & Transfer: A network-dependent min-max calibration aligns proxy predictions with actual hardware latencies at precision extremes. Additionally, a transfer learning mechanism allows the proxy to adapt to new hardware variants (e.g., different RISC-V core configurations) using minimal new profiling data by learning the residual error of a pre-trained model.
End-to-End Deployment:
- The framework includes a deployment pipeline that converts the optimized PyTorch model into a PyTorch FX graph.
- It supports graph conversion for specific simulators (e.g., BitFusion) and generates bare-metal C code for CPUs.
- It utilizes a custom mixed-precision kernel library (extending baremetal-NN) with SIMD-accelerated variants for 8/4/2/1-bit operations, enabling direct execution on RISC-V processors and accelerators.

Key Contributions

Novel Exploration Algorithm: A Random Forest-guided Near-Constraint Sampling (RF+NCS) strategy that efficiently explores MPQ schemes, achieving lower accuracy drops under strict constraints compared to heuristic and reinforcement learning-based methods.
Hardware-Aware Proxy Model: A latency prediction method using CBOPs and transfer learning that significantly outperforms traditional BOPs metrics in predicting actual hardware latency, enabling true hardware-aware optimization.
Holistic Framework: An open-source, end-to-end pipeline that bridges MPQ exploration and deployment, supporting diverse targets including dedicated accelerators (BitFusion) and SIMD-extended RISC-V processors.

Experimental Results
The framework was evaluated on a diverse set of models (CNNs, ResNets, ViT, TinyLLaMa) and hardware targets (BitFusion accelerator, VexiiMiCo RISC-V variants).

Search Efficiency: In PTQ searches, MiCo consistently outperformed state-of-the-art methods (HAWQ-V3, HAQ, BOMP, w-based) across various models and constraints. For example, on SqueezeNet, MiCo achieved 68.22% accuracy in 32 iterations, whereas HAQ required 91 iterations to reach the same level.
QAT Performance: In low-bitwidth (1/2/4/8-bit) QAT searches, MiCo maintained competitive accuracy, particularly on smaller models where other methods struggled with high-dimensional search spaces.
Proxy Accuracy: The HAP model using CBOPs and full features achieved significantly higher prediction accuracy (e.g., $R^2$ of 0.94 vs. 0.50 for BOPs on BitFusion) and lower Mean Absolute Percentage Error (MAPE) compared to BOPs-based baselines.
Deployment Speedup: End-to-end deployment demonstrated that HAP-guided exploration yields superior real-world speedups. On BitFusion, HAP-guided MPQ reduced latency to 60% of the baseline (vs. 72% for BOPs-guided) with negligible accuracy loss. On RISC-V CPUs, HAP-guided exploration achieved up to 40% latency reduction compared to uniform quantization, whereas BOPs-guided exploration yielded significantly lower speedups due to unmodeled hardware overheads.

Significance
The paper claims that MiCoPro addresses the critical gap between theoretical quantization efficiency and practical hardware deployment. By moving beyond simple BOPs metrics to a hardware-aware proxy model and providing a unified deployment stack, the framework enables the rapid exploration and direct deployment of MPQ models that achieve genuine acceleration on edge devices. The results demonstrate that hardware-aware optimization is essential for realizing the full potential of mixed-precision quantization, achieving up to 40% latency reduction with less than 3% accuracy drop across different hardware architectures.

More like this