NAS-Driven Hardware Accelerator Exploration for Edge AI and Quantization Effects on the Pareto Space

Imagine you are trying to build the ultimate robot brain, but you have a strict rule: it must fit inside a tiny, battery-powered gadget like a smartwatch or a drone. This is the world of "Edge AI," where artificial intelligence lives on small devices instead of giant, power-hungry cloud servers. To make this work, engineers have to solve a tricky puzzle: they need a brain that is smart enough to recognize faces or sounds, but small and fast enough to run without draining the battery in minutes.

To find the perfect brain design, scientists use a method called Neural Architecture Search (NAS). Think of this as a super-fast, automated chef who tries millions of different recipes (neural network designs) to find the one that tastes the best (highest accuracy) while using the least amount of ingredients (computing power). Usually, these chefs cook with "full-precision" ingredients, which are like using exact, heavy measurements. But for tiny gadgets, you have to switch to "quantization," which is like switching to smaller, lighter measuring cups (using fewer bits to represent numbers). The big question is: if you change the measuring cups, does the recipe that looked perfect in the kitchen still taste the same on the road? This paper dives into that exact question, exploring how changing the measurement system affects the search for the best robot brain and how to fit that brain onto a special, reconfigurable chip.

---

The Three-Stage Journey to a Tiny Brain

The authors of this paper propose a clever, three-step pipeline to solve the problem of fitting AI onto hardware. They call it a "NAS-then-quantized" approach, which is a bit like designing a car on a computer first, then testing how it handles on a bumpy dirt road, and finally building the engine to match the terrain.

Stage 1: The Hardware-Agnostic Frontend
First, the team uses a "surrogate" model—a smart predictor that guesses how good a design is without actually building it. They train this predictor on a massive library of 15,625 different neural network designs (from a dataset called NAS-Bench-201). This stage is "hardware-agnostic," meaning it doesn't care about the specific chip yet; it just looks for the best designs based on accuracy and how many math operations (FLOPs) they need. They use two search strategies: a simple "Random Search" (picking designs at random) and a "Multi-Objective Evolutionary Algorithm" (which mimics natural selection to evolve better designs). After this stage, they have a shortlist of ten top candidates.

Stage 2: The Quantization Bridge
Next comes the reality check. The team takes those ten top candidates and shrinks them down using a technique called Post-Training Quantization (PTQ), specifically switching them to INT4 (using only 4 bits per number instead of the usual 32). This is like taking a high-definition photo and compressing it to fit on an old phone. They use a tool called Brevitas to do this.

Here is where things get interesting. When they compress the models, some designs that looked great in the first stage might fail or become worse. The "Quantization Bridge" acts as a filter. It re-ranks the designs based on how well they survive the compression. If a design falls apart, it gets tossed out. If no designs survive, the system loops back to Stage 1 to try again. They chose INT4 because, while it's very aggressive, they found that with a tiny bit of extra training (just two epochs), the models could recover their accuracy significantly, jumping from a low of 9.49% up to 83.53%, proving that the original model information was largely preserved.

Stage 3: The Hardware-Aware Backend
Finally, the surviving designs need a home. The team maps these compressed brains onto a specific type of hardware called a CGRA (Coarse-Grained Reconfigurable Array). Think of a CGRA as a Lego board where you can rearrange the blocks to fit any shape. They use an evolutionary algorithm to explore different ways to arrange the hardware (changing the number of processing units, memory depth, and data paths). They don't build the physical chip; instead, they use a "free analytical oracle"—a mathematical simulator—to predict how fast and efficient each arrangement would be. They pick the configuration that minimizes delay and maximizes the use of the hardware's resources.

The Big Surprise: Don't Throw Away the Old Map

The most exciting discovery in this paper isn't just the three-stage pipeline; it's what they learned about the relationship between the "full-precision" world and the "compressed" world.

Usually, when you compress a model, you might think you need to start your search from scratch, training a new predictor specifically for the compressed (INT4) world. The authors tested this idea. They compared two approaches:

1. The Dedicated INT4 Surrogate: A predictor trained specifically on compressed data.
2. The FP32 Zero-Shot Surrogate: A predictor trained on the original, full-precision data, which they then used to guess the performance of the compressed models without retraining.

The results were surprising. In simulations, the FP32 zero-shot surrogate actually performed better than the dedicated INT4 one. When they ran their search strategies, the FP32 predictor found designs that covered a wider range of the best possible trade-offs (the "Pareto space") than the INT4-specific predictor.

Why? The authors suggest that the full-precision data is "cleaner" and less noisy. Even though the compressed world is different, the ranking of which designs are "good" and which are "bad" stays surprisingly similar. The paper notes that while the exact order of the designs changes (about 21% of the time, a design that was better than another in full precision becomes worse after compression), the overall structure of the "best" designs remains stable enough that the old map (the FP32 predictor) is still a reliable guide.

The Stability of the Puzzle

The team didn't just guess; they measured the chaos. They looked at all 15,625 architectures to see how much the "Pareto front" (the list of the absolute best designs) shifted when they switched to INT4.

• The Shift: The list of top designs completely reorganized. None of the original top designs survived as the top designs in the compressed world.
• The Flip: About 21.73% of the time, the relationship between two designs flipped (Design A was better than Design B in full precision, but Design B became better after compression).
• The Correlation: Despite this chaos, the overall ranking correlation was still moderate (0.6655), meaning the general "shape" of the good designs was preserved.

They also found that smaller, simpler models (those with fewer math operations) were much more sensitive to the compression, often losing accuracy. Larger, more complex models were surprisingly robust, likely because their structure naturally smoothed out the errors introduced by the smaller numbers.

The Hardware Results

Finally, they took three of their best surviving designs and mapped them onto the CGRA hardware. They found that despite the designs being different, they all converged on the exact same optimal hardware configuration: a grid of 16 rows and 66 columns of processing units with a specific memory depth. This suggests that for this type of task, the hardware "sweet spot" is quite consistent, regardless of the specific neural network design, as long as the design uses similar building blocks.

Conclusion

This paper suggests that we don't need to reinvent the wheel every time we want to shrink an AI model for a tiny device. By using a three-stage process—finding the best designs in full precision, filtering them through a compression test, and then mapping them to flexible hardware—we can efficiently find the perfect balance between accuracy and size. Most importantly, it suggests that we can trust our full-precision search tools to guide us even in the compressed world, saving time and effort in the quest to put powerful AI into our pockets.