🤖 machine learning

Design Rules for Extreme-Edge Scientific Computing on AI Engines

This paper establishes design rules for deploying extreme-edge scientific neural networks on FPGA AI Engines by introducing a latency-adjusted resource equivalence (LARE) metric and dataflow optimizations to determine when these engines outperform traditional programmable logic for high-performance, low-latency inference.

Original authors: Zhenghua Ma, G Abarajithan, Dimitrios Danopoulos, Olivia Weng, Francesco Restuccia, Ryan Kastner

Published 2026-04-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenghua Ma, G Abarajithan, Dimitrios Danopoulos, Olivia Weng, Francesco Restuccia, Ryan Kastner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-speed train station (the Scientific Edge). Every second, thousands of trains (data from sensors) arrive, and you need to instantly decide which ones are important and which ones can be ignored. You have to make these decisions in the blink of an eye—microseconds—because if you slow down, the whole station crashes.

To do this, you have two types of workers:

The Custom-Built Assembly Line (Programmable Logic/PL): These are workers who can be rearranged to do exactly one specific task perfectly. They are great for small jobs.
The Super-Fast Robot Army (AI Engines/AIE): These are a grid of identical, incredibly fast robots. They are great for heavy lifting, but they are rigid and need to be told exactly how to move their arms.

For a long time, scientists used the Assembly Line for everything. But as the "trains" got more complex (bigger AI models), the Assembly Line ran out of space. They tried to make the workers do more jobs at once by having them juggle (reuse), but this made them slower and prone to mistakes.

This paper is a guide on when to switch to the Robot Army and how to organize them so they don't trip over each other.

1. The Problem: The "Juggling" Limit

When the job is small, the Assembly Line is perfect. But as the job gets bigger, you run out of workers. You start forcing the few remaining workers to juggle multiple tasks at once (this is called a "high reuse factor").

The Analogy: Imagine a chef in a tiny kitchen. If they have to chop 10 onions, they can do it fast. If they have to chop 10,000 onions, they might try to chop 5 at a time with one knife. It takes forever, and they get tired.
The Result: The Assembly Line hits a "Resource Wall." It can't handle the big models fast enough to keep up with the 40 million trains per second.

2. The Solution: The "Robot Army" (AI Engines)

The AI Engines are like a massive army of specialized robots. They are built to do math incredibly fast and have their own memory pockets right next to them.

The Catch: You can't just throw the job at them. If you don't organize them well, they stand around waiting for instructions, or they bump into each other.
The Goal: The paper figures out exactly when the Robot Army is better than the Assembly Line and how to arrange the robots for maximum speed.

3. The "LARE" Metric: The Decision Compass

The authors created a special ruler called LARE (Latency-Adjusted Resource Equivalence).

The Analogy: Think of LARE as a "Tipping Point Calculator." It asks: "At what size does the Assembly Line get so slow from juggling that the Robot Army becomes the faster choice?"
Why it matters: It stops scientists from guessing. If your job is small, stick with the Assembly Line. If your job is big and the Assembly Line is struggling, switch to the Robots.

4. The Rules of the Road: How to Organize the Robots

Just putting robots in a grid isn't enough. The paper discovered specific "traffic rules" to make them run at top speed:

Rule 1: The Shape of the Task Matters.
- Analogy: If you are carrying boxes, it's easier to carry a wide, flat box than a tall, narrow one.
- The Rule: The robots work best when the data is "wide" (many output channels) rather than "tall" (many input steps). If you arrange the data the wrong way, the robots waste time waiting.
Rule 2: Don't Overcrowd the Aisle.
- Analogy: Imagine a dance floor. If you have 4 dancers, they move great. If you squeeze 100 dancers in, they bump into each other, and the dance slows down.
- The Rule: Adding more robots doesn't always mean more speed. After a certain point, adding more robots just creates traffic jams. There is a "sweet spot" for how many robots to use.
Rule 3: The "Band" Problem (Running out of Space).
- Analogy: Imagine the robots are arranged in rows. If you run out of rows, you have to start a second floor. But now, the robots on the first floor have to shout instructions to the second floor, which takes time.
- The Rule: Try to fit your whole job on the "ground floor" (one band of robots). If you have to use a second floor, you lose speed.
Rule 4: The Border Crossing Tax.
- Analogy: Sometimes you need the Assembly Line for one part of the job and the Robots for another. But every time a package moves from the Assembly Line to the Robots, it has to go through security.
- The Rule: Every time you switch between the two systems, you lose about 4% speed. So, try to keep the whole job in one system if possible.

5. The Grand Finale: The Big Win

The authors tested these rules on real, massive scientific problems (like analyzing particle collisions at CERN).

Before: The Assembly Line was too slow to handle the big models. The scientists had to throw away complex data because they couldn't process it fast enough.
After: Using the Robot Army with the new "Traffic Rules," they were able to process these massive models 4 times faster.
The Result: They could now keep up with the 40 million trains per second, saving data that was previously lost.

Summary

This paper is a manual for scientists who are running out of space on their old assembly lines. It tells them:

When to switch to the new Robot Army (use the LARE ruler).
How to arrange the robots so they don't trip over each other (the 7 design rules).
Proof that this works, allowing them to solve problems that were previously impossible.

It's about moving from a "make-do" approach to a "master plan" approach, ensuring that even the most complex scientific data gets processed in the blink of an eye.

1. Problem Statement

Extreme-edge scientific applications (e.g., at CERN's Large Hadron Collider) require real-time machine learning inference on sensor data with stringent constraints:

Latency: End-to-end latency must be in the order of microseconds.
Throughput: Systems must process data at rates up to 40 MHz.
Resource Constraints: All model weights must reside fully on-chip (no off-chip DRAM access), and batch sizes must be minimal (often 1 or 8).

Currently, these workloads are implemented on FPGA Programmable Logic (PL) using spatial dataflow frameworks like HLS4ML. While effective for small neural networks (NNs), this approach hits a "resource wall" as models grow larger. To fit larger models, designers must increase the reuse factor (time-multiplexing arithmetic units), which drastically increases latency and often causes synthesis tools to fail.

The AI Engine (AIE) architecture on modern FPGA SoCs (like AMD Versal) offers an alternative with high compute density, dedicated vector processors, and more on-chip memory. However, the scientific community lacks a systematic methodology to determine:

When to deploy on AIEs versus PL (given their fundamentally different architectures).
How to optimize dataflow and tiling for low-latency, event-driven scientific inference on AIEs, as existing AIE tools are optimized for high-throughput batched GEMM, not extreme-edge inference.

2. Methodology

The authors employed a systematic approach involving architectural characterization, micro-benchmarking, and full-system deployment:

Experimental Setup: Experiments were conducted on an AMD-Xilinx Versal VEK280 board. PL baselines used the HLS4ML toolchain (extended for Versal), while AIE implementations used the AIE4ML framework. Performance was measured via cycle-accurate hardware emulation.
Micro-Benchmarking: The authors isolated single dense layers to analyze the trade-off between resource usage and latency. They swept the reuse factor (rf) on PL to map the performance-resource curve.
Metric Development: They introduced the Latency-Adjusted Resource Equivalence (LARE) metric. LARE calculates the minimum PL resources required to match the latency of a specific AIE implementation. This serves as a decision boundary: if available PL resources exceed the LARE value, PL is preferred; otherwise, AIE is superior.
Optimization Exploration: The team systematically varied spatial tiling (distribution across the AIE array) and API-level tiling (block sizes within a single tile) to identify optimal design rules for low-latency workloads.
Hybrid Analysis: They quantified the latency overhead of crossing the boundary between PL and AIE fabrics to guide hybrid implementations.

3. Key Contributions

A. The LARE Metric

The paper introduces LARE as a quantitative decision boundary. It identifies the crossover point where AIE implementations outperform PL designs.

Insight: LARE is not linear; it depends heavily on layer shape (input/output dimensions) and the specific resource budget.
Utility: It helps designers decide whether to stick with PL (for small models) or migrate to AIE (for larger models) before hitting the PL resource wall.

B. Design Rules for AIE Tiling and Dataflow

Through empirical benchmarking, the authors derived seven specific design rules for extreme-edge scientific computing on AIEs:

API-Level Tiling: The optimal default tile size for API calls is $S = (4, 8, 8)$ . This offers the best balance of performance and granularity.
Workload Asymmetry: Performance is significantly higher when the output dimension ( $N$ ) is larger than the reduction dimension ( $K$ ). Tiling should prioritize the $N$ dimension.
Spatial Tiling Direction: When distributing workloads across the AIE array, column-wise expansion (increasing tiles along the $K$ dimension) yields lower latency than row-wise expansion.
Diminishing Returns: Increasing spatial parallelism yields diminishing returns. A per-tile workload of $8 \times 16 \times 32$ (Input $\times$ Reduction $\times$ Output) is a sweet spot; further splitting often reduces performance due to overhead.
Minimum Workload: To avoid under-utilization, the workload per tile should not drop below $8 \times 16 \times 32$ .
Column Exhaustion: Models should be tiled to maximize the use of a single "band" (row of tiles) without exceeding the physical column limit (31 columns on VEK280). Spilling into a second band causes resource contention and significant latency penalties.
PL-AIE Boundary Cost: Crossing the boundary between PL and AIE incurs a ~3.9% latency overhead per crossing. Hybrid designs are only beneficial if the computational gain of placing a specific layer in its optimal domain outweighs this cost.

4. Results

The authors validated their approach by deploying three real-world scientific neural networks that previously failed to meet the 40 MHz throughput requirement on PL:

VAE (Variational Autoencoder): LHC application.
Qubit Readout: Multi-qubit discriminator.
Deep Autoencoder: MLPerf Tiny benchmark.

Performance Gains:

Naive AIE Implementation: Even without optimization, a naive mapping (one layer per tile) was competitive with highly reused PL designs.
Optimized Implementation: Applying the derived design rules resulted in up to 4 $\times$ performance improvement over naive AIE implementations.
Throughput Achievement: All three optimized models exceeded the 40 MHz trigger frequency requirement, whereas their PL counterparts (even with aggressive reuse) fell short (e.g., VAE achieved 20.8 MHz on PL vs. 97.9 MHz on optimized AIE).

5. Significance

Bridging the Gap: This work provides the first systematic methodology for the scientific community to transition from PL to AIE for extreme-edge computing, addressing the "when" and "how" of deployment.
Expanding the Frontier: By enabling larger, more complex neural networks to run fully on-chip with microsecond latency, this research expands the capabilities of trigger systems in high-energy physics and other extreme-edge domains.
Practical Guidelines: The specific design rules (tiling strategies, asymmetry handling, and boundary costs) serve as a practical guide for engineers developing low-latency AI accelerators on Versal platforms, moving beyond generic high-throughput optimizations.
Toolchain Maturity: The work highlights the maturity of the AIE4ML toolchain and demonstrates its viability for scientific inference, provided that specific architectural constraints are respected.