No Attention, No Problem: DPU-Aware Attention Approximation in Modern YOLO on FPGA

Technical Summary: No Attention, No Problem: DPU-Aware Attention Approximation in Modern YOLO on FPGA

Problem Statement
Edge-based Artificial Intelligence (AI) requires a delicate balance between accuracy, speed, and power efficiency, particularly for real-time object detection. While Field-Programmable Gate Arrays (FPGAs) offer high performance with low power consumption through reconfigurable hardware, deploying state-of-the-art (SOTA) object detection models on FPGA System-on-Chips (SoCs) presents significant challenges. Specifically, modern YOLO variants (such as YOLOv11 and YOLOv26) utilize complex components like spatial attention mechanisms, specific activation functions (e.g., SiLU), and split operations that are not natively supported by commercial Deep Learning Processor Units (DPUs) like the AMD Xilinx DPUCZDX8G. When these unsupported operators are encountered, execution falls back to the CPU, creating bottlenecks that negate the efficiency gains of hardware acceleration. Furthermore, existing literature largely focuses on earlier YOLO versions or standard horizontal bounding boxes, leaving modern attention-based architectures and oriented object detection (OBB) underexplored in the context of FPGA deployment.

Methodology
The authors propose a systematic framework for adapting modern, attention-based YOLO models for deployment on the AMD Xilinx ZCU104 FPGA SoC using the Vitis AI toolchain. The methodology involves four key stages:

1. Dataset Preparation: The study utilizes six benchmark datasets covering both standard (horizontal) and oriented object detection: COCO, Pascal VOC, and KITTI for standard tasks; DOTA and DIOR-R for oriented tasks; and a custom-built human presence dataset.
2. DPU-Aware Architecture Adaptation: To ensure full execution on the DPU, the authors modify the model architectures (YOLOv8, YOLOv11, and YOLOv26, including OBB variants) to replace unsupported components:
   • Activation Functions: The unsupported Sigmoid Linear Unit (SiLU) is replaced with the Rectified Linear Unit (ReLU).
   • Split Operations: In feature-fusion blocks (e.g., C2f), unsupported split operations are substituted with equivalent $1 \times 1$ convolution operations. The weights are initialized as binary selection matrices to replicate the channel splitting behavior.
   • Attention Mechanism Approximation: The complex spatial attention modules (e.g., C2PSA in YOLOv11), which rely on unsupported operations like reshape, transpose, matrix multiplication, and Softmax, are approximated. The authors propose a DPU-compatible version using parallel $1 \times 1$ convolutions to extract Query, Key, and Value features, approximating attention weights via scaled element-wise products, replacing Softmax with Hard-Sigmoid, and projecting results back via $1 \times 1$ convolutions.
3. Training and Quantization: Models are trained using the Ultralytics library with mixed-precision (FP16). Post-Training Quantization (PTQ) is applied to convert models from 32-bit floating-point to 8-bit integer representation, a necessary step for DPU deployment.
4. Compilation and Deployment: Quantized models are compiled using the Vitis AI compiler targeting eight different DPUCZDX8G IP core configurations (B512 to B4096). The hardware design is rebuilt using Vivado and Vitis to generate bitstreams and deployment images for the ZCU104.

Key Contributions
The paper makes four primary contributions:

1. Customized Architecture: It presents a DPU-aware attention approximation architecture that enables the deployment of modern, attention-based YOLO variants (v11 and v26) on commercial FPGA SoCs.
2. Comprehensive Benchmarking: It provides an end-to-end performance evaluation of these models across both standard and oriented object detection tasks using state-of-the-art datasets.
3. Architectural Analysis: It offers a detailed analysis of performance (latency, FPS, power, resource utilization) across all eight DPUCZDX8G IP core configurations.
4. Quantitative Baseline: The work establishes a quantitative baseline for deploying standard and oriented bounding box detection models on FPGA SoCs using the Vitis AI toolchain.

Results
The study evaluates the modified models across all datasets and DPU configurations:

• Accuracy: The architectural modifications (replacing SiLU, approximating attention) resulted in an average absolute reduction of approximately 3% in mAP@0.5:0.95 compared to the original FP32 models. The subsequent deployment on FPGA (involving 8-bit quantization) led to an additional average reduction of roughly 7% compared to CPU execution, resulting in a total accuracy drop of approximately 5% for the final deployed system.
• Performance: YOLOv26n and YOLOv26n-OBB achieved the highest end-to-end throughput. On the largest DPU configuration (B4096), they reached 34.05 FPS for standard detection and 29.55 FPS for oriented detection.
• Latency and Bottlenecks: While DPU-specific FPS increased and latency decreased as the DPU architecture scaled from B512 to B4096, a significant gap remained between DPU performance and end-to-end performance. The authors identify post-processing (specifically Non-Maximum Suppression or NMS) as a critical bottleneck, as it is executed on a single CPU core, limiting overall throughput regardless of DPU capacity.
• Efficiency: The B4096 configuration consumed up to 8.760 W, representing an 83% increase over the B512 configuration (4.792 W). Despite this, the authors claim the solution achieves approximately 3× lower power consumption compared to state-of-the-art alternatives.

Significance
The paper claims that its work addresses a critical gap in the literature by enabling the deployment of the latest attention-based YOLO architectures on FPGAs, a task previously hindered by hardware incompatibilities. By systematically adapting these models and benchmarking them across diverse datasets and DPU configurations, the study provides a practical roadmap for edge AI developers. The results demonstrate that while a trade-off in accuracy (approx. 5%) is necessary for quantization and hardware adaptation, the approach yields significant gains in power efficiency and real-time throughput, making it viable for applications such as autonomous driving, drone navigation, and surveillance. The authors modestly note that future work will focus on removing NMS dependencies to further alleviate the post-processing bottleneck and improve end-to-end performance.