Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integration

1. Problem Statement

Emerging continual learning applications (e.g., robotics, autonomous vehicles) require Neural Processing Units (NPUs) that can efficiently handle both training and inference on the edge.

• The Trade-off: Inference typically uses low-bitwidth integers (INT8/INT4) for energy efficiency, while training requires large dynamic ranges traditionally handled by FP32. Using FP32 for everything is too expensive in area/power; using low-precision integers for training causes accuracy loss.
• The MX Standard: The Microscaling (MX) format bridges this gap by grouping elements under a shared exponent, allowing narrow formats (FP8, FP6, FP4, INT8) to maintain dynamic range.
• The Bottleneck: Existing MX Multiply-Accumulate (MAC) designs face a critical inefficiency in their reduction trees (the accumulation logic):
  • Integer Accumulation: Requires expensive conversion of narrow floating-point products to wide integers.
  • FP32 Accumulation: Requires costly normalization logic and incurs quantization losses.
  • System-Level Issues: State-of-the-art (SotA) NPU platforms (like SNAX) often provision memory bandwidth for static worst-case scenarios. This leads to underutilized memory channels and wasted dynamic power when operating at lower precisions (e.g., MXINT8 vs. MXFP4).

2. Methodology

The authors propose a holistic solution involving arithmetic unit optimization and system-level integration.

A. Hybrid Precision-Scalable Reduction Tree

The core innovation is a new MAC datapath that combines the strengths of two existing SotA approaches:

1. Hybrid Architecture: The design integrates an early-accumulation scheme (from long-integer approaches) into a FP32 reduction tree.

   • It utilizes 28-bit signal widths in the Level-2 (L2) adder to avoid over-provisioning.
   • It employs a Multiplexer (MUX) to dynamically handle alignment. Instead of always extending the product sum by 24 bits (as in pure FP32 approaches) or using massive 119-bit integers, the MUX selects whether to extend the product sum left or right based on the relative magnitude of the stored FP32 partial result.
   • Result: This reduces the normalization input width from 77 bits to 53 bits, significantly lowering hardware complexity.

2. Accumulation Precision Optimization:

   • The authors analyzed the trade-off between addition error and quantization error.
   • They determined that as long as the quantization error (inherent to converting to MX formats) exceeds the addition error, the addition precision can be relaxed.
   • Through simulation with uniform and Gaussian distributions, they found that a 16-bit mantissa for stored partial results is sufficient to keep addition error negligible compared to quantization error, allowing for further hardware reduction.

B. NPU Integration (SNAX Platform)

The proposed MACs are integrated into the SNAX NPU framework (a RISC-V based cluster) with specific optimizations:

• 8x8 Tensor Core: An array of 64 optimized MACs arranged in a spatial mesh for General Matrix Multiplication (GeMM).
• Dynamic Data Streamers: Unlike static bandwidth provisioning, the data streamers use dynamic channel gating.
  • The system activates only the number of memory access channels required by the specific precision mode (e.g., 1 channel for MXINT8, 4 for MXFP8/6, 4 for MXFP4).
  • This reduces memory traffic, bank contention, and dynamic power consumption.
• Unified Control: A Configuration and Status Register (CSR) interface allows the RISC-V core to dynamically switch precision modes, accumulation depth, and matrix tile sizes at runtime.

3. Key Contributions

1. Hybrid Reduction Tree: A novel MAC architecture that merges integer and floating-point accumulation benefits, reducing normalization overhead and adder width while supporting all six MX data types (INT8, FP8, FP6, FP4).
2. Precision Relaxation: A method to reduce the accumulation mantissa width to 16 bits without compromising final model accuracy, further lowering area and energy costs.
3. System-Level Optimization: The first integration of precision-scalable MX MACs into the SNAX NPU platform, featuring dynamic bandwidth adaptation to match the specific requirements of different MX precisions.
4. Comprehensive Evaluation: Validation at both the MAC level and the full system level, comparing against SotA implementations.

4. Results

The design was implemented in GlobalFoundries 22FDX technology and evaluated at 500 MHz.

• MAC Level Performance:

  • Achieves a maximum frequency of 1800 MHz (compared to 1100 MHz for SotA FP32 addition designs).
  • Outperforms SotA in area and energy across the full frequency range for most modes.

• System Level Efficiency (GOPS/W):

  • MXINT8: 657 GOPS/W (1.59x improvement over SotA).
  • MXFP8/6: 1,438 – 1,675 GOPS/W (3.05x – 3.21x improvement).
  • MXFP4: 4,065 GOPS/W (1.13x improvement).
  • Throughput: 64 GOPS (INT8), 256 GOPS (FP8/6), and 512 GOPS (FP4).

• Utilization:

  • The integrated NPU achieves 94% – 99% computational utilization on ResNet-18 and ViT workloads, indicating minimal control and memory bottlenecks.

• Area:

  • Total system area is 0.60 mm², with the MX tensor core occupying ~29.5% of the area.

5. Significance

This work addresses a critical gap in edge AI hardware: the inability to efficiently support both training and inference on a single, energy-constrained platform.

• Unified Platform: It proves that a single MAC array can dynamically switch between high-precision training (FP8/FP6) and low-precision inference (INT8/FP4) without the massive area/power penalties of traditional FP32 or the accuracy loss of pure integer training.
• Efficiency: By optimizing the reduction tree and adapting memory bandwidth dynamically, the design significantly outperforms existing SotA solutions in energy efficiency (up to 3.2x).
• Scalability: The integration into the SNAX framework demonstrates a path toward agile, programmable NPUs that can adapt to the evolving demands of continual learning applications in robotics and IoT.