CascadeLUT: Information-Ordered Streaming Inference for Bandwidth-Constrained FPGAs

Technical Summary: CascadeLUT

Problem Statement
While Field-Programmable Gate Arrays (FPGAs) offer low-latency, energy-efficient inference for neural networks (NNs), particularly through Lookup Table (LUT)-based models that eliminate multipliers, existing architectures face a critical bottleneck in bandwidth-constrained streaming scenarios. Conventional FPGA inference pipelines assume full-sample availability, buffering the entire input vector before computation begins. In environments where data arrives over limited-bandwidth links (e.g., off-chip memory, sensors, or interconnects), this approach causes pipeline stalls and idle compute cycles. Consequently, the system bottleneck shifts from computation to data movement, where the cost of transferring features exceeds the cost of evaluation, leading to increased end-to-end latency and reduced energy efficiency.

Methodology
The authors propose CascadeLUT, an information-ordered streaming inference framework designed specifically for bandwidth-constrained FPGAs. The core innovation is a statically scheduled cascade architecture that eliminates full-input buffering by partitioning features into ordered subsets.

• Information-Ordered Streaming: Instead of waiting for the complete input, features are transmitted in a fixed, learned order based on salience. The network processes these subsets as they arrive, progressively refining predictions. Computation begins immediately upon the arrival of the first subset, overlapping data movement with inference.
• Static Cascade Topology: The network is organized as a cascade where specific layers are designated to consume newly arriving features (feature injection stages), while others propagate intermediate activations. This schedule is determined at design time, ensuring deterministic, branch-free execution without runtime control logic.
• Feature Saliency and Ordering: A feature scheduling method derives a transmission order by analyzing the weights of a fully connected surrogate model. Features are ranked by their contribution to neuron activations (top-k masks) and assigned to subsets such that the most discriminative features arrive earliest.
• Learned Input Quantization: To minimize bitwidth under fixed bandwidth constraints, the framework integrates on-device preprocessing. Each feature is quantized using learned thresholds and scales, implemented directly as LUT-based comparators. This allows raw sensor data (e.g., int8) to be streamed and quantized on-chip without adding pipeline stages or significant latency.
• Hardware Co-Design: The architecture is implemented on a Xilinx Zynq Z-7045. The pipeline is fully static, with registers inserted only between feature stages to meet timing, allowing high clock frequencies (200 MHz) and efficient pipelining.

Key Contributions

1. Novel Architecture: An open-source toolflow implementing a statically scheduled cascade architecture that organizes computation around a learned feature schedule, enabling inference directly on streaming data.
2. Feature Scheduling: A method to derive a transmission order that prioritizes salient features under fixed bandwidth constraints, allowing accurate predictions from partial inputs.
3. Integrated On-Device Preprocessing: A demonstration that learned preprocessing (quantization) can be implemented in FPGA fabric with modest area overhead, enabling end-to-end pipelines that consume raw sensor data.
4. Empirical Validation: Comprehensive FPGA results across four datasets (MNIST, FashionMNIST, KWS, ToyADMOS) demonstrating significant improvements in latency, throughput, and energy efficiency compared to prior LUT-based baselines.

Results
Evaluated on a Xilinx Zynq Z-7045 at 200 MHz with a 112-bit/cycle input constraint, CascadeLUT achieves the following performance gains over prior ultra-low-latency architectures (including NeuraLUT-Assemble and DWN baselines):

• Latency: 4.0× to 12.5× reduction. For example, MNIST inference latency drops from 40 ns to 10 ns.
• Throughput: 3.0× to 5.0× increase (e.g., up to 100.0M samples/s on MNIST).
• Energy Efficiency: Up to 13.8× lower energy per sample, attributed to the reduction in active pipeline cycles.
• Accuracy: Maintains comparable or superior accuracy to baselines (e.g., 98.59% on FashionMNIST with augmentation) despite operating on truncated input subsets.
• Resource Usage: The approach utilizes 1.2× to 4.4× more Look-Up Tables (LUTs) than the smallest DWN baseline per task. This increase is attributed to the fan-in required for feature injection and inter-stage logic. However, it remains significantly more area-efficient than ULEEN (which uses 4–12× more LUTs).
• Preprocessing Overhead: On-device quantization adds a modest area overhead (e.g., 0.5k LUTs for MNIST), which is reduced by feature truncation (only quantizers for used features are instantiated).

Significance
The paper posits that co-designing feature transport, representation, and computation around streaming constraints fundamentally improves FPGA inference efficiency. By shifting the paradigm from "compute after full-buffer" to "compute as data arrives," CascadeLUT addresses the data-movement bottleneck that limits current LUT-based systems. The results demonstrate that deterministic, bandwidth-aware streaming inference can achieve orders-of-magnitude improvements in latency and energy without sacrificing accuracy, making it a viable solution for real-time, resource-constrained edge applications.