Technical Summary: LL-ViT – Edge Deployable Vision Transformers with Look Up Table Neurons

Problem Statement

Vision Transformers (ViTs) have achieved state-of-the-art performance in computer vision but face significant barriers to deployment on edge devices, particularly Field-Programmable Gate Arrays (FPGAs). Their large computational, memory, and energy footprints exceed the capabilities of commercial edge platforms. For instance, compact variants like DeiT-T require approximately 20MB of weight storage, surpassing the on-chip memory of many FPGAs.

Existing optimization strategies, such as off-chip weight loading and aggressive quantization, introduce trade-offs: frequent off-chip access increases energy consumption, while aggressive quantization often degrades model accuracy. Furthermore, recent hybrid solutions using pipelined execution on high-end platforms (e.g., AMD/Xilinx Versal) achieve high throughput but at power levels (e.g., ~40W) unsuitable for battery-operated edge scenarios.

While recent works on Logic and Look-Up Table (LUT) based networks (e.g., LogicNets, NeuraLUT, Differentiable Weightless Neural Networks) offer models with reduced memory and compute footprints, they have historically been limited to small datasets and simple tasks. These models typically perform poorly on standard computer vision benchmarks like CIFAR-10/100 and have not been successfully integrated into the complex architecture of Vision Transformers.

Methodology

The authors propose LL-ViT (Learned-LUT based Vision Transformers), a novel algorithm-hardware co-design that integrates LUT neurons directly into the transformer architecture to optimize for edge inference.

1. Workload Analysis and Motivation

The authors characterize typical ViT architectures (specifically DeiT-T) and identify that Multilayer Perceptron (MLP) layers (channel mixers) are the primary bottleneck. These layers contribute over 60% of the total model weights and 55% of the total Multiply-Accumulate (MAC) operations. This observation motivates replacing the standard MLP block with a more efficient alternative while retaining the Multi-Head Self-Attention (MHA) token mixer, which is less weight-dominant.

2. Learned LUT-based Channel Mixer

Instead of simply replacing multiplications with table lookups post-training, the authors design a learnable LUT-based channel mixer that is trained end-to-end.

• Architecture: The proposed block replaces the standard two-layer MLP. It utilizes thermometer encoding to convert input activations into bit sequences, which are then processed through configurable layers of LUT neurons.
• Differentiability: To enable backpropagation, the authors employ Extended Finite Difference techniques (from Differentiable Weightless Neural Networks) to approximate gradients for LUT entries.
• Conditional Summation: A novel conditional summation layer follows the LUT neurons. It adds encoded values to the output only if the corresponding LUT neuron output is active (1). This layer allows the network to produce real-valued outputs required by the transformer while maintaining a multiplication-free inference path. During training, these encoded values are learned in full precision (fp32) and quantized post-training (e.g., to int4) to match the rest of the network.

3. FPGA Accelerator Design

The authors co-design a hardware accelerator specifically for LL-ViT:

• Processing Elements (PE): A dedicated PE is designed for the LUT-based channel mixer. It processes input activations row-wise in a pipelined fashion using ping-pong buffers to avoid stalls.
• Resource Mapping: The LUT neurons are mapped directly to the LUT slices within the FPGA logic fabric, eliminating the need for DSPs or BRAMs for weight storage in these layers. This contrasts with conventional accelerators that require DSPs for multiplication and BRAMs for weight storage.
• Stationary Weights: All model weights are stored on-chip in dedicated BRAMs, eliminating off-chip memory access during inference.
• Token Mixer: The MHA block is implemented using a systolic array for matrix multiplications, interfaced with the LUT-based channel mixer PE.

Key Contributions

1. Novel Architecture: Proposal of a learnable LUT-based channel-mixer block that seamlessly integrates into Vision Transformers, replacing the computationally heavy MLP layers.
2. Edge-Optimized Model: Development of an edge-optimized ViT (LL-ViT) that combines self-attention token mixing with multiplication-free channel mixing, significantly reducing computational demand and memory consumption.
3. Hardware Co-Design: Design of a low-power FPGA accelerator that avoids off-chip weight movement and leverages the native LUT resources of FPGAs for the channel mixer.
4. Performance on Complex Tasks: Extension of LUT-based neural networks from small-scale datasets to complex computer vision tasks (CIFAR-10/100, TinyImageNet) within a ViT backbone, demonstrating that LUT neurons can learn positional invariance and complex feature interactions.

Experimental Results

The authors evaluated LL-ViT on an AMD Xilinx Virtex FPGA (xcvu9p) against a fully quantized (INT8) baseline (I-ViT-T) and other prior works.

• Accuracy: LL-ViT achieves accuracies comparable to the baseline:
  • CIFAR-10: 95.5% (vs. 95.4% baseline)
  • CIFAR-100: 78.8% (vs. 79.2% baseline)
  • TinyImageNet: 60.9% (vs. 60.4% baseline)
  • Flowers-102: 91.6% (vs. 91.3% baseline)
• Model Size & Compute:
  • 60% reduction in model weights (1.93 MB vs. 5.06 MB).
  • >50% reduction in MAC operations (0.65 GMACs vs. 1.31 GMACs).
  • The reduced size allows the entire model to fit within on-chip memory.
• Efficiency:
  • 1.9× improvement in energy efficiency (2.14 mJ/inference vs. 4.05 mJ for baseline).
  • 1.3× lower latency (5.33 ms vs. 6.93 ms).
  • Throughput: Achieves 1083 FPS at a power budget of 10.9W.
• Comparison: LL-ViT outperforms prior LUT-based works (e.g., TreeLUT, DiffLogicNet) which report significantly lower accuracies (40–80% on CIFAR-10) and demonstrates superior efficiency compared to other ViT accelerators (e.g., HG-PIPE, ViTA) at a fraction of the power.

Significance and Claims

The paper claims that LL-ViT represents a significant step forward in making Vision Transformers viable for edge deployment. By targeting the MLP layers—the primary source of memory and compute overhead—with learnable LUT neurons, the authors demonstrate that it is possible to:

1. Drastically reduce model size and energy consumption without sacrificing accuracy on standard vision benchmarks.
2. Fully utilize FPGA resources by mapping neural computations directly to LUT slices, eliminating the need for off-chip memory access and reducing reliance on power-hungry DSPs.
3. Bridge the gap between efficient LUT-based neural networks and complex transformer architectures, proving that LUT neurons can be trained to handle the complexities of vision tasks previously reserved for dense, multiplication-heavy networks.

The authors position LL-ViT as a "promising lightweight alternative" to traditional ViTs, paving the way for "tiny ViTs" that can be deployed on resource-constrained, battery-powered edge devices. They note that while the technique was demonstrated on a tiny ViT baseline, the structural benefits are expected to scale to larger variants.