DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computation

Technical Summary: DeVIT – Low-Power Vision Transformer Acceleration Using Delta Computation

Problem Statement
Vision Transformers (ViTs) have achieved state-of-the-art performance in computer vision by leveraging self-attention to model long-range dependencies. However, their deployment on resource-constrained devices is hindered by substantial computational and memory requirements. Unlike Convolutional Neural Networks (CNNs), ViTs exhibit quadratic complexity in sequence length for self-attention, leading to massive multiply-accumulate (MAC) loads. For instance, a ViT-B/16 model requires 17.6 billion MACs for a single 224×224 image, significantly exceeding the 4.1 billion MACs required by ResNet-50. While quantization (e.g., 8-bit) is a standard method to reduce model size and complexity, it introduces a secondary property: value locality. In low-bit quantized matrices, weights are restricted to a limited range of discrete values, causing many weights within a row to be identical or very close in value. Existing hardware accelerators often fail to fully exploit this locality to eliminate redundant multiplications, particularly in the dense linear layers (projections and Feed-Forward Networks) that dominate ViT computation.

Methodology: DeVIT
The paper proposes DeVIT (Delta-coded Vision Transformer), an acceleration method that leverages differential computation to replace expensive multiplications with shift-and-add operations. The core methodology involves three key steps:

1. Weight Sorting and Delta Encoding:
   Instead of storing absolute weight values, DeVIT sorts the weights within each row of the weight matrix in ascending order. It then stores the differences ($\Delta W$) between consecutive sorted weights rather than the weights themselves.

   • Delta Computation: The multiplication of an input element $a$ with a sorted weight $W_2$ is computed as $a \times W_2 = (a \times W_1) + (a \times \Delta W)$, where $W_1$ is the preceding weight.
   • Shift-and-Add: Due to value locality, $\Delta W$ values are typically small (often 0, 1, or 2). These small differences are approximated as powers of two, allowing the multiplication $a \times \Delta W$ to be replaced by a simple bit-shift operation.
   • Zero Delta Skipping: When consecutive weights are identical, $\Delta W = 0$. The system skips the computation for these instances and reuses the previous partial product, effectively eliminating operations for redundant weights.

2. Input-Stationary Dataflow:
   To enable the reuse of multiplication results, DeVIT adopts an input-stationary execution order. Input elements are fetched sequentially and reused across all required computations for a specific row of weights. This ensures that a single input operand is multiplied by a sequence of sorted weights, making the differential approach feasible.

3. Partitioning Strategy:
   To manage buffer sizes and power consumption, the weight matrices are partitioned into smaller blocks (e.g., 64 or 16 elements) before sorting and encoding.

   • Trade-offs: Smaller partitions reduce the storage required for destination indices and output buffers but slightly increase the magnitude of deltas (reducing the frequency of zero/small deltas).
   • Index Encoding: Each delta is paired with an index indicating its original column position to restore the correct output order. The encoding uses a 4-bit delta code (sign + 3-bit magnitude for powers of two) and a variable-width index depending on the partition size.

Key Contributions

• Delta-Based Acceleration for ViTs: DeVIT is the first architecture to adapt differential computation (previously explored in CNNs like $\Delta$NN) specifically for the dense, token-based dataflow of Vision Transformers.
• Multiplier-Less Matrix Multiplication: By transforming weight matrices into delta-encoded formats, the approach converts the dominant GEMM (General Matrix Multiply) operations in ViTs into basic shift-and-add processes, significantly reducing hardware complexity.
• System-Aware Optimization: The paper introduces a specific dataflow, partitioning strategy, and index-management scheme tailored to the dense projection and feed-forward layers of ViTs, distinguishing it from prior work that focused on sparsity or attention mechanisms.
• Hardware Implementation: The authors present a RTL design synthesized in a 15 nm technology process, demonstrating the practical feasibility of the approach.

Experimental Results
The authors evaluated DeVIT on several benchmarks, including ViT-B/16, DeiT, Swin Transformer, and DETR-ResNet-50, using 8-bit quantized weights.

• Computation Load: DeVIT reduces the normalized computation load to 0.53 of an unoptimized baseline. This is lower than the FACT method's attention-skip (0.70) and FC1-skip (0.61) configurations, though slightly higher than FACT's most aggressive FC2-skip (0.47).
• Energy Efficiency: The measured single-GEMM energy for DeVIT is 159.57 nJ. This is approximately 5.5% lower than the lowest-energy ShiftAddLLM configuration (168.84 nJ) included in the comparison.
• Accuracy: The method maintains high accuracy with minimal degradation. For example, ViT-B/16 Top-1 accuracy drops from 83.71% (FP32) to 83.48% (DeVIT $\Delta$-64). The largest observed accuracy drop was 3.11 percentage points for Swin Top-1 and 2.53 mAP points for DETR in the 32-element partition configuration.

Significance and Claims
The paper claims that DeVIT significantly enhances the practicality of deploying transformer models for image processing tasks across a wider range of devices by reducing both computational load and energy consumption beyond what is achievable with quantization alone. By exploiting value locality through differential computation, DeVIT simplifies the essential matrix multiplication operations in transformer layers into shift-and-add processes without requiring complex dynamic scheduling or per-input prediction. The authors position DeVIT as a complementary system-aware optimization that targets the dense linear layers—the primary bottleneck in ViTs—offering a balanced trade-off between hardware efficiency, memory usage, and model accuracy.